More than Just Lines on a Map: Best Practices for U.S Bike Routes
SOA sucks, Fix it with Missions and FatDB
1. SOA SUCKS
FIX IT WITH “MISSIONS” AND FATDB
Justin Weiler, CTO, FatCloud
2. AGENDA
• Challenges of building a web scale application
• Essential ingredients of any web scale application
• Shortcomings of N-Tier and SOA and the real cost of “glue”
• “Mission” Oriented Architecture and FatDB as one solution
• Wrap-up and questions
5. WHAT’S SO HARD ABOUT DOING IT RIGHT?
• Configuration and Maintenance
• Uniformity and Complexity
• Scaling, Upgrades, and Downtime
• Pivoting and Extensibility
• Latency and Performance
• Single Points of Failure and Bottlenecks
• Fault Tolerance and High Availability
• Data Consistency
• Synchronization
• Legacy Integration / SQL
• Portability and Lock-in
• Telemetry
• Security
• TCO & TTM
SingleLegacy ofConsistency
FaultUniformity & &Maintenance
Scaling, UpgradesLock-in
ConfigurationPerformance
PointsTCO & Complexity
LatencySecurity &Availability
Tolerance & HighDowntime
Pivoting& Extensibility
Portability TTM /Bottlenecks
Data &
SynchronizationSQL
Telemetry
Integration
Failure
&
6. THE ESSENTIAL INGREDIENTS
• Store, retrieve, cache, and query object data
• Store, retrieve, and query file data
• Host synchronous business logic
• Host asynchronous batch job business logic
• High performance, scalable, consistent,
and fault tolerant plumbing
• Comprehensive management tooling
• Full SQL integration
• Portable
• Easy to use
7. AN OLD ARCHITECTURE
3-Tier Monoliths and Beefy Hardware
Typically:
• Rigid and Brittle
• Hard to Scale
• High Latency and Bottlenecks
• High Hardware and Maintenance Costs
• Low learning curve
PRESENTATION
LOGIC
Relational DB
8. WHAT IS NOSQL?
A NoSQL (Not Only SQL) database provides a mechanism for storage and
retrieval of data that employs less constrained consistency models than
traditional relational databases. Motivations for this approach include
simplicity of design, horizontal scaling and finer control over availability.
(Source: Wikipedia)
•
•
•
•
•
Distributed computing – scale out versus scale up
Fault tolerance through mirroring/replication
Ability to be schema-less and handle unstructured data
Eventually consistent as opposed to fully consistent
Several approaches – Key Value, Document, Graph etc
9. NOSQL STRENGTHS AND WEAKNESSES
Strength
Multi-node scalability
Weakness
Instantaneous data consistency
Source: Data Access for Highly-Scalable Solutions:
Using SQL, NoSQL, and Polyglot Persistence
10. A MORE MODERN ARCHITECTURE
A-La-Carte SOA with Costly Glue
Typically:
•
•
•
•
•
•
•
•
•
•
More Scalable & Flexible
Lower Upfront Costs
Low Synergy
Still Hard to Scale
Still High Latency
Impedance Mismatches
High Integration Costs
Long Learning Curve
High Maintenance Costs
PaaS Better, but Lock-In
PRESENTATION
SERVICE 1
SERVICE 2
NoSQL
Message Q
11. SYSTEMS INTEGRATION PAIN: TOO MANY GUI’S
= Mongo
+ RabbitMQ
+ Hadoop
+ Cassandra
+ Memcached
Multiple GUI’s
Multiple technologies
Difficult learning curve
Linux is primary O/S
Windows is secondary
O/S if at all
12. RECIPE FOR EASY SCALING
• Think holistically (NoSQL is just the beginning)
• Traffic and Data analysis
• Test and meter
• Provide ALL the essentials (NoSQL, Files, Sync, Async)
–Leverage an extensible, integrated platform
–Maximize existing investments (WCF, SQL)
• Organize “missions” rather than services
• Concentrate on your business logic
13. FATDB PLATFORM STANDARD FEATURE SET
ASYNC WORK QUEUE & PROCESSORS
Asynchronous job processing, able to
handle “big data” problems. Utilizes your
business logic created as FatProcs
NoSQL DATABASE &
CACHE
Document or keyvalue store (standalone DB and can be
easily integrated with
SQL)
High performance inmemory distributed
cache (IMDG)
FILE MANAGEMENT SYSTEM
Flexible storage of file-based
assets across the cloud
CORE FOUNDATION
Transparently provides the
distributed plumbing for routing,
redundancy, fault tolerance, data
consistency, synchronization, and
service discovery
APPLICATIONS
Support for your custom and
community synchronous
applications or business logic
created as FatApps
14. THE FATDB ARCHITECTURE
Mission Oriented Architecture (MOA) on an Integrated Platform
Typically:
•
•
•
•
•
•
High Scalability
High Flexibility
High Synergy
Low Latency
Low Impedance
Low Integration Costs
GROUP 1
CONSUMERS
• Short Learning Curve
• Low Maintenance Costs
• Portable
MISSION = BUSINESS
LOGIC + RELATED DATA
MISSION = TEMPLATE
FOR EVERY SERVER
PRESENTATION
GROUP 2
PRODUCTS
GROUP 3
ORDERS
15. WHAT IS POLYGLOT PERSISTENCE?
Polyglot Persistence, like polyglot programming, is all about choosing the
right persistence option for the task at hand.
Scott Leberknight
Different databases are designed to solve different problems. Using a single
database engine for all of the requirements usually leads to non- performant
solutions.
Martin Fowler
Polyglot Persistence is about using hybrid storage approach (RDBMS,
NOSQL,BLOB,FILE) that allows you to use the best tool for the job versus
being locked into one approach.
FatCloud
16. POLYGLOT DATA STORE: SQL WITH FATDB
Data Driven Request Routing
Calls for relational and
transactional data
SQL Server
Invoice
Invoice
Invoice
024
175
832
Web / Mobile or
other Client app
Calls for unstructured
and scale sensitive data
FatDB Server Cluster
Invoice
Invoice
Invoice
936
492
751
Invoice
Invoice
Invoice
595
037
275
Invoice
Invoice
Invoice
275
Invoice
037
Invoice
936
Invoice
275
Invoice
037
Invoice
936
Invoice
275
037
936
Invoice
Invoice
Invoice
832
Invoice
492
Invoice
024
Invoice
832
Invoice
492
Invoice
024
Invoice
832
492
024
Invoice
Invoice
Invoice
751
Invoice
175
Invoice
595
Invoice
751
Invoice
175
Invoice
595
Invoice
751
175
595
17. POLYGLOT DATA STORE: SQL WITH FATDB
Automatic Synchronization
FatDB Server Cluster
SQL Server
Invoice
Invoice
Invoice
024
175
832
Invoice
Invoice
Invoice
936
492
751
Invoice
Invoice
Invoice
595
037
275
Automate routine
synchronization
tasks for effortless
data consistency.
SQL Write Back
Invoice
Invoice
Invoice
275
Invoice
037
Invoice
936
Invoice
275
Invoice
037
Invoice
936
Invoice
275
037
936
Invoice
Invoice
Invoice
832
Invoice
492
Invoice
024
Invoice
832
Invoice
492
Invoice
024
Invoice
832
492
024
Invoice
Invoice
Invoice
751
Invoice
175
Invoice
595
Invoice
751
Invoice
175
Invoice
595
Invoice
751
175
595
1. Changes to SQL Server automatically sent to FatDB
2. Changes to FatDB automatically sent to SQL Server
18. EXAMPLE OF A POLYGLOT DATA STORE
Mobile Clients
1. Legacy apps continue unchanged
updating SQL Server
2. Changes in SQL Server are
transmitted to FatDB
3. FatDB is accessed by mobile, web,
cloud clients delivering scale and
fault tolerance
3
New Web App
FatDB Server Cluster (In house or In cloud)
Invoice
Invoice
Invoice
Invoice
275
Invoice
037
Invoice
936
Invoice
275
Invoice
037
Invoice
936
275
037
936
Invoice
Invoice
Invoice
Invoice
751
Invoice
175
Invoice
595
Invoice
751
Invoice
175
Invoice
595
751
175
595
Invoice
Invoice
Invoice
832
Invoice
492
Invoice
024
Invoice
832
Invoice
492
Invoice
024
832
492
024
2
4. Updates to FatDB are transmitted
back to SQL Server
4
SQL Server
Invoice
Invoice
Invoice
024
175
832
5. BI and Reporting tools can continue
accessing SQL Server as an
accurate data repository
1
Legacy Apps
Invoice
Invoice
Invoice
936
492
751
Invoice
Invoice
Invoice
595
037
275
BI & Reporting
5
19. WHY FATDB OR A SIMILAR TECHNOLOGY?
Do you want this…
20. WHY FATDB OR A SIMILAR TECHNOLOGY?
…or this?
PRESENTATION
GROUP 1
CONSUMERS
GROUP 2
PRODUCTS
GROUP 3
ORDERS
• Reduces complexity
• Decreases risk
• Enhances flexibility and
portability
• Increases performance
• Faster TTM
• Leverages existing
investments
• Lower TCO
21. FATDB MANAGEMENT STUDIO: A SINGLE PANE OF GLASS
One GUI
One set of API’s
Easy learning curve
Windows based
Manage servers,
Databases, file
Systems, work
queue and apps
from one tool
22. KICKING THE TIRES…
• Community edition is FREE!
• Installer, tutorials, code, and whitepapers available at
www.fatcloud.com
• POC program
• justin@fatcloud.com