4. • It
isn’t
just
about
geKng
all
the
data
there
• Algorithms
are
cool
but
we
need
humans
in
the
loop
• Hire
the
right
people
• Test
!
Test
!
Test!
hEp://theamazingios6maps.tumblr.com/
13. Adheres
to
Policy
Governed
by
Binds
to
Endpoint
Exposes
Serves
Service
consumer
Understands
Contracts
Implements
Service
Describes
Key
Component
Sends/receives Messages Sends/receives Rela>on
23. Cell
Architecture
HBase HBase HBase
Categories Customers ORCA
BUS
Interactions Reference …
Data
HDFS HBase HBase
24. Orchestra>on
Initiate business process
Endpoint
Manage
process
Service
Schedule
Route
request Invoke
servic
Workflow instance
es
Workflow Host
Manage
Monitor
Endpoint
engine workflows
workflows
workflows
Service
28. Memcached
GigaSpaces
Redis
GridGain
Caching Data grid
Oracle
Coherence
Columnar
WebSphere
eXtreme
Scale
Hama
HBase
Cassandra
In-memory
Pregel
Accumulo
Hypertable
Key-value store Neo4j
Graph
Hadoop
GlusterFS
Distributed file systems
RavenDB
ScaleBase
MongoDB
Relational
Document
CouchDB
NewSQL
IndexTank
Amazon
RDS
Analytics/MPP
VoltDB
Apache
Solr
AKvio
Aster
Data
Microsoo
PDW
Indexing Columnar
ParAccel
SAP
HANA
Oracle
Exadata
HP
Ver>ca
IBM
Netezza
EMC
Greenplum
29. Data
is
mul>-‐>ered
Datamart(s)
Cube
Real-‐>me
Datawarehouse
(RDBMS)
(MOLAP)
(in
memory)
(Hadoop/Hbase)
1-‐7
days
detailed
6-‐12
months
6-‐?
Months
Detailed
aggregated
20
years
detailed
1-‐3
years
aggregated
aggregated
30. Data
is
mul>-‐>ered
Datamart(s)
Real-‐>me
Data
warehouse
(Columnar)
(Hadoop/Hbase)
1-‐7
days
6-‐12
months
detailed
Detailed
20
years
detailed
aggregated
32. Aggregated
Repor>ng
SQL endpoint
Endpoint
Produce
Request
reports
ODS/DM
Report Report
Raw
data
Out
Transpose
Endpoint
Pull data
Ingest
Join
Subscribed/ Load
Landing
area
Clean
pulled data
Transform
Service Data backend
SQL endpoint
33. 5
Report
service
Views
2 Raw
data
Load
DW/ODS
1 service
4
2
3
Transforma>on
service
Landing
1
34. 3
Report tool
Drill through
7
REST API 5
9 6
8 10
HBase 2 Aggregation 4
map/reduce
2
1
ETL Details Aggregates
Raw data (map/reduce
(HDFS) +ETL) Data mart