Abundant data is all around. The most important aspect is how you as an organization can access the data, process it, and present information to the relevant authorities on time. To gain competitive advantage the means of accessing, processing and presenting the data should be optimal, highly available and scalable.
In this talk, we will discuss how you can leverage WSO2 Data Analytics Server, WSO2 IoT Server, WSO2 Enterprise Service Bus and other WSO2 products in order to analyze the data. We will also discuss different deployment patterns that can provide you with a suitable solution that lets you analyze relevant data historically, in real-time or interactively and predict future states to make better decisions for your organization’s success.
8. Collect Data Internally
• Don’t worry about
– Data formats
– Data sources
– Platforms
– Protocols
Start with WSO2 DAS
it has a unified data capturing framework !
11. Deployment for Data Analytics
Batch & Interactive Analytics
• Enable Searchability
– Full text data
– Drill down search
• See what has happened
– Summarise the Data
– Understand patterns and behaviors
12. Deployment for Data Analytics
Batch & Interactive Analytics
• Enable Searchability
– Full text data
– Drill down search
• See what has happened
– Summarise the data
– Understand patterns and behaviors
• Simple Deployment
– 2 Nodes
– Use RDBMS to store the data
13. Deployment for Data Analytics
Batch & Interactive Analytics
2 Node
Deployment
15. Deployment for Data Analytics
Realtime Analytics
• Keep informed
– Dashboard
– Alerts
– Feedback loops
16. Deployment for Data Analytics
Realtime Analytics
• Keep informed
– Dashboard
– Alerts
– Feedback loops
• High Availability
– Zero downtime
– Zero data loss
18. Deployment for Data Communication
Alerting & Communicating
Legacy & Internal
Services
19. Realtime + Batch Analytics
• Filter Data before you store
– Realtime → Store & Process
• Summarize and store
– Realtime → Store & Process
• Cross check with history
– Lambda Architecture
– Graph with Batch & Realtime
• Alerts based on batch processing
– Batch → Realtime
From Batch
From Realtime
24. Deployment for Data Collection ...
From 3rd Party Apps & Cloud
HTTP
Utilize API Analytics !
25. Analyse Business with API Analytics
• APIs involved
• Who invokes the APIs
• Extract business information from
– Payloads
– Resources URIs
Monetize APIs !
27. Scaling Analytics Deployment
The Changes !
• Realtime
– Supported by Apache Storm
• For High Memory Requirement or CPU Intensive Processing
– No query changes
• Batch
– Move from RDBMS to HBase/Cassandra
• WSO2 DAS have a Data Abstraction Layer
• Independent of underlying Data Store
Seamless migration :)
33. Deployment for Data Collection
From Sensors
Analytics on the Edge
with WSO2 Siddhi
Push
34. Deployment for Data Communication
Mobile & 3rd Party Apps
● Expose analytics results
as API
○ Mobile Apps, Third Party
● Provides
○ Security, Billing,
○ Throttling, Quotas & SLA
● How ?
○ Write data to database from DAS
○ Build Services via WSO2 Data Services Server or use Analytics
REST API
○ Expose them as APIs via WSO2 API Manager
35. Analytics Life Cycle
Predefined analytics
• Artifacts bundled as CApps to and moved
Dev → Test → Preprod → Prod
Analytics on Production Environment
• Interactive Analytics
• Personalizing Dashboards
• Customised Alerts
36. Summary
• Start small and scale as you grow
• Minimum HA Deployment
– 2 Nodes
• Fully Distributed Deployment
– 8+ Nodes
– Scale based on need, horizontally and vertically
• Analyser, Indexer, Receiver,
Realtime (With Apache Storm), Dashboard
37. In God we trust;
all others must bring data
- William Edwards Deming -