ApacheCon 2015 - A Stock Prediction System Using OSS
•
3 gostaram•743 visualizações
Finance market prediction has always been one of the hottest topics in Data Science and Machine Learning. However, the prediction algorithm is just a small piece of the puzzle. Building a data stream pipeline that is constantly combining the latest price info with high volume historical data is extremely challenging using traditional platforms, requiring a lot of code and thinking about how to scale or move into the cloud. This session is going to walk-through the architecture and implementation details of an application built on top of open-source tools that demonstrate how to easily build a stock prediction solution with no source code - except a few lines of R and the web interface that will consume data through a RESTful endpoint, real-time. The solution leverages in-memory data grid technology for high-speed ingestion, combining streaming of real-time data and distributed processing for stock indicator algorithms
Recomendados
Recomendados
Mais conteúdo relacionado
Mais procurados
Mais procurados (20)
Semelhante a ApacheCon 2015 - A Stock Prediction System Using OSS
Semelhante a ApacheCon 2015 - A Stock Prediction System Using OSS (20)
Último
Último (20)
ApacheCon 2015 - A Stock Prediction System Using OSS
- 2. A Stock Prediction System using open-source software Fred Melo fmelo@pivotal.io @fredmelo_br William Markito wmarkito@pivotal.io @william_markito
- 4. It's all about DATA Data Sources Look for patterns Prediction
- 7. Machine Learning is the answer Neural Networks Clustering Genetic Algorithms
- 8. Train with historical dataset Apply model to the new input Applying Machine Learning
- 9. Hard to add new data sources Why? Hard to scale Why so hard? Hard to make it real-time
- 10. HDFS Data Lake Store Analytics Hard to change Labor intensive Inefficient No real-time information ETL based Data-source specific Traditional models are reactive and static
- 11. HDFSData Lake Expert System / Machine Learning In-Memory Real-Time Data Continuous Learning Continuous Improvement Continuous Adapting Data Stream Pipeline Multiple Data Sources Real-Time Processing Store Everything Stream-based, real-time closed-loop analytics are needed
- 12. Info Analysis Look at past trends (for similar input) Evaluate current input Score / Predict Neural Network How can it be addressed?
- 13. Info Analysis Filter [ json ] Neural Network How can it be addressed?
- 14. Info Analysis Filter Enrich Neural Network How can it be addressed?
- 15. Info Analysis Neural Network Filter Enrich Transform How can it be addressed?
- 16. Info Analysis Filter Enrich Transform Neural Network How can it be addressed?
- 17. Info Analysis Filter Enrich Transform Transform Neural Network How can it be addressed?
- 18. Neural Network In-Memory Data Grid Real-time scoring How can it be addressed? Train
- 19. Neural Network In-Memory Data Grid Front-end Update Push How can it be addressed?
- 20. Ingest Transform Sink SpringXD Store / Analyze Fast Data Distributed Computing Predict / Machine Learning Other Sources and Destinations JMS Streaming real-time analytics architecture
- 21. Transform Sink SpringXD Extensible Open-Source Fault-Tolerant Horizontally Scalable HTTP Machine Learning Fast Data Filter Predict Sink HTTP Split Dashboard Push Demo Architecture
- 23. SpringXD INGEST / SINK PROCESS ANALYZE • Little or no coding required • Dozens of built-in connectors • Seamless integration with Kafka, Sqoop • Create new connectors easily using Spring • Call Spark, Reactor or RxJava • Built-in configurable filtering, splitting and transformation • Out-of-box configurable jobs for batch processing • Import and invoke PMML jobs easily • Call Python, R, Madlib and other tools • Built-in configurable counters and gauges Data Stream Pipelining
- 24. SpringXD XD NodesXD NodesXD NodesXD Nodes Ingest SpringXD Split Filter Transform Sink XD admin XD Nodes Ingest Split Filter Transform Sink Stream Deployment Messaging Scale-Out and HA Architecture
- 25. Transform Sink SpringXD Extensible Open-Source Fault-Tolerant Horizontally Scalable HTTP Machine Learning Fast Data Filter Predict Sink HTTP Split Dashboard Push Demo Architecture
- 28. Event handling
- 29. Transform Sink SpringXD Extensible Open-Source Fault-Tolerant Horizontally Scalable HTTP Machine Learning Fast Data Filter Predict Sink HTTP Split Dashboard Push Demo Architecture
- 30. Neural Networks
- 31. Neural Networks
- 32. medium avg (x+1) relative strength (x) medium avg (x) price(x) Neural Network
- 33. Neural Network
- 34. Transform Sink SpringXD Extensible Open-Source Fault-Tolerant Horizontally Scalable HTTP Machine Learning Fast Data Filter Predict Sink HTTP Split Dashboard Push Demo Architecture
- 36. Demo Time
- 40. Follow-up: In-Memory Unconference "A place for all things in-memory: projects, people, ideas, roadmaps, discussions." Location: Hill Country A/B” Weds 4:15pm - 6pm. (after this talk) The demo code is on GitHub! @fredmelo_br @william_markito