In this workshop, learn how to use machine learning to analyze the Deutsche Börse Public Dataset, which consists of trade data aggregated to one-minute intervals from the Tradex and Eurex engines, comprising a variety of equities, funds, and derivative securities. The public dataset provides initial price, lowest price, highest price, final price, and volume for every minute of the trading day, and for every tradeable security. Learn how to apply a variety of ML models to the data to find patterns and methods to predict price movements or identify trends in the market. Also learn how to interact with data staged for analysis in Amazon S3, use AWS Glue to transform data into analysis-ready formats, and use Amazon SageMaker and Amazon EC2 to experiment with a variety of ML models to derive insights from the data.

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Market Prediction Using ML
Experiment with Amazon SageMaker and the Deutsche Börse Public Dataset
Binoy Das
Partner Solution Architect
Amazon Web Services
F S V 3 0 9
Dave Rocamora
Partner Solution Architect
Amazon Web Service

3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Our mission
Put machine learning
in the hands of every
developer,
data scientist, and
architect

4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Agenda
 Overview of Amazon SageMaker
 Overview of Deutsche Börse Public Data Set
 Overview of Amazon Athena and Amazon QuickSight
 Hands-on—data preparation
 Hands-on—data analysis
 Hands-on—custom container on Amazon Elastic Container
Registry (Amazon ECR) to implement RNN model
 Hands-on—Use DeepAR on Amazon SageMaker

5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
The Amazon Machine Learning stack

7. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon SageMaker
1
I
Notebook instances
2
I
Algorithms
3
I
ML training service
4
I
ML hosting service

8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
End-to-end machine
learning platform
Zero setup Flexible model
training
Pay by the second
Amazon SageMaker—Value proposition
The quickest and easiest way to get ML models from idea to production
$

9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Stock market data maintained by Deutsche Börse
• Opening, closing, highest and lowest price, and trading
volume for every security traded
Published on Registry of Open Data on AWS
https://registry.opendata.aws/deutsche-boerse-pds/

11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Deutsche Börse Public Data Set
Source:
s3://deutsche-boerse-xetra-pds (for Xetra data)
s3://deutsche-boerse-eurex-pds (for Eurex data)
Data dictionary:
Field Description Type
Mnemonic Stock exchange ticker symbol Text
Date Date of trading period Date
Time Minute of trading to which this entry relates Time
StartPrice Trading price at the start of period Decimal
MaxPrice Maximum price over the period Decimal
MinPrice Minimum price over the period Decimal
EndPrice Trading price at the end of the period Decimal
TradedVolume Total value traded Decimal
NumberOfTrades Number of distinct trades during the period Integer

12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Serverless data lake & analytics with AWS
Crawlers scan your data sets and populate the AWS Glue Data Catalog
The AWS Glue Data Catalog serves as a central metadata repository
Once catalogued in AWS Glue, your data is immediately available for analytics
1
2
3
Amazon
Redshift
Spectrum
Amazon
EMR
Amazon
Athena
AWS Glue
Data Catalog
AWS Glue
crawler
Amazon S3 Amazon
QuickSight
1 2
3

14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Visualization for a trading analyst

15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Begin with a notebook
• First step: create lifecycle configurations
• Clone from GitHub
• Create folder structures
• Obtain cleaned up data
https://github.com/aws-samples/db-reinvent-workshop

17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Explore stock behavior
• Generate database
• Create AWS Glue crawler
• Create tables
• Queries for introspection
• Create Athena queries
• Create custom views
• Stock visualization
• Create Amazon QuickSight dashboard
• Explore similarities in behavior

19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Clustering of similar stocks
• Clustering algorithm
• Run custom clustering algorithm
• Find correlation
• Group clustered stocks
• Retrieve metrices for clustered stocks
• Create separate data files for clusters

20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Containerize custom MLP code
• Data channels
• Peek into data
• Load to Amazon SageMaker S3 bucket
• Container image
• Bundle code and libraries
• Create Amazon ECR repository
• Model training
• Create Amazon SageMaker estimator
• Fit estimator to the data

22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Deploy custom MLP model
• Model deployment
• Create Amazon SageMaker predictor
• Deploy to Amazon SageMaker endpoint
• Inference generation
• Generate prediction with test data
• Visualize prediction against observed
• Performance evaluation
• Measure accuracy
• Scope for improvement

23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Train DeepAR model
• DeepAR data set
• Format clustered data for DeepAR
• Create training and test data
• Model training
• Create Amazon SageMaker training job
• Observe training behavior

25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Deploy DeepAR model
• Model deployment
• Create endpoint configuration
• Create Amazon SageMaker endpoint
• Inference generation
• Generate prediction with test data
• Visualize prediction against observed
• Performance evaluation
• Measure accuracy
• Repeat with different cluster

26. Thank you!
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Binoy Das
binoyd@amazon.com

27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.