Modernize Customer Data with AWS ML

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Hoang Nguyen
Solutions Architect, Amazon Web Services
Thông tin chuyên sâu về khách hàng và
Machine Learning (Cấp 200 – 300)
Tìm hiểu khách hàng của bạn – Cấu trúc
dữ liệu hiện đại

Summit Webinar Edition | Vietnam
Topic
Thông tin chuyên sâu về khách hàng và Machine Learning (Cấp 200 – 300) | Tìm hiểu khách hàng của bạn – Cấu trúc dữ liệu
hiện đại
Hãy thay đổi! Chuyển đổi sang AWS (Cấp 200) | Chuyển đổi & hiện đại hóa các ứng dụng Microsoft truyền thống với
container
Chuyển đổi sang AWS (Cấp 200) | Quản lý dự án chuyển đổi DB – Quy tắc thực tiễn tốt nhất

Ask the AWS Experts
Our Experts are online to answer any questions you have during the
presentation.
Ask your questions via the Questions Window on the GoToWebinar Control
Panel

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Let’s go on a common customer journey

Meet EarEcstasy, as they move from B2B to B2C
* This case is representative of a common customer journey, but EarEcstasy isn’t an actual business
EarEcstasy manufacturers headsets. They ran
a traditional B2B business since 2005, selling
through distribution and retail channels.
2005
In 2018, they launched their first “Smart
Buds”. These wireless headsets have voice
enablement, GPS tracking, and heartrate
monitors built in, and the device syncs with
the users mobile phone via Bluetooth. The
mobile app also supports scene detection.
2018

EarEcstasy needs to answer new questions and move faster
Raymond, Head of ProductLim, Head of Finance
Which regions are the new earbuds selling well?
What is the demand forecast by product category?
What is the social sentiment about our products?
How do quality issues impact cost of production?
Can I look at supplier performance over time?
How can we reduce our inventory holding costs?

To answer new questions quickly, we look to a
modern data architecture design
Massive upfront costs
Overprovisioned capacity
Long implementation times
Pay as you go, for what you use
Decoupled pipelines and engines
Experimentation platform
Ingest/
Collect
Consume/
visualize
Store Process/
analyze
1 4
0 9
5

Outcome 1: Modernize and consolidate

Start with a set of specific questions to answer, then work
backwards to the data required
Lim, Head of Finance
How do quality issues impact cost of production?
Can I look at supplier performance over time?
How can we reduce our inventory holding costs?
Order History /
Returns (CRM)
Inventory /
Production (ERP)

Ingest ServingData
sources
Modern data architecture
Insights to enhance business applications, new digital services
Transactions
ERP
Data analysts
DATA PIPELINES
Ingest/
Collect
Consume /
visualize
Store Process /
analyze
1 4
0 9
5

Start small and iterate

Ingest ServingData
sources
Transactions
ERP
DATA PIPELINES
Data
Lake
expdp
Data Data analysts
Data Warehouse
Amazon Redshift
Direct Query
Amazon Athena
She asks for the SMALLEST amount of data to answer her questions.
If it isn’t good enough, she asks for another small slice to be loaded to the DATA LAKE

Amazon Redshift – Modern Data Warehousing
Fast, scalable, fully managed data warehouse at 1/10th the cost
Massively parallel, scales from gigabytes to exabytes
Queries data across your Redshift data warehouse and Amazon S3 data lake
Fast at scale
Columnar storage
technology to improve I/O
efficiency and scale query
performance
Open file formats
Analyze optimized data
formats on direct-attached
disks, and all open file
formats in S3
Cost-effective
Start at $0.25 per hour;
as low as $250-$333 per
uncompressed terabyte
per year
$
Secure
Audit everything; encrypt
data end-to-end; extensive
certification and compliance

Characteristics of a Data Lake
Future
Proof
Flexible
Access
Dive in
Anywhere
Collect
Anything

Raymond, Head of Product
Which regions are the new earbuds selling well?
What is the demand forecast by product category?
What is the social sentiment about our products?
Trending /
Mentions (Social)
Order History /
Returns (CRM)
NOW IN THE DATA LAKE

Experiment, validate, then scale

Ingest ServingData
sources
DATA PIPELINES
Data
Lake
He first looks to the DATA LAKE, and imports only the category data he needs
He imports JUST ENOUGH data to see if the market is responding to products.
Business users
Transactions
ERP
Social media
Data
Stream
Capture
Amazon
Kinesis
Events
Amazon
QuickSight
Data Warehouse
Amazon Redshift
Stream Data
Amazon
ElasticSearch

Common data pipeline configuration
Raw Data
Amazon S3
Highly decoupled configurations scale better, are more fault tolerant, and cost optimized
ETL (Hadoop)
Amazon EMR
Triggered Code
Amazon Lambda
Staged Data
(Data Lake)
Amazon S3
ETL & Catalog Management
AWS Glue
Data Warehouse
Amazon Redshift
Triggered Code
Amazon Lambda

Data security
and management
Encryption
Access Controls
Monitoring and Metrics
Audit Trails
Automation
Serverless Computing
Data Discovery and
Protection
Data Visualization
Data movement
Physical Appliances
Hybrid Storage
Private Networks
File Data
WAN Acceleration
Third-party Applications
Streaming Data
Complete set of building blocks
FileBlock
Object Archival
Storage types

Ingest ServingData
sources
Transactions
ERP
Data analysts
Business users
DATA PIPELINES
EVENT PIPELINES
Data
Event
Insights
Data
Lake
Social media

Outcome 2: Innovate for new revenues

EarEcstasy has its first direct relationship with consumers
Krzysztof, Data ScientistBala, Head of Marketing
What are our customer segments, based on usage?
Can predict music preference from location and HR?
Are there additional signals in the voice commands?
Can we infer user activity, from scenes in pictures?
How are people using the Smart Buds?
How to understand what they listen to and when?
What kinds of people are in/decreasing usage?

Bala, Head of Marketing
How are people using the Smart Buds?
How to understand what they listen to and when?
What kinds of people are in/decreasing usage?
Media
consumption
(Partner API)
Registration,
usage [time/place]
(Mobile app)

Krzysztof, Data Scientist
What are our customer segments, based on usage?
Can predict music preference from location and HR?
Are there additional signals in the voice commands?
Can we infer user activity, from scenes in pictures?
HR, Voice, GPS,
Images (Device
data)
DATA LAKE, OR NOT?
Registration,
usage [time/place]
(Mobile app)
LOAD TO DATA LAKE

Sandboxes - fast, cheap, low risk

Ingest ServingData
sources
Transactions
Data scientists
Business users
Connected
devices
DATA PIPELINES
EVENT PIPELINES
Data
Event
Insights
Data
Lake
Sandbox
ML / Analytics / DLWeb logs /
clickstream

Ingest ServingData
sources
Innovate for new revenues - personalization and forecasting
Transactions
ERP
Data analysts
Data scientists
Business users
Connected
devices
DATA PIPELINES
EVENT PIPELINES
Data
Event
Insights
Data
Lake
ML / Analytics
Social media
Web logs /
clickstream

Outcome 3: Real-time engagement

EarEcstasy offers a personalized life soundtrack
Personalized, based on
past preferences,
people with similar behaviors,
and environments detected

Use EarEcstasy voice enablement to play music
I’m tired, play me
some music!
Amazon Transcribe
/ Comprehend
Action: PLAY
Category: MUSIC
Genre: <RECOMMEND>
Request content
HISTORY
Twenty One Pilots!
PEOPLE LIKE YOU
Amazon Kinesis
Streams
Connected device data
Location: <FIND GPS>
Mood: <FIND HR>

Use the mobile app to take a picture to identify
activity
A QUIET OFFICE
Amazon SageMaker
Image Classification
Amazon Rekognition
Image
CHAIR
LAPTOP
LAMP
DESK
97%
95%
88%
82%
Object Identification
WORKING!
<HISTORY>

Ingest ServingData
sources
Real-time engagement and interactive customer experiences
Transactions
ERP
Data analysts
Data scientists
Business users
Engagement platformsConnected
devices
Automation / events
DATA PIPELINES
EVENT PIPELINES
Data
Event Action
Insights
Data
Lake
ML / Analytics
Predict /
Recommend
AI Services
Social media
Web logs /
clickstream

Business Outcomes on a Modern Data Architecture
Outcome 1 : Modernize and consolidate
• Insights to enhance business applications and create new digital
services
Outcome 2 : Innovate for new revenues
• Personalization, demand forecasting, risk analysis
Outcome 3 : Real-time engagement
• Interactive customer experience, event-driven automation, fraud
detection

Ready to build better business from your ideas?
Short list projects that
directly impact
customer engagement
and adoption
Build simple data
pipelines that allow you
to test new ideas, and
fill your data lake
Ask our solution architects
and professional services
teams to help you build

Hoang Nguyen
Hãy thay đổi! Chuyển đổi sang AWS (Cấp 200)
Chuyển đổi & hiện đại hóa các ứng dụng
Microsoft truyền thống với container

VM
Container
Who is this new kid in town?

Server
Host OS
Hypervisor
Server
Host OS
VM Containers
Guest OS
Lib,bin
App
VM 1
C 1
AppGuest OS
Lib,bin
App
VM 2
C 2
App
C 3
App
C 4
App
C 5
App
C 6
App
Common bin,lib Common bin,lib
OS Image 1 OS Image 2
Docker
Images
Running
Containers
2 Apps
Vs
6 Apps

Why do we care?
https://pixabay.com/en/baby-boy-child-childhood-computer-84626/

Customers
AWS cloud
Identity Federation
Active
Directory
Microsoft
Technologies
on AWS
AWS Tools for Windows PowerShell
MSFT SCVMM Plug-in
2008 Today
Innovation
CustomerAdoption
WS 2008 & SQL Server 2008
WS 2003
.NET SDK
WS 2008 R2
SQL Server 2008 R2
Visual Studio
Toolkit
EC2 Dedicated Instances
(BYOL)
MSFT SCOM
plug-in release
Amazon RDS adds SQL Server
WS 2012 & SQL Server 2012
EC2 Run Command
EC2 Dedicated Hosts (BYOL)
MSFT SharePoint 2016 (Marketplace)
AWS Directory Service
WS & SQL 2016
EC2 Systems Manager
.NET on Lambda
Buy license included
instances
Bring your own licenses
Licensing

Windows Instance Example
m4.4xlarge -> $1.736/hr (SIN)
m4.xlarge -> $0.434/hr (SIN)
CPU Utilization ~ 15%
Change Instance Type
CPU Utilization
~60%
Still has ~40% head
room
75%
Savings

Bring Your Own License
(BYOL) – Windows Server
Running two m4.4xlarge instances
~2500 USD
On Demand
~1600 USD
3 Year Reservation
36% Savings
~1600 USD
3 Year Reservation
~800 USD
Dedicated Host
Reserved Instance
50% Savings

m4.4xlarge
25% Utilization
m4.large
50%
Utilization
~1200 USD ~800 USD
m4.large
50%
Utilization
m4.large
50%
Utilization
m4.large
50%
Utilization
33% Savings
m4.4xlarge
75% Utilization
How can I isolate?

So far what we found
Reserved instances
Bigger instances
Right sizing
Easy deployment
Easy Patching
Isolation
Increase
Utilization
Savings
Headache Free IT

Slicing & Isolating Resources
12am 1am 2am 3am 4am 5am 6am 7am 8am 9am 10am 11am
Job 1
Job 2
Job 3

Slicing & Isolating Resources
docker run -d --cpu-percent 10 mycompany/myapp c:AppBatch.exe
12am 1am 2am 3am 4am 5am 6am 7am 8am 9am 10am 11am
Job 1
Job 2
Job 3

Docker Images & Layers
Application 1 Application 2
Layer 110GB
Layer 22GB
Layer 350MB
Layer 410MB
Layer 320MB
Layer 45MB
Shared Layers

Patching and Maintenance
My application 1 layer
version 10
version 200
My application 2 layer
ASP.NET:latest
Windows Server:latest

CC https://pixabay.com/en/container-port-loading-stacked-3118783/

Set of containers
E.g. SQL server, Web sites
instance instance instance instance
Cluster
Constrains
E.g. HR, Finance, Instance Size
Resource Demand
E.g. Memory, CPU
Task

Cluster
Task
Service
How many task?
Deployment
Strategy?
Auto scaling
Strategy?

Amazon ECS Amazon ECS
Example Deployment
Bin Packing Balance Spread

Cluster
Service 1
Service 2
Service 3
Service N

CI/CD Pipeline
Microsoft
Visual Studio
Application
Load Balancer
v 1
v 1
container
container
Microsoft
TFS
Build Agent Amazon
ECR
Amazon ECS
container
container
v 2
v 2
Security Checks
Unit Test

What we learnt 1/3
Containers
CC https://pixabay.com/en/container-port-loading-stacked-3118783/

What we learnt 2/3
Isolation
CC https://pixabay.com/en/horse-barn-the-horses-are-stallion-2649609/

What we learnt 3/3
Automation
https://commons.wikimedia.org/wiki/File:KUKA_Industrial_Robots_IR.jpg

Thank You
You will receive today’s webinar recording and presentation deck,
look out for it in your inbox.

Hoang Nguyen
Chuyển đổi sang AWS (Cấp 200)
Quản lý dự án chuyển đổi DB – Quy tắc thực
tiễn tốt nhất

What to Expect from the Session
• Database Migration Context
• Database Migration Tools
• Introduction to the AWS Migration Framework
• Database Migration Effort
• Customer References
• Next Steps

Database Migration Context

© 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Amazon
S3
EC2
Amazon
EFS
Amazon
EBS
Aurora Amazon
EMR
Amazon Glacier
Amazon
RDS
IoT
Amazon
Redshift
DATA
Migrating Data: Five Key Questions
1) What kind of data is it, and where is it going?
4) How much data & time do you have?
2) One-time or continuous movement?
3) One-way or bidirectional access?
5) How might your WAN be a factor?
Files Block Volumes Databases IoT Devices

Amazon RDS Engines
CommercialOpen sourceAurora

Amazon DynamoDB
F a s t a n d f l e x i b l e N o S Q L d a t a b a s e s e r v i c e f o r a n y s c a l e
Fast, consistent
performance
Highly scalable Fully managed Business critical
reliability
Consistent single-digit
millisecond latency; DAX in-
memory performance reduces
response times to microseconds
Automatic scaling to
hundreds of terabytes of
data that serve millions
of requests per second
Automatic provisioning,
infrastructure
management, scaling,
and configuration with
zero downtime
Data is replicated across
fault-tolerant Availability
Zones, with fine-grained
access control
The
picture
can't be
displayed.

© 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Migration Was Costly, Complex, & Slow
6<
1001001
0101001
1000100

Database Migration Tools

AWS Database Migration Service (AWS DMS)
DMS migrates databases to AWS easily and
securely with minimal downtime. It can migrate
your data to and from most widely used
commercial and open-source databases.
Amazon Aurora
S3 Bucket
DynamoDB

Customer
Premises
Application Users
AWS
Internet
VPN
Start a replication instance
Connect to source and target
databases
Select tables, schemas, or
databases
Let AWS DMS create tables,
load data, and keep them in
sync
Switch applications over to the
target at your convenience
Keep Your Apps Running During the Migration
AWS
Database Migration
Service

AWS Schema Conversion Tool (AWS SCT)
SCT helps automate many database schema and
code conversion tasks when migrating between
database engines or data warehouse engines
Amazon Aurora

SCT can tell you how hard the migration will be
1. Connect SCT to
Source and Target
databases.
2. Run Assessment
Report.
3. Read Executive
Summary.
4. Follow detailed
instructions.

When to use DMS and SCT?

Tools for Migration Project Phases
Phase Service/Tool Notes
Assessment AWS Schema Conversion Tool
Reports on the database objects, complexity and types of
migration issues
Schema Migration AWS Schema Conversion Tool
Copies a schema or migrates a schema depending on
whether it is a homogeneous or heterogeneous
migrations
Data Migration
AWS Database Migration Service
AWS Schema Conversion Tool
Bulk load and change data capture (CDC) options
Extraction and load for large data warehouses, including
AWS Snowball integration
Application Migration AWS Schema Conversion Tool SQL statement migration in application code
Data Validation AWS Database Migration Service Ensure data is the same on source and target
Functional Testing Various Tools on Marketplace Ensure the application runs as intended
Performance Testing Various Tools on Marketplace Ensure the application performance as intended

Tools for Migration Scenarios
Scenario Example Recommendation
Homogeneous
migration to the same
database version and
edition
Migration of Oracle Database
11gR2 Enterprise Edition from
on-premise to EC2
Use the native replication technology to
create a standby database and then failover
to the standby database
Homogeneous
migration to a different
version
Migration of MySQL 5.5 to
MySQL 5.7
AWS Schema Conversion Tool and AWS
Database Migration Service
Homogeneous
migration to a different
edition
Migration of SQL Server
Enterprise Edition to Standard
Edition
Heterogeneous
migration
Migration from Oracle
Database to PostgreSQL

That’s the tools, but how to
manage migration projects?

Introducing the AWS Migration Framework

EXECUTE
AWS Migration Framework
• Project Control
⎼ Strategy (business driver)
⎼ Key Stakeholders and
Team
⎼ Plan (Scope, Schedule,
Resources)
⎼ Cost Estimation
• Portfolio discovery
• Migration plan
• Operations Integration
• Security
• Prioritized Backlog
⎼ Application groups
⎼ Migration strategy
⎼ Success criteria
• Ops Integration –
Foundation and Landing
Zone (target zone setup)
• Setup Factory (Tools,
Teams, Processes)
• Pilot Migration
Discover
Design
Build
Integrate
Validate
Cutover
Prioritized
Backlog
(PLAN)
• Application optimization
• Process optimization
• Operational optimization
• Cost optimization
READINESS AND
PLANNING
ACTIVATE
OPTIMIZE

AWS Migration Framework-
Readiness and Planning

AWS Migration Framework - Readiness &
Planning
• Project Control
⎼ Strategy (business
driver)
Team
Resources)
⎼ Cost Estimation
• Migration plan
• Security
READINESS AND
PLANNING
Project Control focuses on ensuring there is a migration
strategy in place that is supported by key stakeholders in
the organisation. Additionally, we look at defining the team
that will carry out the work, with associated timelines and
cost estimations.
Sample decision points:
• Who is the executive sponsor?
• Are there any compelling events that will affect the
migration strategy?
• Do we have the right resources? How are they organized?
• What are the timeframes we are working with?
• Do we have the necessary budget?

AWS Migration Framework - Readiness &
Planning
• Project Control
⎼ Strategy (business driver)
Team
Resources)
⎼ Cost Estimation
• Migration plan
• Security
READINESS AND
PLANNING
Quickly understand which applications are Cloud Eligible,
Cloud Friendly, or Cloud Native and then execute a dive
deep analysis on just that subset of applications.

Application Assessment
• Business driver and intended ROI?
• Migration sponsor (business owner, C-level)?
• ISV application? Does the ISV support the target?
• Maintenance window for the migration?
• Design documentation?
• Original developers/DBAs still available?

Database Assessment
• How many database objects (tables, triggers, SPs, users, etc.)?
• How much data?
• Complexity of the SPs and triggers?
• Proprietary DB features?
• Non-standard or custom data types?
• Character set conversions?
• Time zone or UTC?
• User authentication method?
• Licensing mechanism (cores, users, ULA etc.)

Application Technical Assessment
• Database Access:
• SQL statements throughout the code?
• Calls to a data abstraction layer?
• API calls?
• ANSI SQL used where possible?
• SQL complexity, e.g. analytics with many joins or simple CRUD?
• Number of lines of SQL code?
• Application access, e.g. LDAP, DB Users, etc.

The 6Rs of Migration Planning
Discover, Assess &
Prioritize Applications
Use Migration Tools
Transition Production
Redesign Application/
Infrastructure Architecture
App Code
Development
Purchase COTS/
SaaS & licensing
Validation
Modify underlying
Infrastructure
Full ALM / SDLC
Manual Config
Manual Deploy
Manual Install
Determine
Migration Path
Automate
Manual Install & Setup
Integration
Determine
new platform

AWS Migration Framework -
Activate

AWS Migration Framework - Activate
• Determine your application priorities and group
integrated applications together
• Outline the success criteria for each application
migration
• Create your AWS landing zone (accounts, VPC, subnets,
IAM roles, VPN/Direct Connect, etc.)
• Configure DMS, SCT and other migration tools
• Team creation
• POC/pilot
• Prioritized Backlog
⎼ Application groups
⎼ Migration strategy
⎼ Success criteria
• Ops Integration –
Foundation and Landing
Zone (target zone setup)
• Setup Factory (Tools,
Teams, Processes)
• Pilot Migration
ACTIVATE

Building a Migration Team
Application architect/developer: Application expert who can identify
what components are important, complex, redundant, etc.
Source DBA: Knows the database design, schema, features used and
what must be migrated to the target.
Target DBA: An expert in the target database to help map features
from the source DB with the Source DBA.
AWS Solution Architect: Determines the correct target architecture in
AWS and is familiar with DMS/SCT.
Application/Database Developers: Customer and/or partner
resources to migrate the stored procedures, triggers and application
code.

Hiring and Developing Talent
New skills are needed for the target DB and often AWS if
migrating from on-premises
Develop training plans for existing employees
Hire in required skills if necessary
Retrain, redeploy or make people redundant who’s skills
are no longer relevant

Pilot/POC
Choose a reasonably complex module/component to
migrate to validate your assumptions in the Activate phase
You should:
• Obtain more accurate migration assessments
• Determine what can be automated
• Learn how the migration tools behave (limitations, bugs,
improvements needed)
• Learn what skills are missing from your team

Execute

AWS Migration Framework - Execute
• Always have a back up plan!
• Execute according to lessons from Pilot/POC
• Typically the same amount of time to migrate the
DB as to migrate the application (assuming DAL)
• Determine how you will cutover
• Parallel run: expensive and difficult
• Minimal downtime: DMS+CDC
• Large maintenance window: application and
data verification before go live
EXECUTE
Discover
Design
Build
Integrate
Validate
Cutover

Optimize

AWS Migration Framework - Optimize
• DMS instance and task optimization
• Database tuning
• Database instance right sizing
• Application tuning
• Application instance right sizing
• Use EC2 and RDS stop/start to optimize costs
• Purchase reserved instances and use spot
instances
• Look for contention and evaluate caching, NoSQL,
federation and adoption of a microservices
strategy
• Perform HA/DR scenarios and optimize the use of
AWS managed services to help, e.g. RDS MAZ,
Auto-scaling
• Application optimization
• Process optimization
• Operational optimization
• Cost optimization
OPTIMIZE

Migration Effort

Database migration – multi phase process
Phase Description Automation Effort (%)
1 Assessment SCT 2
2 Database Schema Conversion SCT/DMS 14
3 Application Conversion/Remediation SCT 25
4 Scripts Conversion SCT 7
5 Integration with 3rd party applications 3
6 Data Migration DMS 4
7 Functional testing of the entire system 29
8 Performance tuning SCT 2
9 Integration and deployment 7
10 Training and knowledge 2
11 Documentation and version control 2
12 Post production support 3

Database Migration Process
Conversion
Data migration

Oracle to Aurora Migration Playbook
• Topic-by-topic overview of Oracle to Aurora
PostgreSQL migrations and “hands-on” best
practices
• How to migrate from proprietary features and
the different database objects
• Migration best practices
SCT DMS Playbook
Schema Data Best practices
https://aws.amazon.com/dms/getting-started/

Customer / Partner References

>50,000 Databases Migrated with DMS

Expedia migrated from SQL Server to AWS

Trimble migrated from Oracle to
Amazon RDS PostgreSQL

Next Steps

Next Steps
• Talk to your AWS account team and AWS Partner
• Ask us about funding for POCs and commercial DB
migrations (e.g. Oracle Database to Aurora)
• Read Documentation, White Papers, Playbooks
• Links:
• DMS & SCT: https://aws.amazon.com/dms/
• Getting Started Guides and Playbooks:
https://aws.amazon.com/dms/getting-started/

Modernize Customer Data with AWS ML

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Semelhante a Modernize Customer Data with AWS ML

Semelhante a Modernize Customer Data with AWS ML (20)

Mais de Amazon Web Services

Mais de Amazon Web Services (20)

Modernize Customer Data with AWS ML