AWS DataSync is a new online data transfer service that automates movement of data between on-premises storage and Amazon S3 or Amazon Elastic File System (Amazon EFS). In this session, we will introduce the service, showing how you can use DataSync to move active on-premises data to the cloud for one-time migration, timely in-cloud analysis, and replication for data protection and recovery. We’ll demonstrate how to get started with DataSync, and you’ll hear how it is helping Cox Automotive to move their archive of millions of images to AWS.

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
SFTP?

3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
New Launch: AWS Transfer for SFTP
Integrate Your SFTP Workflows with AWS
Asa Kalavade
General Manager
AWS Storage Gateway, AWS
Transfer for SFTP, and AWS
DataSync
S T G 3 2 6
Smitha Sriram
Senior Product Manager
AWS Storage Gateway & AWS
Transfer for SFTP
Ranga Rajagopal
Senior Director, Enterprise Data Platforms
FINRA

4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
• Why did we build AWS Transfer for SFTP?
• What is it: Benefits, feature, and pricing details
• How FINRA is making file exchanges easy with AWS SFTP
• Demo
• Deep dive
• Q&A
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Agenda

5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
SFTP: It’s here, it’s everywhere
Protocol is deeply embedded in workflows across a variety of industries
Financial services
$
Retail
Healthcare ..and more

6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
SFTP architecture on-premises today
Hundreds of SFTP
clients and apps
On-premises
- Operate infrastructure
- Host, manage, and
monitor
- Data stored on-premises
- Upfront costs (licenses)

7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Migrate SFTP workflows to AWS
To benefit from
• Durable storage in Amazon S3
• Processing, archival and retention of data
• Integration with AWS services
SFTP SFTP server and data
Will your hundreds of partners
change their workflows to
integrate with the cloud??

8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Self-hosted SFTP in the cloud
Hundreds of SFTP
clients and apps
VPC
- Lacks enterprise features
- No directory integrations
- Host, scale, and monitor
- Complex to use
Amazon S3

9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Transfer
for SFTP
Launched this week!

10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Transfer for SFTP
Fully managed SFTP service for Amazon S3
Seamless migration
of existing workflows
Native integration
with AWS services
Simple
to use
Cost-effective
Fully managed
in AWS Secure and Compliant

11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
In as simple as 3 steps
Your users can now use your AWS SFTP server endpoint to transfer data
Map your hostname Select your S3 bucket(s) Set up your users
1 2 3

12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Your SFTP architecture now
The same hundreds of users now
serviced by AWS Transfer for
SFTP, without requiring changes
+ Enterprise-ready
+ No end-user disruption
+ Fully managed servers
+ Simple to use
+ Pay as you use
+ Native cloud integrations
Amazon S3AWS SFTP
AWS Transfer
for SFTP

13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Seamless migration: The details
Migration to AWS Transfer for SFTP completely
transparent to end users
• Route existing SFTP domain to service endpoint using Amazon Route 53
• Continue to use same transfer clients
• Hostname and credentials stay the same
Integrate existing Identity providers
(Microsoft AD, LDAP) for end-user credentials

15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Time saved from managing SFTP servers
Automatically scales to meet your needs in real-time
Redundant across Availability Zones in Region
Fully managed: The details

16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Store data in your Amazon S3 buckets
for archiving, processing, or analyzing
Automate post-upload processing with Amazon S3
events
Control end user access to resources by using IAM
Encrypt your data using server-side encryption using
Amazon S3 or AWS KMS
Native AWS integrations: The details

17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
HIPAA-eligible and PCI-compliant
Encryption at rest options such as SSE-S3 or SSE-KMS
End-user activity tracking in Amazon CloudWatch
Secure and Compliant: The details

18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
SFTP endpoint fee @$0.30/hour
SFTP uploads and downloads @$0.04/GB of transfer
Examples of using the service to transfer
• 10 GB/day costs $2.7K/year
• 100 GB/day costs $3.6K/year
Cost-effective: The details

19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Easy to set up and configure using the
AWS Management Console or service API
No IT expertise required for SFTP
server or user access configuration
Simple to use: The details

20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Simple to use: The details

21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
File exchanges made easy,
with AWS Transfer for SFTP
Ranga Rajagopal
Senior Director, Enterprise Data Platforms

22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
• FINRA: Who we are
• Our on-premises challenges and cloud journey
• FINRA’s data lake in AWS
• On-premises SFTP challenges
• FINRA’s FileX platform powered by AWS Transfer for SFTP
• Next Steps
Our story

23. Confidential | Copyright 2018 FINRA 23
Tech Highlights
FINRA manages approximately
30 Petabytes
of storage
FINRA is dedicated to investor protection
and market integrity,
by regulating
3,700
securities firms and
630k+
brokers
FINRA processes and analyzes
99%
of US Equities market transactions &
65%
of options transactions
in AWS, looking for fraud, abuse, and
insider trading.
FINRA runs up to
50,000
compute nodes per days
FINRA processes up to
½ trillion
data validation checks per day
FINRA consumes up to
135 billion
market events and
6 terabytes
of data daily with AWS

24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Cloud Drivers
On-premises data and infrastructure pains
Data growth Infrastructure costs
Data governance questions Data management problems
Growing 20 to 30 percent YoY Too costly to build for peak; constant EoL cycles
Spend more on infrastructure or core mission?
How do we scale manage data at scale?
How can we run analytics despite fragmentation?
What do we have? Source? Versions? Retention?
Tracking 40M+ tables is not easy…

26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Scalable Data Management
Storage
Compute Scalable
Scalable
ETL Batch Analytics Interactive
Analytics
Catalogs
Golden Copies
Amazon S3 & Amazon Glacier
Performance Copies
Amazon S3
Data ValidationData
Manager
Metastore
8

28. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Where is my data? (Now)
All data is in Amazon S3 source of truth
One location of master data, security, versioning,
availability, cross-region data replication, and so on
Separation of storage from compute

29. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
FINRA usage statistics on AWS
 50k+ Amazon EC2 nodes per
day
 93 percent+ of Amazon EC2
usage is Amazon EMR-based
(mostly SPOT)
 30Pb+ storage (Amazon S3,
Amazon Glacier)
13
0
5000
10000
15000
20000
25000
30000
35000
40000
45000
1
2
3
4
5
6
7
40289 41770
40512
36589
33275
16023
8710
2145
2323
2542
2363
2363
1686
1590
231 231
231
231
231
231
231
Series1 Series2 Series3

30. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

31. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Usage
• 1000+ active users
• 6 million files transferred monthly
• 7 million Amazon S3 API calls monthly
On-premises architecture
• FTP farm with 6 servers in production
• 20 TB NAS Storage in production
• DR environment same as production
• Critical SLAs based on SFTP file submissions
3
SFTP @ FINRA

32. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
FINRA’s AWS Architecture
On-premises data center
NAS
FTPIncoming Files
Validation Data Management
Linkage
Data Analytics
Normalization Amazon
EC2
Amazon
S3
Amazon
Glacier
Amazon
Redshift
Amazon
EMR
VPC
Amazon
EMR
Amazon
RDS
Machine
Learning
AWS
KMS

33. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
• Multiple endpoints for customers
• Different flavors of implementation
• Security approach is inconsistent across different
implementations
• Multiple user accounts, multiple authorization mechanisms
• Not all implementations are configured for redundancy
• Multiple versions of COTS product with customizations
• Redundancy in operations
Challenges of managing file transfers on-premises

34. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Cloud-based SFTP service (our initial options)
Options Challenges
ROI was not very attractive1. In-house development
Challenges with maintaining FINRA AMIs2. Use the same on-premises COTS product
Several feature gaps3. Use AWS native third-party product

35. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

36. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
fileX architecture powered by AWS SFTP
AWS KMS
Bucket
File handling
FileX internal stack
Elastic IP
Amazon Route 53
Filex.finra.org
External M2M
External
FINRA
Data Lake
Elastic IP
BYOA
SFTP external stack
Public Subnets
API AWS Lambda Amazon ECS
AWS STS
Amazon ECS Amazon SNS
Amazon SNS Amazon SQS Amazon ECS
Amazon SQS
FINRAFirewall
Cross-zone NLB
AWS SFTP
Auth

37. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Reduced operational burden
• Goal=Less ops work; focus on business needs
• Fully managed=No manual labor, no infrastructure
management, no patching, etc…
• Automatic scaling
• Inherently meets our disaster recovery requirements with Multi-AZ
support
How FINRA benefits from AWS Transfer for SFTP

38. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
DevOps ❤ AWS-native
Native AWS service integrations helps us standardize
DevOps with a consistent management experience
So we ❤ AWS SFTP integrations with:
• Amazon S3 for our data lake
• AWS KMS encryption
• IAM
• API Gateway for BYO authentication
• Industry-standard AuthN and AuthZ support
Enables our single source of truth in Amazon S3 data lake
No more confusion about which version of a file from which data store
How FINRA benefits from AWS Transfer for SFTP

39. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Dev engagement and PoC testing
AWS Transfer for SFTP is helping us
reduce our operational burden, while
maintaining critical connection to our
existing authentication systems for
external users – so they don’t experience
any disruption as we complete our
migration of SFTP services to AWS
Early and extensive engagement
with AWS product management
and development teams

40. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Migrating SFTP workloads to AWS Transfer for SFTP by Q1,
2019
Extending FileX to support HTTPS
Build a standard file sharing platform
for external-facing FINRA customers
• Drag-and-drop uploads and downloads
• API support for external customers
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Next steps

41. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

42. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Service managed user access
Store and manage user identities and keys inside the service
1. Configure your users credentials and keys using the AWS Management Console
2. Users serviced using their existing clients and credentials
3. Amazon S3 accessed by using IAM during file transfers
AWS SFTP Amazon S3
1
2
3
AWS Transfer
for SFTP

43. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Plug in your custom identity provider (IdP)
If you have custom-built IdP, integrate by using Amazon API Gateway
1. Use API Gateway method to integrate your IdP
2. The service authenticates users using your IdP via API Gateway
3. The service assumes the IAM role to access bucket during file transfers
AWS Transfer
for SFTP
VPC
1
2
3
Amazon S3
AWS SFTP

44. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

45. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Set up
• Amazon S3 bucket
• AWS IAM Role and policy
• SSH Key pair
AWS Transfer for
SFTP
• Create Server
• Add User
• Upload and Download files!
Get Started
Demo

46. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

47. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS data transfer & hybrid storage
Online
data transfer
Hybrid
storage
Offline
data transfer
Load streaming
data into
Amazon S3
Ship static data
into and out of
Amazon S3
Access AWS
storage from
on-premises
Edge locations
for Amazon S3
enabled
applications
Online
transfer of
active data
AWS
DataSync
AWS
Transfer
for SFTP
Managed file
transfers into
Amazon S3
NEW
Storage and
compute in
disconnected
environments

48. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Transfer for SFTP
Fully managed service makes it easy to integrate SFTP-based file transfers into AWS
Move your existing SFTP workflows to AWS in 3 steps
Seamless migration
of existing workflows
Data available for
archiving and
processing in
Amazon S3
Simple
to use
Cost-
effective
Fully managed,
highly available, and
elastically scalable
1 2 3

49. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Available in 13 AWS regions • US East (N. Virginia)
• US East (Ohio)
• US West (N. California)
• US West (Oregon)
• Canada (Central)
• EU (London)
• EU (Frankfurt)
• EU (Ireland)
• EU (Paris)
• Asia Pacific (Seoul)
• Asia Pacific (Singapore)
• Asia Pacific (Sydney)
• Asia Pacific (Tokyo)

50. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Get started at
https://aws.amazon.com/sftp
SFTP: The AWS way!

51. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Related sessions
Thursday, November 29
STG-381: Don’t Let SFTP Weight Down Your Migration to the Cloud
11:30 AM - 12:30 PM | Venetian, Level 4 Lando 4305
Friday, November 30
STG-307 Start Shifting SFTP Workflows to the Cloud Now
10:00 AM – 11:00 AM | Mirage, Grand Ballroom B, Table 6

52. Thank you!
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Asa Kalavade
kalavade@amazon.com

53. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.