With increase in popularity of online engagement as a means of entertainment, broad use of wide range of communities have become popular. These communities need to be highly available and resilient at scale. Failure of availability could be fatal to the product that are used by the customer. We will share the process you should use to develop your architectural principles that will allow you to reap the benefits of reduced complexity.

1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
A R C 3 0 6
H i g h R e s i l i e n c y & A v a i l a b i l i t y O f O n l i n e E n t e r t a i n m e n t
C o m m u n i t i e s U s i n g M u l t i p l e A W S R e g i o n s
S t e p h a n i e F o k – S W E n g i n e e r i n g M a n a g e r , S I E
S w a t h i S o m a n c h i – S r . E n g i n e e r i n g M a n a g e r , S I E
A m y C h e – S r S o l u t i o n s D e l i v e r y M a n a g e r , A W S
N o v e m b e r 2 0 1 7

2. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

3. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Who are we?
● Stephanie Fok - Software Engineering Manager, SIE
● Swathi Somanchi - Senior Engineering Manager, SIE
● Amy Che – Senior Solutions Delivery Manager, AWS

4. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Considerations for Multi-Region
Design

5. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What to expect from this session?
• Do you NEED Multi-Region Architecture?
• Do you WANT Multi-Region Architecture?
• What are the Multi-Region Building Blocks?
• What does Multi-Region mean to PlayStation’s use case?
• How to architect multiple region infrastructure?
• How to make online communities available active/active?

6. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What does Multi-Region mean to AWS?

7. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Lower latency to a subset of customers
Legal and regulatory compliance
Business Continuity / Disaster Recovery
Why go Multi-Region?

8. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Stuff breaks all the time, but rarely in multiple placesStuff breaks all the time
Design for Failure
at the same time

9. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
We could talk about Alphabet Soup…
MTTR
MTBF
RTO
RPO

10. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Disaster
Recovery point (RPO) Recovery time (RTO)
Data loss Down time
BC/DR: Objectives and impacts
How much data can you afford
to recreate or lose?
How quickly must you recover?
What is the cost of downtime?

11. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Do you WANT Multi-Region Architecture?

12. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
The Four Strategies
Backup &
Restore
Pilot
Light
Warm
Standby
Multi
Site
AWS Multi-Region

13. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
So You’ve Decided to go Multi-Region

14. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS building blocks for Multi-Region
Region 1
Region 2
AZAZAZ
Access Layer
Data Layer
RDS
SQL DB
S3
Objects
DynamoDB
NoSQL
Snapshots/
Backups
AMIs (images)
Route 53
DNS
AZAZAZ
Access Layer
Data Layer
RDS
SQL DB
S3
Objects
DynamoDB
NoSQL
Snapshots/
Backups
AMIs (images)
Cross-Region Snapshot
Copy
Cross-Region
AMI Copy
Cross-Region Replication
Cloud
Formation
Deploy
CloudWatch
Monitor
DX
DX
DC DC

15. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
“Loosely coupled systems”
The looser they are coupled,
the bigger they will scale,
the more fault tolerant they will be,
the less dependencies they will have,
the faster you will innovate.

16. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Loose coupling sets you free
The looser they're coupled, the bigger they scale
Build independent components; decouple interactions
Use common interfaces or common APIs between the components
Messaging enables decoupling
Reliable, durable, fault tolerant delivery of messages
Create unidirectional – non-blocking operations

17. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
• Proven reliability with
messages are stored across
multiple AZs
• Flexible message delivery
over multiple transport
protocols
• Nearly unlimited throughput
• Instantaneous or delayed,
push-based delivery
• Simple APIs and easy
integration
• Amazon CloudWatch metrics
and alerts
• Message payloads up to 256
KB
Amazon SNS

18. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
• Persistent message queue
with high durability and
availability
• Messages are stored across
multiple AZs
• Messages retained until
deleted—up to 14 days
• Nearly unlimited throughput
• Amazon CloudWatch metrics
and alerts
• Message payloads up to 256
KB (5 TB using Amazon S3)
• Message batching to
increase throughput and
reduce cost
• Secure: uses AWS Identity &
Access Management (IAM)
and HTTPS/TLS
Amazon SQS

19. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Now What?

20. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

21. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What services do we support?

22. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Where to begin?

23. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What does Multi-Region means to us?

24. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Multiple instance failures

25. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Multiple zone failures

26. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Minimize customer impact

27. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Minimize customer impact

28. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
How to…
• Architect infrastructure for multiple regions?
• Architect application for multiple regions?
• Test out new region application performance?
• Enable services on multiple regions with no downtime?

29. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Architect infrastructure for multiple regions?

30. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Subnet (10.1.2.0/24)
Scope of IaaS
Route Table
Elastic Network Interface
Amazon VPC (CIDR 10.1.0.0/16)
Internet
Gateway
Virtual Private Gateway
VPN Connection
Network ACL
Security group
Enhanced Networking
VPC Peering
AWS Direct Connect
Env. Block
VPC Endpoints
Amazon VPC
Region
Availability Zone - A

31. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
• User affinity
• Minimize downtime
• Provide enough time to scale
other region
• Percentage-based routing
• Testing before going live
• Multiple routing rules
o Path-based routing
o Hostname-based routing
Routing considerations

32. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Monitoring and logging considerations
● Region specific monitoring and logging
● Monitoring and logging data is sent to other region in case of
region failure
● Monitoring tools are connected to all regions to visualize data
through one tool
● Monitoring pipeline is monitored using different monitoring
tools

33. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Communities Group

34. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Communities

35. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Where Do We Start?

36. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
● Does our data need to be in-sync in all regions?
● How quickly does the data have to be in-sync?
● Does our technical stack already support multiple regions?
● How do you perform a seamless rollout?
Questions one should ask

37. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Technical stack
AWS
ElastiCache
Redis

38. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Technical stack
AWS
ElastiCache
Redis
Service Service Service Service Service Service

39. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Technical stack consolidation
Service Service Service Service Service Service
AWS
ElastiCache
Redis

40. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

41. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Migration with Dynamic Switches Step 1
AWS
ElastiCache
Redis
Read
Write
Read
Write
S3
bucket
Property
File
Update
Service
1. Run migration process
2. Write to both old and
new stack
3. Live service continues
to read from old stack
1
2 2
3

42. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Migration with Dynamic Switches Step 2
AWS
ElastiCache
Redis
Read
Write
Read
Write
S3
bucket
Property
File
Update
Service
1. Migration completes
2. Test and validate data
on new stack in
background
3. Live service continues
to read from old stack
1
3
2

43. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Migration with Dynamic Switches Step 3
AWS
ElastiCache
Redis
Read
Write
Read
Write
S3
bucket
Property
File
Update
Service
1. Switch live service to
read from new stack
2. More test and
validation from live
service
3. Keep writing live
updates to both old
and new stack
1,2
3 3
1

44. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Migration with Dynamic Switches Step 4
AWS
ElastiCach
e Redis
Read
Write
Read
Write
S3
bucket
Property
File
Update
Service
1. When ready, stop
writing to old stack
1

45. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Migration with Dynamic Switches Step 5
AWS
ElastiCache
Redis
Read
Write
Read
Write
S3
bucket
Property
File
Update
Service
● Decommission old
stack

46. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Lessons learned
● Prefer feature switches over code change
● Dynamic updates at runtime without server restarts
● When using dual-write, keep writing to old stack until you are
100% certain
● Wait an extra week before you stop writing to the old stack

47. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Communities Search

48. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Communities Search

49. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
SolrCloud Replication

50. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Questions one should ask
● Do we need search on multiple regions?
● Do you need Active/Active or Active/Standby setup?
● User impact if one region is down?

51. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Option 1
SolrCloud Enterprise Replication

52. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Active/passive out of box
• SolrCloud DC replication supports active/passive architecture
• Primary SolrCloud DC will handle writes
• Secondary DCs will be used for search (reads)
• Data is replicated in near to real-time
• Replication supports new index and pre-built indexes

53. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Current SolrCloud architecture

54. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Active/passive architecture Step 1

55. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Active/passive architecture Step 2

56. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Active/passive architecture Step 3

57. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Challenges
• SSL is not supported between Zookeeper and SolrCloud
communication across regions
• Replication doesn’t handle copy of existing documents to new
region
• Need to maintain same number of shards on both regions

58. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Option 2
Active/Active for SolrCloud Using Kafka

59. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Active/active replication using Kafka

60. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Challenges
● Operational overhead
● More failure points
● Expensive

61. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Option 3
Active/Active for SolrCloud via SQS

62. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Data sync for SolrCloud using SQS
● Keep SolrCloud search local to each region
● Get the upstream source of truth to send updates to all regions
● Relying on SQS instead of using Kafka

63. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Communities Search - Single Region
Communities
Service
Indexer
Send
Update
Message
SQS
Consume
Message
Create/update
document
Update
Community
Query
Client
Search for
Communities
SolrCloud

64. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Consume
Message
Communities Search – Multi-Region
Communities
Service
Indexer
Live
update
message
Index
Update SQS
Send
Update
Search Query
SolrCloud
Region 1
Indexer
Index
Update SQS
Send
Update
SolrCloud
Region 2
Consume
Message
Communities
Service
Live
update
message
Search Query

65. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Index Creation – Single Region
Re-indexing
Service
Indexer
Send re-
index
Message
Consume
Message
Create
document
SolrCloud
Index
Creation
SQSData source

66. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Index re-creation Step 1
Communities
Service
Indexer
Live
update
message
Index
Update SQS
Consume
Message
Send
Update
Query
SolrCloud
Region 1
Indexer
Live
update
message
Index
Update
SQS
Send
Update
SolrCloud
Region 2
Index
Creation
SQS
Re-indexing
Service
1. Temporary
components
2. Send live updates
3. Don’t process live
updates
4. Process index
creation
5. Live query is
served by Region 1
1
2
1
3
4
5

67. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Index re-creation Step 2
Indexer
Live
update
message
Index
Update SQS
Consume
Message
Send
Update
Query
SolrCloud 5
Region 1
Indexer
Live
update
message
Index
Update SQS
Send
Update
SolrCloud 5
Region 2
Consume
Message
Index
Creation
SQS
Re-indexing
Service
Send re-
index
message
Communities
Service
1. Re-indexing
completes
2. Stop processing
index creation
messages
3. Process index
update
messages2
3
1

68. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
SolrCloud multi-region with SQS
Indexer
Send
update
message
Index
Update SQS
Consume
Message
Send
Update
Search Query
SolrCloud
Region 1
Indexer
Index
Update SQS
Send
Update
SolrCloud
Region 2
Consume
Message
Send
update
message
Search Query
Communities
Service
Communities
Service

69. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
SolrCloud multi-region with SQS
Indexer
Send
update
message
Index
Update SQS
Consume
Message
Send
Update
Search Query
SolrCloud
Region 1
Indexer
Index
Update SQS
Send
Update
SolrCloud
Region 2
Consume
Message
Search Query
Indexer
Index
Update SQS
Send
Update
SolrCloud
Region N
Consume
Message
Send
update
message
Search Query
Communities
Service
Communities
Service
Communities
Service

70. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
SolrCloud multi-region with SQS and SNS
Subscribe
Index
Update SQS
Region 1
Index
Update SQS
Region 2 ● Publish to single SNS
topic
Subscribe
SNS
Index
Update SQS
Subscribe
Region N
Communities
Service
Communities
Service

71. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Experiment 1 Experiment 2 Success
SolrCloud multi-region experiment
SolrCloud 5 SolrCloud 6
SolrCloud 6
SolrCloud 5 SolrCloud 6
SolrCloud 6
SolrCloud 5
SolrCloud 5
Region 1 Region 1 Region 1
Region 2 Region 2 Region 2

72. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Lessons learned

73. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Don’t rely on existing database replication strategies till you test it
Trust, but verify!

74. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Warm your cache in the new region

75. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Wait for 30 days before you celebrate. You might encounter latency
and region-related issues that will get you back to the whiteboard

76. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

77. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Thank you!

78. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
PlayStation is hiring