AWS Korea MMN & SA team is delivering 5 deep dive sessions in 5 months, as one of AWS Korea's customer obsession program. This is the 1st preso.

1. Amazed By AWS Series #1
Play with Databases
정윤진
솔루션아키텍트
younjin@amazon.com

2. 소개
AWS 솔루션아키텍트
Gaming/HPC
DevOps
System Admin
(십년전)Kernel driver dev.

3. 오늘내용
a. INTRO –Ice breaking
b. Game Play Patterns
c. Relational DBMS
d. NoSQL
e. Search

4. 시작하기전에.

5. AWS 명령줄도구
http://aws.amazon.com/ko/cli/
$ sudo pip install awscli
Downloading/unpacking awscli
Downloading awscli-1.5.4.tar.gz (253kB): 253kB downloaded
Running setup.py (path:/private/tmp/pip_build_root/awscli/setup.py) egg_info for package awscli
…
..
.
Successfully installed awscli botocore bcdoc colorama docutils rsa jmespath pyasn1
Cleaning up...
$
$ awsconfigure
AWS Access Key ID [None]: XXXXXXXXXXXXXXXXX
AWS Secret Access Key [None]: XXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Default region name [None]: ap-northeast-1
Default output format [None]: json
$

6. $ aws s3 ls s3://yz-ngs-outputs
PRE Redis-snapshots/
PRE hohoho/
PRE pig-apache/
PRE transfer-test/
PRE wordcount/
2013-06-07 17:56:52 6097048 Elastic Beanstalk for Startups.pptx
2013-05-31 13:33:49 0 HG00099.mapped.SOLID.bfast.GBR.exome.20111114.bam
2013-05-06 15:56:56 132644272 TrendMicro-TTi_6.0_HE_Full.exe
2013-11-10 03:16:23 5188 fabricrc.txt
$
AWS 명령줄도구

7. Ice breaking

8. Amazon Web Services

11. Amazon Corporate Services Korea LLC
Our office
29thSep 2014, 3AM
AWS 관련문의사항은여기로
hwanbin@amazon.com
jungin@amazon.com
woosang@amazon.com

12. 개발자에게AWS란?
http://www.toptiertactics.com/wp-content/uploads/2011/10/Workaholic-Cartoon.jpg

14. Java
Python
PHP
.NET
Ruby
nodeJS
iOS
Android
AWS SDK

15. https://github.com/awslabs

16. 하나이상의배포도구의사용이중요
OpsWorks
CloudFormation

17. Game type:
모바일전략

19. 데이터모델에대해
각종상태정보:레벨,엑슬러,트로피,골드
건물배치정보:담벼락,성의위치,대포위치
게임플레이상태정보: 온라인,오프라인,공격받는중,공격하는중
전투기록정보:언제어느유닛을어느좌표에,공격/방어기록
기타:
트로피를통한순위비교
실시간채팅
사용자이용행태로깅
실시간대전조건이붙지않으면,
또는붙으면?

20. 상태정보
위치정보

21. Game type:
모바일캐주얼

22. 데이터모델에대해
각종상태정보:레벨,스코어,골드,아이템현황
기록정보:언제로그인,로그아웃,마지막플레이레벨
기타:
스코어를통한순위비교
사용자이용행태로깅
클라이언트에서게임데이터가완성
Backend 는데이터
기록,순위,로깅,분석을위해존재

23. Game type:
FPS, TPS

24. 데이터모델에대해
각종상태정보:레벨,스코어,골드,아이템현황
기록정보:언제로그인,로그아웃,마지막플레이레벨
로비정보:전투방생성을위한채팅,중계를위한세션정보의저장
상태정보:현재속한세션id 및플레이중인room id 등
기타:
스코어를통한순위비교
사용자이용행태로깅
로그인및캐릭터관련정보를제외하면거의대부분세션의중계및결과업데이트에집중

25. Game type:
MMORPG

26. 데이터모델에대해
각종상태정보:레벨,스코어,골드,아이템현황이실시간으로변경
기록정보:언제로그인,로그아웃,마지막플레이레벨,데미지,몬스터및PvP 정보
로비정보:게임내실시간사용자간채팅,고객지원이포함
상태정보:서버간캐릭터의위치정보,HP 및데미지,방어력등다양한스텟정보필요
기타:
NPC 와캐릭터간인터렉션
사용자플레이형태로깅
사용자의모든행위에대해실시간에가까운서버간동기화등이필요

28. 올바른데이터베이스타입의선택
데이터엔티티간의관계가필요한가?
1명–수억명의사용자접근에대비,확장이가능한가?
데이터가영속적으로저장되어야하는가?
데이터가특정시점에만유효한가?
한번생성된데이터가자주변경될가능성이있는가?
한번생성된데이터는영원히변하지않는가?
Insert / Update / Delete / Select 중어떠한type 의데이터변경이많은가?
단말(클라이언트플랫폼)에서지원하는로컬DB와동일한데이터타입을사용할수있는가?
http://www.raywenderlich.com/913/sqlite-tutorial-for-ios-making-our-app
http://www.slideshare.net/MindfireSolutions/web-sql-database-in-html-5

29. 게임데이터베이스와AWS

30. 1.RDBMS 만으로처리하는데는한계가있다.
2.NoSQL 만으로는Rich Query 를수행할수없다.
3.분산환경에적합한게임서버구조를만들어야한다.
1.RDBMS 의사용비율축소,샤딩구조확립
2.NoSQL + Search engine 의혼합사용
3.RPC, Coroutine (yield)과같은기법의적극사용
4.Tight coupled 의방법보다는Loosely coupled 의구현
5.Pub/Sub 을통한서버간메세지전달
Game dev on AWS

31. 구조화된데이터베이스–복잡한쿼리를지원
-SQL : Amazon RDS
-Data Warehouse : Amazon Redshift
-Search : Amazon CloudSearch
구조화되지않은데이터베이스–커스텀쿼리를지원
-Hadoop : Elastic MapReduce
Database Overview

32. 구조화된데이터베이스–단순한쿼리를지원
-NoSQL : DynamoDB
-Cache : ElastiCache
구조화되지않은데이터베이스–쿼리를지원하지않음
-클라우드스토리지:S3, Glacier
Database Overview

33. 관계형데이터베이스
SELECT customer_info.firstname, customer_info.lastname, purchases.item
FROM customer_info INNER JOIN purchases
ON customer_info.customer_number = purchases.customer_number;

34. Relational Database Service
Database-as-a-Service
No need to install or manage database instances
Scalable and fault tolerant configurations
Feature
Details
Platform support
Create MySQL, SQL Server and Oracle RDBMS
Preconfigured
Get started instantly with sensible default settings
Automated patching
Keep your database platform up to date automatically
Backups
Automatic backups and point in time recovery and full DB backups
Provisioned IOPS
Specify IO throughput depending on requirements
Failover
Automated failover to slave hosts in event of a failure
Replication
Easily create read-replicas of your data and seamlessly replicate data across availability zones
Compute
Storage
AWS Global Infrastructure
Database
App Services
Deployment & Administration
Networking

35. 사용이권장되는경우
-트랜잭션
-복잡한쿼리가요구되는경우
-최대30,000 IOPS (15K reads, 15K writes)
-100기가바이트에서수TB수준
-워크로드가1대수준에서처리가능한경우
-높은내구성이요구되는경우
RDS

36. 사용이권장되지않는경우
-대량의read/write 가요구될때(150,000 write request/sec)
-데이터의크기또는처리량으로샤딩을결정해야하는경우
-NoSQL 로처리가가능한수준의쿼리또는Get/Put의요청이대부분인경우
-복잡한분석이필요한경우
RDS

37. Windows 기반의게임회사에서많이사용
EC2 / RDS 에서사용가능
EC2 에서사용시PIOPS 적용된다수의EBS 를통해확장가능
i2 / r3 / c3계열의인스턴스사용가능
종전의데이터센서에서사용하던모든구성그대로사용가능(SAN 기반클러스터링제외)
RDS 에서는3개의region에서Multi-AZ지원
10,000 PIOPS 적용가능
SYSADMIN 계정확보불가
DB 가동작중인OS 에접근불가
최소볼륨크기200G, 최대볼륨크기1T
MSSQL

38. RDS 에서는Multi-AZ,Read replication 등DB engine 중가장다양하고많은기능을지원
DB engine 에서는timezone 변경이아직불가능.UTC 기반
클라이언트접속시설정변경또는코드에서적용해야함
사용이매우편리,dump/restore 역시기존에사용하는mysqldump / mysql 클라이언트를사용하여가능
OS root 는확보불가능
순후하게MySQL 만사용하면서EC2에직접설치하여사용하는경우요구사항은많지않음.
Maria 와같은DB 의사용
MMM / Galera 와같은별도의3rdparty 도구를사용하고자하는경우에유용
Scale-up 의방법은EC2 의방법과동일
MySQL

39. Region
Availability
Zone A
Availability
Zone B
S3
EC2
• 게임은 언제나 “Write heavy”
• 캐싱이 언제나 통하는것은 아님
• 키 밸류 / 키 밸류
• 잦은 바이너리의 사용
• 데이터베이스는 언제나 병목
ELB
EC2
CloudFront
CDN
CUID 는 괴롭다
ㅠㅠ
(creat/update/delete/insert)

40. Availability
Zone A
C2
Shard (No Fun)

41. 관계형데이터베이스성능의극대화방법
1.EBS PIOPS 볼륨의적극적인사용
2.Instance Store 의적극적인사용
3.테이블을많은볼륨에분산저장,테이블을서로다른볼륨에저장

42. Availability Zonea
Availability Zoneb
MSSQL
Master
MSSQL
Standby
4 SSD
Stripped Instance Store
(Main DB data store)
4 SSD
Stripped Instance Store
(Mirrored DB data store)
30G
EBS
nG
EBS
root
TX
LOG
30G
EBS
nG
EBS
Mirroring
root
TX
LOG
Create volume snapshot
Volume create
From snapshot
Attach volume
For restoration

43. Availability Zonea
Availability Zoneb
MSSQL
Master
MSSQL
Standby
4 SSD
Stripped Instance Store
(Main DB data store)
4 SSD
Stripped Instance Store
(Mirrored DB data store)
30G
EBS
nG
EBS
root
TX
LOG
30G
EBS
nG
EBS
Mirroring
root
TX
LOG
Create volume snapshot
Volume create
From snapshot
Attach volume
For restoration
Instance Store 는재부팅시데이터가사라짐
따라서복제/복구/백업에대해확실한대비가필요
확실한대비없이는사용에주의!!!!!

44. Availability Zonea
Availability Zoneb
MSSQL
Master
MSSQL
Standby
n X PIOPS EBS
Not raided, for Windows Server
(table partition, table space)
(Main DB data store)
n X PIOPS EBS
Not raided, for Windows Server
(table partition, table space)
(Mirrored DB data store)
30G
EBS
nG
EBS
root
TX
LOG
30G
EBS
nG
EBS
Mirroring
root
TX
LOG
Create volume snapshot
Volume create
From snapshot
Attach volume
For restoration

45. NoSQL
redis> MSET one 1 two 2 three 3 four 4
OK
redis> KEYS *o*
1) "four"
2) "two"
3) "one"
redis> KEYS t??
1) "two"
redis> KEYS *
1) "four"
2) "two"
3) "one"
4) "three"
redis>

46. 사용이권장되는경우
-예측가능한,빠른속도가요구되는경우
-순식간에대규모확장요구사항이있는경우
-샤딩을자동으로해야하는경우
-데이터의정합성과낮은지연시간이동시에필요한경우
-데이터저장의크기및처리량의한계를설정할수없는경우
-아주높은내구성이요구되는경우
-간단한쿼리또는KV 저장소가요구되는경우
DynamoDB

47. 사용이권장되지않는경우
-다수의아이템또는row및서로다른테이블참조트랜잭션이필요한경우
-복잡한쿼리및조인이필요한경우
-로그성데이터에대한실시간분석이필요한경우
-자주참조되지않는데이터를저장하고자하는경우
DynamoDB

48. 클러스터자동생성–Master : Multi replication nodes
Auto Backup
Auto Failover
Restore from rdb file
ElastiCache -Redis
최근의게임에서는pub/sub 을통한데이터공유에사용
종전의UDP/TCP broadcast 를대처가능
In-memory 처리로매우고속

49. Redis –pubsub
게임서버간이벤트공유

50. 사용이권장되는경우
-확장된데이터모델과함께KV 저장소가필요한경우:string, lists, sets, sorted sets, hashes
-캐싱
-Leader board (리더보드)
-고속의정렬
-원자단위의카운팅
-큐시스템이필요한경우
ElastiCache –Redis

51. 사용이권장되지않는경우
-Native 한샤딩또는Scale-out 이필요한경우
-아주높은수준의데이터보존성이요구되는경우
-메모리에적합하지않은데이터를보유하고있는경우
-예외상황에서도특정트랜젝션에대한롤백이필요한경우
ElastiCache –Redis

52. http://techblog.netflix.com/2013/02/announcing-evcache-distributed-in.html
In-memory caching solution based on memcached& spymemcached
-Ephemeral
-Volatile
-Cache (KV Store)
https://github.com/Netflix

53. http://media.amazonwebservices.com/AWS_NoSQL_Couchbase.pdf

54. Minimum production
45GiB (20 x m1.large)
5Gbps access EBS
Up to 80,000 IOPS
Medium production
410GiB
10Gbps access EBS
Up to 640,000 IOPS

55. Large production
360GiB (30 instances)
10Gbps access EBS
Up to 500,000 IOPS
Extra large production
1.5TiB (30 instances)
20Gbps access EBS
Up to 640,000 IOPS

56. https://media.amazonwebservices.com/AWS_NoSQL_MongoDB.pdf

65. http://www.youtube.com/watch?v=hYtRw3Zb6AY
Cassandra + OpsWorks

67. DynamoDB
Provisioned throughput NoSQL database
Fast, predictable performance
Fully distributed, fault tolerant architecture
Feature
Details
Provisioned throughput
Dial up or down provisioned read/write capacity
Predictable performance
Average single digit millisecond latencies from SSD backed infrastructure
Strong consistency
Be sure you are reading the most up to date values
Fault tolerant
Data replicated across availability zones
Monitoring
Integrated to Cloud Watch
Secure
Integrates with AWS Identity and Access Management (IAM)
Elastic MapReduce
Integrates with Elastic MapReduce for complex analytics on large datasets
Compute
Storage
AWS Global Infrastructure
Database
App Services
Deployment & Administration
Networking

68. DynamoDB, 조금더깊게

69. NoSQL 의종류
MongoDB
Riak
Hbase
Cassandra
Neo4j
CouchDB
DynamoDB
이외에도더많은NoSQL 이존재

70. NoSQL vs. RDB
BASE 의특성
대용량데이터
확장가능
빠른쓰기속도
유연한스키마설정
ACID 특성
-Atomicity
-Consistency
-Isolation
-Durability
트랜젝션기반
Rich한쿼리
조인
NoSQL
RDB

71. BASE 특성이란
Basically Available, Soft state, Eventually consistent
즉,
데이터의즉시반영을통한높은일관성을일부포기하는대신,
확장성을특징으로.
모든NoSQL 이완전히BASE의특성을가지지는않지만기존RDB 의ACID 특성을포기해야하는것이요구사항이된다.

72. API
SDK
Database
Client Side
Service Side
Client application
DynamoDB 의사용

73. 특징１：관리가필요없는높은신뢰성
SPOF가존재하지않는구성
데이터는3개소의AZ에분산저장되어높은신뢰성
스토리지는필요에따라자동으로분산처리
Client

74. 특징２：처리량을프로비저닝가능
Read 및Write, 각각필요한만큼의처리용량을할당
예를들어일반적인Read Heavy DB 라면
Read : 1,000
Write : 100
약간Heavy 한DB의경우
Read : 500
Write : 500이설정값은DB 운영중에온라인으로변경가능
Scale-down에관해서는하루4회만가능

75. 특징３：저장용량에제한이없는
사용한만큼지불하는종량제스토리지
데이터용량증가에따른노드추가와같은작업이불필요

76. DynamoDB를시작하려면
1.테이블의Key 와Index 를결정
2.Read/Write 처리량을결정
That’s it, write your code!

77. DynamoDBDemo
-Tic TacToe Game
https://github.com/awslabs/dynamodb-tictactoe-example-app

78. DynamoDB 사용

79. 제공되는API
PutItem
UpdateItem
GetItem
DeleteItem
Query
Scan
BatchWriteItem
BatchGetItem
CreateTable
DescribeTable
UpdateTable
DeleteTable

80. 주로사용되는기능
•Get/Put/Update/Delete/BatchGet
•Scan
전체테이블을싹쓸이로긁어옴
•Query
Hash + 범위키만

81. 공식SDK 이외에도
Perl
Net::Amazon::DynamoDB
Erlang
wagerlabs/ddb
https://github.com/wagerlabs/ddb
Go
go-dynamodbhttps://github.com/fabiokung/go-dynamodb

82. 테이블디자인을위한요소（１）
Table
Primary Key 로”Hash key” 혹은”Hash key & Range key”를선택
기본키：Hash key
Hash key 단체로데이터를고유하게식별할수있는경우사용
기본키：Hash key & Range key
Hash key에해당하는여러데이터에서Range key 로검색가능
Local Secondary Indexes
Range key 이외에필터검색을위한키를가질수있음
Global Secondary Indexes
기존Hash key & Range key 외다른attribute 를key 로지정가능

83. 테이블디자인을위한요소（２）
Attributes
데이터의내용. Hash key 에해당하는Attributes 이외에는미리정의할필요는없다. 또한레코드에서Attributes 가불규칙하더라도문제없다.
Attributes 형식
String
Number
Binary
Array of String
Array of Number
Array of Binary

84. DynamoDB의데이터모델
테이블
아이템
기본키
속성
Hash key 또는Hash key & Range key

85. DynamoDB의기본키
•Hash key
간단한키값
Hash 이므로정렬이필요없음
•Hash key + Range key
복합기본키
Range key 는sort 가있음

86. 샘플（１）상품카탈로그
스키마
테이블명Products
상품ID(ProductId) 를테이블의Hash key 로사용
ProductId
(Hash key)
ProductName
Price
・・・
・・・
1
Pen
100
・・・
・・・
2
Pencil
50
・・・
・・・
3
Eraser
100
・・・
・・・

87. node.js SDK 사용예
데이터를테이블에입력
샘플（２）로그테이블
var data = new Array(
{ AudienceId: { N:"1" }, Timestamp: { S:"2013-10-01 00:01:01" }, Action: { S:"Login" } },
{ AudienceId: { N:"2" }, Timestamp: { S:"2013-10-01 00:02:02" }, Action: { S:"Login" } },
{ AudienceId: { N:"1" }, Timestamp: { S:"2013-10-01 00:21:00" }, Action: { S:"Login" } },
{ AudienceId: { N:"1" }, Timestamp: { S:"2013-10-01 00:42:00" }, Action: { S:"ViewHoge" } },
{ AudienceId: { N:"1" }, Timestamp: { S:"2013-10-01 00:56:22" }, Action: { S:"PostHoge" } }
);
for (var i =0; i < data.length; i++){
dynamoDb.putItem(
{
TableName:"Audience",
Item: data[i]
},
function(err,data){
if(err){
console.log(err);
}else{
console.log(data);
}
}
);
}

88. node.js SDK 사용예
사용자의10월1일0시10분이후의로그를가져오기
샘플（２）로그테이블
dynamoDb.query(
{
TableName:"Audience",
KeyConditions: {
"AudienceId": {
ComparisonOperator:"EQ",
AttributeValueList:[ {N:"1"} ]
},
"Timestamp": {
ComparisonOperator:"GT",
AttributeValueList:[ {S:"2013-10-01 00:10:00"} ]
}
}
},
function(err,data){
if(err){
console.log(err);
}else{
console.log(data);
}
}
);

89. 샘플（３）LSI를적용한로그테이블
사용자로그에Local Secondary Indexes(LSI)를보면
AudienceId
(Hash key)
Timestamp
(Range key)
Action
(Action-Index)
Url
・・・
1
2013-10-01 00:01:01
Login
・・・
・・・
2
2013-10-01 00:02:02
Login
・・・
・・・
1
2013-10-01 00:21:00
Login
・・・
・・・
1
2013-10-01 00:42:00
ViewHoge
1
2013-10-01 00:56:22
PostHoge

90. Local Secondary Indexes（LSI)
•Get/Put/Update/Delete/BatchGet
•Scan
테이블의전체내용을가져옴
•Query
Hash + Range 키만가져옴Local Secondary Index 를사용하여이제한을없앨수있다!

91. Local Secondary Indexes (LSI)
테이블에서어떠한Index 를Query 로사용할지에대한예
5개의LSI 지정이가능
LSI없는테이블에서는Range Key를사용한Query 만가능
LastPostIndex
RepliesIndex
LSI를정의하면Key가아닌속성에대해서도Query 가능
예： -Replies 가10개이상-마지막게시물이XX 이후
LSI
Forum 게시물을보관하는테이블의구조

92. 어째서”Local”Secondary Indexes 인가？
Hash key 가일치하는범위내에서의Secondary Index 이므로
Hash Key 가다른항목에대해Query 결과로가져올수없다
RepliesIndex 를사용하여１개의Query로가져올수있는데이터는?
•ForumName 이S3 이며, Replies 가9개이상
•모든ForumName 에서Replies가９이상 3개의Hash key가있으므로3 회의Query 필요
RepliesIndex

93. LSI 를설정한경우의실제동작
Thread 테이블의LastPostDateTime에LSI 를설정하면,
LastPostIndex
별도의Index 테이블이뒤에서생성, 관리됨
•지정된속성이Range-key
•원래테이블의Range-key 가속성으로

94. 속성의프로젝션
Index 용테이블에지정된속성을복제저장하는기능
예：Thread 테이블에Replies를LastPostIndex에프로젝션
LastPostIndex
Query 수행시Replies를취득하는경우Load 비용을절감
성능향상ReadThroughput 을절약가능

95. User
(Hash)
Timestamp
(Range)
Opponent
Result
Alice
2014-02-21 12:21:20
Bob
Lost
Alice
2014-02-21 12:42:01
Bob
Won
Alice
2014-02-24 09:48:00
Dan
Won
Alice
2014-02-25 16:21:11
Charlie
Won
Battle History
사용자가자신의전투기록을확인하는경우
–User(Alice)가timestamp range-key 를이용하여지난7일간의전투기록을확인
Charlie
02-25 16:21
Won!
Your Battle History
Dan
02-24 09:48
Won!
Alice
02-21 12:42
Won!
게이머의플레이데이터이력

96. 사진공유SNS 앱
Home
My Posts
My Profile
Bob
Steak!
10:18
Carol
BBQ! w/Alice
10:12
Dan
Riajuee…
10:11
Alice
Beer!
10:21
Alice
BBQ! w/Carol
10:12
Alice
Starting BBQ!
10:09
Name:
Alice
Mail: foo
Profile: some texts

97. Users Table
Friends Table
테이블2개를정의
•사용자정보테이블
•친구목록테이블
테이블설계

98. User(Hash)
Nicknames
Bob
[ Rob, Bobby]
Alice
[Allie ]
Carol
[ Caroline ]
Dan
[ Daniel, Danny ]
Users Table
Item
Attribute
(string, number, binary, set)
Primary Key
(Hash)
친구목록검색

99. User(Hash)
Nicknames
Bob
[ Rob, Bobby]
Alice
[Allie ]
Carol
[ Caroline ]
Dan
[ Daniel, Danny ]
Friends Table
User(Hash)
Friend(Range)
Bob
Alice
Alice
Bob
Alice
Carol
Alice
Dan
Users Table
Hash + Range
Primary Key
친구목록검색

100. Friends Table
Users Table
User(Hash)
Nicknames
Bob
[ Rob, Bobby]
Alice
[Allie ]
Carol
[ Caroline ]
Dan
[ Daniel, Danny ]
User(Hash)
Friend(Range)
Bob
Alice
Alice
Bob
Alice
Carol
Alice
Dan
Alice의친구목록검색
1.Query (Table = Friends, Hash = Alice, Range = *)
2. BatchGetItem(Bob, Carol, Dan)
친구목록검색

101. Friends Table
Users Table
User(Hash)
Nicknames
Bob
[ Rob, Bobby]
Alice
[Allie ]
Carol
[ Caroline ]
Dan
[ Daniel, Danny ]
User(Hash)
Friend(Range)
Bob
Alice
Alice
Bob
Alice
Carol
Alice
Dan
Alice의게시물및이미지검색
1.Query (Table = Friends, Hash = Alice, Range = *)
2. BatchGetItem(Bob, Carol, Dan)
게시물검색

102. Images Table
User（Hash)
Image（Range)
Date
Link
Bob
aed4c
2013-10-01
s3://…
Bob
cf2e2
2013-09-05
s3://…
Bob
f93bae
2013-10-08
s3://…
Alice
ca61a
2013-09-12
s3://…
Bob
Bob의게시물목록을검색
Query (Table=Images, Hash= Bob, Range=*)
만약특정시간이후의사진목록을가져오고싶다면?
이미지검색

103. Images Table
User
Image
Date
Link
Bob
aed4c
2013-10-01
s3://…
Bob
cf2e2
2013-09-05
s3://…
Bob
f93bae
2013-10-08
s3://…
Alice
ca61a
2013-09-12
s3://…
User
Date
Image
Bob
2013-09-05
cf2e2
Bob
2013-10-01
aed4c
Bob
2013-10-08
f93bae
Alice
2013-09-12
ca61a
Table
ByDate Local Secondary Index
Local Secondary Index 를Date속성에적용
날짜및시간이미지수집

104. ImageTags Table
Image
User
aed4c
Alice
aed4c
Bob
f93bae
Alice
f93bae
Bob
Image f93bae에Alice를태그PutItem(Table = ImageTags,
Hash = f93bae, Range = Alice)
Bob
만약사용자가태그된이미지목록으로검색하고자한다면?
Image f93bae에태그된사용자목록Query(Table = ImageTags,
Hash = f93bae, Range = *)
이미지에사용자를태그

105. ImageTags Table
UserImage의Range를Global Secondary Index로적용
User(Hash)
Image(Range)
Bob
aed4c
Bob
f93bae
Alice
aed4c
Alice
f93bae
ByUser Global Secondary Index
Image(Hash)
User(Range)
aed4c
Alice
aed4c
Bob
f93bae
Alice
f93bae
Bob
Table
Bob
Alice가태그된사진목록
사용자의태그이미지목록

106. 테이블디자인: 기본키의사용
Hash key와Range key의개념
A,D
B,E
C,F
1 23456789
Parition1
Partition2
Parition3
Range key
Partition내에서데이터의순서를보장
Hash key
Partition간의데이터분산에이용되는key
DynamoDB의처리량은각Partition에부하가적절히분산되었을때를기준으로설계되어있다. 하나의Partition 으로프로비저닝된처리량이나오는것이아님에주의

107. A,D
B,E
C,F
1 23456789
Parition1
Partition2
PartitionN
Partition
DynamoDB는프로비저닝된처리량을보장하기위해데이터를여러파티션에분산저장한다.
Partition은저장된데이터량과프로비저닝한처리량에의해자동으로다시파티셔닝한다.
지금현재의파티션수를확인하는것은불가능
테이블디자인: 기본키의사용

108. A,D
B,E
C,F
1 23456789
Parition1
Partition2
ParitionN
처리량은파티션에균등하게분할된다.
프로비저닝된처리량은각파티션에균등하게분할되며,각합은지정한처리량이나오도록설계되어있다.
따라서엑세스키에편향이발생할가능성이많은구조로테이블이설계되어있다면원하는만큼성능이나오지않을가능성이있으므로주의해야한다.
프로비저닝된처리량: x
x/N
x/N
x/N
테이블디자인: 기본키의사용

109. 테이블디자인: 기본키의사용
Hot Key문제：특정Hash key에데이터가집중되면성능이저하（예상대로의성능이아님）됨에주의
위의Audience 테이블에서AudienceId처럼데이터가특정키에집중하기어려운것을선택한다
예를들어,「년월」과같이데이터의집중이발생하기쉬운데이터를Hash key로선택하는것은안티패턴

110. 테이블디자인: LSI
Local Secondary Indexes는저장소와IO 비용증가를발생시킬수있음
LSI는소위“Projection”에서구현된다
따라서Projection 되는속성이많을수록스토리지비용이증가
Write시쓰기비용역시증가
반드시필요한조건에만LSI 를사용
Projection이필요한Attributes만선택하도록한다

111. DynamoDB Local
개발및테스트용도구
이전까지는개발및테스트를위해서는반드시DynamoDB테이블을생성해야했다. 이는”비용발생의우려”、”내부테스트환경구성불가”、”오프라인으로개발불가능”과같은문제가있었음
이도구를사용하여개발및테스트를보다편리하게수행할수있음
JAR파일로제공되어로컬에설치및실행가능(Java7)
단순히API의기능재현을위한테스트용도구이기때문에서비스환경에는사용이불가능
프로비전된처리량은반영되지않으므로주의
자세한내용은AWS document 를참조

112. Amazon EMR의Hive 에서사용
CREATE EXTERNAL TABLEAudience
( AudienceId Int, ActionTimestamp string, Action string )
STORED BY'org.apache.hadoop.hive.dynamodb.DynamoDBStorageHandler'
TBLPROPERTIES(
"dynamodb.table.name" = ”Audience2",
"dynamodb.column.mapping" =
”AudienceId:AudienceId,
ActionTimestamp:Timestamp,
Aciton:Action“);
hive의External Table로이용가능
DynamoDB의데이터집계등이필요한경우등에이용
자세한내용은http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/EMR_Interactive_Hive.html

113. hive를사용하여S3에데이터백업
EMR에서hive는DynamoDB뿐만아니라S3 역시External Table 사용할수있는장점을살려DynamoDB 를External Table 로Select한데이터를다시S3 의External Table로Insert 가능
INSERT OVERWRITE TABLE
s3_as_external_table
SELECT *
FROM dynamodb_as_external_table;

114. Redshift 의COPY 를사용해데이터를로드
자세한내용은http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/RedshiftforDynamoDB.html
Amazon Redshift 에데이터로딩
COPY audience
FROM ‘dynamodb://Audience2’
CREDENTIALS
'aws_access_key_id=<Your-Access-Key- ID>;aws_secret_access_key=<Your-Secret-Access-Key>'
READRATIO 50;

115. http://www.youtube.com/watch?v=Dh8kp1AcRg0

118. https://github.com/Aconex/scrutineer
-오픈소스: elasticsearch.com 통해서포트가가능
-분산검색엔진
-클러스터링형태로구성이가능
-고가용성
-다중사용자환경

119. https://github.com/kzwang/elasticsearch-river-dynamodb
River

120. http://docs.couchbase.com/couchbase-elastic-search/

121. http://www.elasticsearch.org/guide/en/elasticsearch/reference/0.90/modules-gateway-local.html
http://www.elasticsearch.org/guide/en/elasticsearch/reference/0.90/modules-gateway-s3.html

123. 사용이권장되는경우
-검색에대해내가전문성이없다고판단되는경우
-Full-text 검색이필요한경우
-랭킹
-검색
-구조화된데이터와구조화되지않은데이터가혼재하는경우
CloudSearch

124. 사용이권장되지않는경우
-다수의아이템또는row및서로다른테이블참조트랜잭션이필요한경우
-복잡한쿼리및조인이필요한경우
-로그성데이터에대한실시간분석이필요한경우
-자주참조되지않는데이터를저장하고자하는경우
CloudSearch

125. http://www.youtube.com/watch?v=mdj8RQhqOEg

126. Write once
Read many then,
s3://mys3database/HASHasFILENAME.JSON

127. S3 -Durable storage, any object
99.999999999% durability of objects
Unlimited storage of objects of any type
Up to 5TB size per object
Feature
Details
Flexible object store
Buckets act like drives, folder structures within
Access control
Granular control over object permissions
Server-side encryption
256bit AES encryption of objects
Multi-part uploads
Improved throughput & control
Object versioning
Archive old objects and version new ones
Object expiry
Automatically remove old objects
Access logging
Full audit log of bucket/object actions
Web content hosting
Serve content as web site with built in page handling
Notifications
Receive notifications on key events
Import/Export
Physical device import/export service
Compute
Storage
AWS Global Infrastructure
Database
App Services
Deployment & Administration
Networking

128. 1.RDBMS 만으로처리하는데는한계가있다.
2.NoSQL 만으로는Rich Query 를수행할수없다.
3.분산환경에적합한게임서버구조를만들어야한다.
1.RDBMS 의사용비율축소,샤딩구조확립
2.NoSQL + Search engine 의혼합사용
3.RPC, Coroutine (yield)과같은기법의적극사용
4.Tight coupled 의방법보다는Loosely coupled 의구현
5.Pub/Sub 을통한서버간메세지전달
Summary

129. aws.amazon.com/ko/game-hosting