1. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
최용호
Vinus Entertainment
모바일 게임을 만들기 위한
AWS 고군분투기

2. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
발표자 소개
• 최용호
• (전) Vinus Entertainment 서버팀
• (현) 넥슨 게임클라우드개발팀
• 자바카페 커뮤니티 운영진

3. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
게임 소개

4. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
발표 순서
자동화에 대한 고민
네트워크 구성에 대한 고민
서비스 선택에 대한 고민
부하 테스트에 대한 고민
게임 데이터 활용에 대한 고민
최종 아키텍처

5. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
자동화에 대한 고민

6. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
개발 환경

7. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
기존의 개발 환경 구성 방식
OS 설치
(리눅스)
패키지
업데이트
설치 가이드
확인
설치 가이드
확인
프로그램
설치
설치 가이드
확인
OS 환경설정
프로그램
설정

8. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
기존의 환경 구성 방식
OS 설치
(리눅스)
패키지
업데이트
설치 가이드
확인
설치 가이드
확인
프로그램
설치
설치 가이드
확인
OS 환경설정
프로그램
설정
설치 가이드에 변경이 발생하면?

9. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
기존의 환경 구성 방식
OS 설치
(리눅스)
패키지
업데이트
설치 가이드
확인
설치 가이드
확인
프로그램
설치
설치 가이드
확인
OS 환경설정
프로그램
설정
재설정 및 재설치 필요

10. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
자동화 해줄 무언가가 필요

11. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Docker

12. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
서비스 별로 디렉토리를 만들어 Docker 파일 관리

13. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Docker 파일과 설정 파일들

14. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.

15. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
tomcat
mariadb redis
Logstash elasticse
arch
kibanahttpd
Client
Pinpoint-
hbase
Pinpoint-
collector
Pinpoint-
web
Docker-
compose
Docker-
compose
Docker-
compose
Amazon
ECR
Kibana와 Pinpoint를 통해 서버 상태 확인
Docker Image들은 ECR을 사용하여 버
전 관리
사내 개발 환경 아키텍처

16. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
빌드 및 배포

17. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
배포 절차
Build Server

18. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
배포 절차
Build Server
EC2 instance contents
Tomcat

19. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
배포 절차
Build Server
Building
EC2 instance contents
Tomcat

20. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
배포 절차
Build Server
WAR
EC2 instance contents
Tomcat

21. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
배포 절차
Build Server
WAR
EC2 instance contents
Tomcat
SCP

22. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
자동화 해줄 무언가가 필요

23. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Jenkins

24. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
My Company Private Network
Jenkins
Server
EC2 instance contents
mariadb
redistomcat
Operating
Tool
tomcat
Game
각 서버 빌드 및 배포
개발 서버 배포

25. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
My Company Private Network
Jenkins
Server
S3
Elastic Beanstalk container
instances
1. 빌드 된 war파일 업로드
2. 배포
3. 업로드 된 war 파일로 배포 수행
라이브 서버 배포

26. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
네트워크 구성에 대한 고민

27. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
VPC

28. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Availability Zone 1 Availability Zone 2
VPC

29. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Availability Zone 1 Availability Zone 2
VPC
subnet subnet

30. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Availability Zone 1 Availability Zone 2
VPC
subnet subnet

31. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Availability Zone 1
Private subnet
Public subnet
Availability Zone 2
Private subnet
Public subnet
VPC

32. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Availability Zone 1
Private subnet
Public subnet
Availability Zone 2
Private subnet
Public subnet
VPC
Elastic
Load balancer
User
Admin
instance
instance instance
Bastion

33. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
서비스 선택에 대한 고민

34. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
사내 환경과 동일하게
AWS에서도 Docker로 구성할 수 있을까?

35. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Amazon ECS

36. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
검토 당시 서울리전에서
제공되지 않아 포기
Amazon ECS

37. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Amazon
Elastic
Beanstalk

38. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Elastic Benastalk 워크 플로우
Application
생성
Environment
생성
Application
Version
업로드
Application
Version
배포

39. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Elastic Beanstalk 구성
Application

40. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Elastic Beanstalk 구성
Environment

41. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Environment 설정

42. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Environment 구성 저장 기능

43. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Application 버전 관리 및 배포

44. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
주의해서 변경해야 할 설정들
• 오토스케일링의 시작 구성(Launch Configuration)
• VPC 설정
• 인스턴스 type 변경
• AMI 변경
• SSH key-pair 변경

45. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
부하테스트에 대한 고민

46. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
원하는 만큼 확장 가능한 부하테스트가 목표
instances
instance
instance
instance
Classic
Load
Balancer

47. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Spot
Instance
instances
부하 전달
Spot
Instance
Spot
Instance
Classic
Load
Balancer
비용 절감을 위해 스팟 인스턴스 사용

48. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Spot
Instance
ssm
Run Command
instances
부하 전달
Spot
Instance
Spot
Instance
Classic
Load
Balancer
System Management를 사용한 스크립트 실행

49. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
instance
Spot
Instance
ssm
Run Command
instances
부하 전달
Loadtest
Server
Spot
Instance
Spot
Instance
스팟 인스턴스를 생성하고,
생성된 인스턴스들에 대한 상태 확인과
명령을 내리기 위한 서버
Classic
Load
Balancer
명령을 내리기 위한 툴 제작

50. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
부하테스트를 위한 툴

51. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Jmeter의 진행 상황 수집을 위한 로그

52. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
성능 측정을 위한 툴 - pinpoint

53. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
성능 측정을 위한 툴 - X-Ray

54. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
게임 데이터 활용에 대한 고민

55. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
지표 데이터 설계
사용자 계정
캐릭터
무기
아이템
상점
...
가입
로그인
구매
판매
...
Title SubTitle

56. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
지표 데이터 예
상점
노점상 뽑기 전투
아이템
재화
Title
SubTitle

57. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
데이터 계층 구조
스테이지 클리어
재화 획득
아이템 획득
캐릭터 획득
경험치 획득
퀘스트 달성
...
ID : 10000
ID : 10001, parentID : 10000
ID : 10002, parentID : 10000
ID : 10003, parentID : 10000
ID : 10004, parentID : 10000
ID : 10005, parentID : 10000

58. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
데이터를 어디에 저장하고,
어떻게 시각화를 할까?

59. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
데이터 수집/가공/저장/시각화 - 엘라스틱스택

60. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
{
"id": "b46adf7e5bf14c4ca18868f930e2670f",
"uid": 5824735036189904000,
"nickname": "yongho",
"title": "GOODS",
"subTitle": "TAKE",
"detail": {
"gold": 120,
"diamondBonus": 0,
"diamondCash": 0,
"survivalCoin": 0,
"pvpCoin": 0,
"stamina": 1,
"supplies": 0,
"component": 0,
"exp": 0
},
"parentId": "d9388af36d63482b851e33f78225a9be",
"parentTitle": "STAGE",
"parentSubTitle": "CLEAR",
"timestamp": "2017-11-08T06:00:19.551Z",
"loglevel": "TRACE"
}
엘라스틱서치에 기록된 데이터
재화 획득

61. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
{
"id": "b46adf7e5bf14c4ca18868f930e2670f",
"uid": 5824735036189904000,
"nickname": "yongho",
"title": "GOODS",
"subTitle": "TAKE",
"detail": {
"gold": 120,
"diamondBonus": 0,
"diamondCash": 0,
"survivalCoin": 0,
"pvpCoin": 0,
"stamina": 1,
"supplies": 0,
"component": 0,
"exp": 0
},
"parentId": "d9388af36d63482b851e33f78225a9be",
"parentTitle": "STAGE",
"parentSubTitle": "CLEAR",
"timestamp": "2017-11-08T06:00:19.551Z",
"loglevel": "TRACE"
}
엘라스틱서치에 기록된 데이터
스테이지 클리어를 통해 재화 획득

62. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
{
"id": "b46adf7e5bf14c4ca18868f930e2670f",
"uid": 5824735036189904000,
"nickname": "yongho",
"title": "GOODS",
"subTitle": "TAKE",
"detail": {
"gold": 120,
"diamondBonus": 0,
"diamondCash": 0,
"survivalCoin": 0,
"pvpCoin": 0,
"stamina": 1,
"supplies": 0,
"component": 0,
"exp": 0
},
"parentId": "d9388af36d63482b851e33f78225a9be",
"parentTitle": "STAGE",
"parentSubTitle": "CLEAR",
"timestamp": "2017-11-08T06:00:19.551Z",
"loglevel": "TRACE"
}
특정 행위에 연관된 다른 행위들 확인

63. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
엘라스틱스택을 활용한 데이터 파이프라인
Web
Server
EC2
Instance
filebeat

64. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
엘라스틱스택을 활용한 데이터 파이프라인
Web
Server
EC2
Instance
EC2
Instance
filebeat
logstash

65. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
엘라스틱스택을 활용한 데이터 파이프라인
Web
Server
EC2
Instance
EC2
Instance
EC2
Instance
filebeat
logstash elasticsearch

66. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
어떻게 머신 사양을 선택하고,
언제 엘라스틱서치 노드를 추가해야할까?

67. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
한대의 머신에서 한계치 측정
원활하게 동작하는 상태 (r4.large)

68. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
한대의 머신에서 한계치 측정
GC가 급격하게 발생하면서 CPU가 바빠짐

69. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
한대의 머신에서 한계치 측정

70. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
장애가 발생하면 어떻게 하지?

71. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
장애 대응
Web
Server
EC2
Instance
EC2
Instance
EC2
Instance
filebeat
logstash elasticsearch

72. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
장애 대응
Web
Server
EC2
Instance
EC2
Instance
EC2
Instance
filebeat
logstash elasticsearch

73. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
장애 대응
Web
Server
EC2
Instance
EC2
Instance
EC2
Instance
filebeat
logstash elasticsearch

74. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
관리할게 너무 많음…ㅠㅠ

75. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Amazon
Elasticsearch
Service

76. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
데이터 파이프라인을 AWS로 이전
ElasticsearchAmazon
Kinesis
Streams
Lambda
function
Amazon
Kinesis–
enabled app

77. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Filebeat 대신 Kienesis Agent
ElasticsearchAmazon
Kinesis
Streams
Lambda
function
Amazon
Kinesis–
enabled app

78. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Logstash 대신 Kinesis와 Lambda
ElasticsearchAmazon
Kinesis
Streams
Lambda
function
Amazon
Kinesis–
enabled app

79. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Elasticsearch 대신 Amazon Elasticsearch
ElasticsearchAmazon
Kinesis
Streams
Lambda
function
Amazon
Kinesis–
enabled app

80. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Lambda 샘플 코드 제공
https://docs.aws.amazon.com/ko_kr/elasticsearch-service/latest/developerguide/es-
aws-integrations.html
https://github.com/aws-samples/amazon-elasticsearch-lambda-
samples

81. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
엘라스틱서치에서
데이터가 유실되면 어떻게 하지?

82. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
데이터 이중화
S3Amazon
Kinesis–
enabled app

83. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
파일로그 확인 - Athena
S3 Amazon
Athena
Amazon
Kinesis–
enabled app

84. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
로그를 위한 S3 버킷 구조
로그레벨 / 연도 / 월 / 일 / *.txt

85. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
로그를 위한 S3 버킷 구조
2018년 2월의 DEBUG 레벨 로그 파일들
일자별로 로그 파일 관리

86. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Athena 테이블 생성
S3 버킷 경로 지정.
파일 스캔 범위를
좁힐 수록 성능 향상

87. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Athena 테이블 구조
Athena에서는 월별, 일자별로 테이블을 생성하여 파일 탐색 범위를 최소화

88. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Athena 테이블 구조
일자별로 테이블을 생성해서 스캔 범위 축소

89. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
테이블이 너무 많아짐…ㅠㅠ

90. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Athena 테이블 파티션 생성

91. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Athena 테이블 파티션 등록
Alter Table trace_partitioned add Partition (year=2018, month=4, day=18)
LOCATION ‘s3://hive-live-log/TRACE/2018/04/18/’
파티션 등록

92. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Athena 테이블 파티션 쿼리
Alter Table trace_partitioned add Partition (year=2018, month=4, day=18)
LOCATION ‘s3://hive-live-log/TRACE/2018/04/18/’
파티션 등록
Select * from trace_partitioned where timestamp between timestamp '2018-
04-18 00:00:00.000' and timestamp '2018-04-19 00:00:00.000’
and year = 2018 and month = 4 and day = 18
파티션 쿼리

93. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
데이터 시각화

94. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
데이터 시각화 - 키바나
대외비 대외비 대외비 대외비
대외비 대외비

95. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
대외비 대외비 대외비 대외비
대외비 대외비
데이터 시각화 - 키바나
인덱스만 남기는 실수
어떠한 아이템인지 알아볼 수 없음

96. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
대외비 대외비 대외비 대외비
대외비 대외비
데이터 시각화 - 키바나
인덱스만 남기는 실수
어떠한 아이템인지 알아볼 수 없음
잔존율과 같이 로직이
필요한 경우에는 키바나로 불가능

97. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
키바나만으로 불가능한 기능들을 위한 툴

98. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
최종 아키텍처

99. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
VPC
Availability Zone 1
Private subnet
Public subnet
Availability Zone 2
Private subnet
Public subnet
Elastic Beanstalk container
Auto Scaling group
instance instanceAurora ElastiCache
Redis
ElastiCache
Redis
Aurora
instance
Classic
Load
Balancer
NAT
Gateway
Route 53
User
CloudFront S3
ElasticsearchAmazon
Kinesis
Streams
Lambda
function
Web
Serv
er
Web
Server
Operating
Tool
instance
Statistics
Server
Amazon
CloudWatch
SMS
API Provider
Admin
Amazon
Athena
Log data
Statistics
data
Client Resource
instance
Bastion
X-ray

100. 감사합니다