[KGC 2012] Online Game Server Architecture Case Study Performance and Security

Performance and Security
신승민
CTO
㈜윈디소프트
Online Game Server Architecture
Case Study

Performance
Security
Server
Architecture Case Study
Publishing
Platform

論語『雍也』 6-28
己欲立而立人
己欲達而達人

On game,
what is my role
as system engineer?

“TCP vs. UDP 속도는 뭐가 빠를까?”
“ODBC vs. ADO 뭐가 좋을까?”
“분산 시스템 어떻게 구축해야 하는가?”
“HDD vs. SSD 뭘 선택해야 할까?”
“데이터베이스 접속을 위하여 DBC를 따로 두는 것이 좋을까?”
“개발자와 엔지니어 어떤 역할을 수행해야 할까?”
“게임 서버도 가상화를 도입할 수 있을까?”
“암호와 압축은 어떤 순서로 동작하는가?”
“IP 주소 vs. DNS 기반 어떤 것이 효율적일까?”
Questionnaire●

“Scale-out vs. Scale-up 선택은?”
“DB 분산 솔루션 좋은 것 없을까?”
“가상화는 어떤 이점이 있는 거지?”
“Memcached 서버를 게임 서버에도 적용할 수 있을까?”
“클라이언트 패치 효율적인 방안은?”
“그럼 서버 패치 효율적인 방안은?”
“서버에 안티바이러스 제품 설치해야 하는 건가?”
“RAID 10 또는 RAID5 어느 것을 선택해야 하지?”
“SSD에서 RAID 구성은?”
Questionnaire●

Lobby
Gateway
PvP
World
NAT
Database
DBC
Validation
Login
GameDungeon
Terms
Chat Management
Relay

Sample of Server Configuration●
N모사의 C모 게임○
Rookie Level
CPSvr CPSvr CPSvr
RoomSvr
GameDB
AuthSvr
Client Client
UserDB
PatchSvr NATSvr
DBGW
UDP
Game
Data
DBGW

“Room Server: 게임 룰과 방 관리를 담당하는 서버”
“CP Server: 병렬적으로 수가 증가할 수 있는 구조”
“CP Server: 연결 접속점 서버”
Function of Servers○
“Patch Server: 사용자가 게임을 시작했을 때 현재
설치된 클라이언트 버전이 최근 버전과 동일한지 확인”
“NAT Server: UDP Hole Punch를 담당하는 서버”
“Auth Server: 로그인하는 작업 처리”

“CP Server와 GameDB 통신 중간에 데이터베이스
미들웨어를 두어 데이터베이스 때문에 CP Server의
성능이 떨어지는 것을 최대한 방지”
Function of Servers○
Rookie Level
CPSvr CPSvr CPSvr
RoomSvr
GameDB
AuthSvr
Client Client
UserDB
PatchSvr NATSvr
DBGW
UDP
Game
Data
DBGW

How can a client connect among CP Servers?○
“사용자 접속 이전에 어느 CP Server로 접속할지를
관리해 주는 서버를 따로 만드는 것”
“L4 스위치를 이용하는 방법”
“클라이언트에서 CP Server 주소 테이블을 가지고
랜덤하게 접속을 시키는 방법”
“가장 간단하지만 접속량을 고르게 분포하지 못함”
“접속 위치 관리 서버도 병렬로 구성하는 등의 작업이
많이 필요하다”
“대부분 웹 서비스를 서비스할 때 사용자를 분산시켜주기
위해서 사용하는 방법이지만 비싸다”

Rookie Level
Single point of failure○
CPSvr CPSvr CPSvr
RoomSvr
GameDB
AuthSvr
Client Client
UserDB
PatchSvr NATSvr
DBGW
UDP
Game
Data
DBGW
?
?
?
? ? ?
?

High Availability
Virtualization
Scale-out vs. Scale-up
Cloud
Distribute system
Open Source
CompilerScalability
Terms
Synchronous vs.
Non-synchronous system
KVS

Cryptography
Stream
Cipher
Kerckhoff
Protocol
Block
CipherSSL CAPTCHA
Key: Private, Public, Session, Symmetric
Algorithm
Security
Hash
Terms
RSA

Online Game Server Platform
Network Program
DB
Chat
LobbyLogin
Business Logic
Cryptography
The three key functions for game server●

Online Game Server Platform
Game
+ Dungeon
+ NPC
+ PvP
World
+ Channel
+ Lobby
Authentication
+ Login
Chat
+ Messenger
+ Mail
+ Friends
Management
+ Duplicate login
+ OP Tools
Logical Architecture●
Module

Online game server
Architecture

Build a new game server architecture●
Client and Server○
Lobby
Client
① Web Launcher
② Game Client

Authentication Servers○
Lobby1
Client
Web
① What is the IP address
or Hosts name of Authentication Server?
② Auth1
Auth1
UserDB

Patch Servers○
Lobby1
Client
Web
or Hosts name of Authentication Server?
② Auth1
Auth1
UserDB
Patch1

Game Server Farm
Lobby & Game Servers○
Lobby1
Client
Web
or Hosts name of Lobby Server?
Lobby2 Lobby3
③ Lobby1
Game1 Game2 Game3
④ Game1

Game Server Farm
Web Server Farm○
Lobby1
Client
or Hosts name of Lobby Server?
Lobby2 Lobby3
③ Lobby1
Game1 Game2 Game3
④ Game1
Web Server Farm
Web1 Web2 Web3
L4
Virtualization

Authentication system○
Client
① ID & Password for authentication?
Web Server Farm
Web1 Web2 Web3
L4
Virtualization
② Authentication Cookie

Authentication system○
Client
① ID & Password for authentication?
Web Server Farm
Web1 Web2 Web3
L4
Virtualization
② Authentication Cookie
③ Authentication Cookie = ? Auth1
GameUserDB

Authentication Cookie●
• 7aec312v1285345f31073df1e745caf6aeb0d3
cc0c9690b100b6c3011898a5b0d3fa22a98f52
cb6xf4c53e0344a1fe20c75670b01c310daab9
5a31b75f426a48722c7fbc16f8ec05e7988d36
8b94735b017048d77cffeb060cb20a602f8be3
e8f3ed6f8f43fffddc7bb7cdc6606df9df7a33b6
o42
javascript: document.cookie○

Game Server Farm
Game Database Server○
Lobby1
Client
WebLobby2 Lobby3
Game1 Game2 Game3
Auth1
UserDB
Patch1
GameDB

K Online
Architecture
C/S MMORPG

●
Online game system○
Client Internet Game Server Game DB
① Client display
& Business logic
process
② Client side internet
bandwidth
③ Business logic process
④ Database performance
Bottleneck points

“화면에 보여주는 데이터를 어떻게 효과적으로 처리할지”
●
“이미지 데이터를 어떻게 작게 만들지”
Client display & Business logic process○
“Video RAM을 어떻게 효율적으로 올려둘지”
“맵 사이즈 & 구조에 따라 어떻게 유저를 효과적으로
배치할지”
Bottleneck points

●
Client Display○
Bottleneck points

② 인스턴스법(인스턴스 던전)
●
① 공간 분할법(공간의 지리적 분할)
Game Server & Database Performance○
③ 패러렐 월드 방식
“게임의 세계를 지리적 구조에 근거하여 분할하여 다른
서버 프로세스, 혹은 서버 머신에 처리를 할당한다”
“부하가 높고 유저가 집중하는 부분만에 전용 서버를 할당
한다. 그 부분은 반드시 던전일 필요는 없지만, 일반적으로
던전에서 사용되므로 인스턴스 던전이라고 불린다.”
“보틀넥 발생이 쉬운 백엔드 DB 자체를 여러 개로
병렬화하여 보틀넥을 나눈다.”
Bottleneck points resolve

●
One game server○
 Game DB
 Game Server

●
Space Partition Method○
Game DB
 Game Server  Game Server
 Game Server  Game Server

●
Space Copy Method○
Game DB
 Game Server  Game Server  Game Server  Game Server
“②랜덤으로 선택하게 한다”
“①명시적으로 선택하게 한다”
“③비어 있는 곳을 자동적으로 선택한다”

●
Instance Dungeon Method○
 Game DB
 Game Server
ID#1
ID#2
ID#3

●
Parallel World Method○
Game
DB#1
Game Server
Game
DB#2
Game Server
Game
DB#3
Game Server
Game
DB#4
Game Server

●
Use in combination with Parallel World and Space Copy Method○
Game
DB#1
Game Server Game Server
Game Server Game Server
Game
DB#3
Game
DB#2
Game
DB#4

●
All in One○
Game
DB#1
1
2
3
Game
DB#3
1
2
3
Game
DB#2
1
2
3
Game
DB#4
1
2
3

●
Character Database○
Game
DB#1
Server Server
Server Server
1
2
3
Game
DB#2
Server Server
Server Server
1
2
3
Game
DB#3
Server Server
Server Server
1
2
3
Game
DB#4
Server Server
Server Server
1
2
3
Billing DB
Character DB

Web: 10% < 90%
Game: 90% > 10%
“SSD: Solid State Drive”
Write : Read○
“KVS: Key-Value Store”
How to resolve○
“Database Table Compress”
Compare between Web and Game DB●

Scale horizontally (scale out)●
Database scalability: http://en.wikipedia.org/wiki/Scalability○
“One technique supported by most of the major
database management system (DBMS) products is
the partitioning of large tables, based on
ranges of values in a key field. In this manner, the
database can be scaled out across a cluster of
separate database servers”

Scale vertically (scale up)●
○
“With the advent of 64-bit microprocessors,
multi-core CPUs, and large SMP
multiprocessors, DBMS vendors have been at the
forefront of supporting multi-threaded
implementations that substantially scale up
transaction processing capacity”
Database scalability: http://en.wikipedia.org/wiki/Scalability

“Storage Area Network”
Scale up●
“HDD vs. SSD”
Database: Hardware spec○

“RAID Configuration”
Scale up●
“RPM(Revolutions Per Minute)”
Hardware○
“BUS Interface”

“RPM”
Scale up●
High speed disk○
HDD Spindle [rpm] Average rotational latency [ms]
4,200 7.14
5,400 5.56
7,200 4.17
10,000 3.00
15,000 2.00
Game
Database

Scale up●
“RAID Configuration”
http://en.wikipedia.org/wiki/RAID○

Level
Minimum #
of drives
Read
Benefit
Write
Benefit
Image
RAID 0 2 nX nX
RAID 1 2 nX 1X
RAID 4 3 (n-1)X (n-1)X
RAID 5 3 (n-1)X (n-1)X

Nested (hybrid) RAID●
• RAID 0+1: striped sets in a mirrored set (minimum four
drives; even number of drives) provides fault tolerance and
improved performance but increases complexity.
– The key difference from RAID 1+0 is that RAID 0+1 creates a second
striped set to mirror a primary striped set. The array continues to
operate with one or more drives failed in the same mirror set, but if
drives fail on both sides of the mirror the data on the RAID system is
lost.
• RAID 1+0: (a.k.a. RAID 10) mirrored sets in a striped set
(minimum four drives; even number of drives) provides fault
tolerance and improved performance but increases complexity.
– The key difference from RAID 0+1 is that RAID 1+0 creates a striped set
from a series of mirrored drives. The array can sustain multiple drive
losses so long as no mirror loses all its drives.
From Wikipedia○

RAID 10 versus RAID 5●
• A common opinion (and one which serves to illustrate the
dynamics of proper RAID deployment) is that RAID 10 is inherently
better for relational databases than RAID 5, because RAID 5 requires
the recalculation and redistribution of parity data on a per-write
basis.
• While this may have been a hurdle in past RAID 5 implementations,
the task of parity recalculation and redistribution within modern
storage area network (SAN) appliances is performed as a back-end
process transparent to the host, not as an in-line process which
competes with existing I/O. (i.e. the RAID controller handles this as a
housekeeping task to be performed during a particular spindle's idle
timeslices, so as not to disrupt any pending I/O from the host.) The
"write penalty" inherent to RAID 5 has been effectively masked since
the late 1990s by a combination of improved controller design,
larger amounts of cache, and faster drives. The effect of a write
penalty when using RAID 5 is mostly a concern when the workload
cannot be de-staged efficiently from the SAN controller's write cache.
From Wikipedia○

RAID 10 versus RAID 5●
• The choice between RAID 10 and RAID 5 for the purpose of
housing a relational database depends upon a number of factors
(spindle availability, cost, business risk, etc.) but, from a performance
standpoint, it depends mostly on the type of I/O expected for a
particular database application. For databases that are expected to
be exclusively or strongly read-biased, RAID 10 is often chosen
because it offers a slight speed improvement over RAID 5 on
sustained reads and sustained randomized writes. If a database
is expected to be strongly write-biased, RAID 5 becomes the more
attractive option, since RAID 5 does not suffer from the same write
handicap inherent in RAID 10; all spindles in a RAID 5 can be utilized
to write simultaneously, whereas only half the members of a RAID 10
can be used. However, for reasons similar to what has eliminated the
"read penalty" in RAID 5, the 'write penalty' of RAID 10 has been
largely masked by improvements in controller cache efficiency and
drive throughput.
From Wikipedia○

Scale up●
http://en.wikipedia.org/wiki/SATA○
Name Raw bandwidth(Mbit/s) Transfer speed(MB/s)
eSATA 3,000 300
SATA revision 3.0 6,000 600
USB 3.0 5,000 400
USB 2.0 480 60
Fibre Channel
over optic fibre
10,520 1,000
Fibre Channel
over copper cable
4,000 400
Thunderbolt 10,000 1,250
“Bus Interface”

Performance Charts HDD vs. SSD●
HDD: Write Access Times: score (in ms)○
[SRC] http://www.tomshardware.com/charts/hard-drives-and-ssds,3.html

HDD: Read Access Times: score (in ms)○

SSD: Write Access Times: score (in ms)○
x80

SSD: Read Access Times: score (in ms)○

SSD: Sequential Write: score (in MB/s)○

SSD: Sequential Read: score (in MB/s)○

Angelbird
Crest 6 Master
Samsung
MZ-7PC256N
Samsung
MCCOE64G5MPP
IOMeter Web Server Benchmark 6537.10 6876.00 1241.85
IOMeter 4K Random Reads
Benchmark
37298.00 58946.00 6166.33
IOMeter Database Benchmark 9883.00 8910.00 319.63
CrystalDiskMark Random Read,
4KB(QD =32) [MB/s]
212.00 319.00 25.22
CrystalDiskMark Random Write,
4KB(QD =32) [MB/s]
19.00 154.00 0.76
CrystalDiskMark Sequential Read,
[MB/s]
184.00 403.00 100.54
CrystalDiskMark Sequential Write,
[MB/s]
262.00 537.00 78.96
CrystalDiskMark Random Read,
4KB(QD =1) [MB/s]
33.00 26.00 24.27
CrystalDiskMark Random Write,
4KB(QD =1) [MB/s]
512.00 121.00 0.74
SSD: Compare with Angelbird and Samsung○
x5
x6
x30
x8
x25
x1.8
x3.3
x1.3
x691
x5
x9
x27
x12
x202
x4
x6
x1
x163

ORCA Disk Run Percent○
HDD
SSD

RAID configuration for SSD●
Papers○

Flash SSD상의 RAID 구성의 고찰●
Papers○
 Intel’s iOmeter를 이용하여 테스트

Flash SSD상의 RAID 구성의 고찰●
Papers○

Scale out●
“a-1, b-2, … , z-26”
Distribute Game Database System by Partitioning○
RDBMS
.
.
.
Game DB(1)
Game DB(26)
Login Server Lobby ServerUser
.
.
.
.
.
.
.
.
.

Scale out●
“Hash(user id) = n”
Distribute Game Database System by Partitioning○
Hash(user_id) = n
RDBMS
.
.
.
Game DB(1)
Game DB(26)
Login Server Lobby ServerUser
.
.
.
.
.
.
.
.
.

Game Server Farm
View of high availability○
Lobby1
Client
Web
Lobby2 Lobby3
Game1 Game2 Game3
Patch1Auth
Database Server Farm
GameDB UserDB
M M
S S
Replication

Scale Out●
“KVS(Key-Value Store)”
Database: Middleware○
User RDBMS
Game DB
Login Server Lobby Server
.
.
.
.
.
.
.
.
.
.
.
.
KVS Server

Cache●
Write-through vs. Writer-back○
“Write-back(Write-behind)”
Write is done synchronously both to the cache and
to the backing store.
“Write-through”
Writing is done only to the cache. A modified cache
block is written back to the store, just before it is
replaced.

Write-through vs. Write-back●
• Write-back cache is more complex to implement,
since it needs to track which of its locations have
been written over, and mark them as dirty for later
writing to the backing store. The data in these
locations are written back to the backing store only
when they are evicted from the cache, an effect
referred to as a lazy write. For this reason, a read
miss in a write-back cache (which requires a block
to be replaced by another) will often require two
memory accesses to service: one to write the
replaced data from the cache back to the store,
and then one to retrieve the needed datum.
http://en.wikipedia.org/wiki/Cache_(computing)#Writing_policies○

Write-through vs. Write-back●

http://ihchoi.tistory.com/11○
Compress of Database Table●
* 샘플 데이터 1000만 건
Page 압축 時 48.5% 감소,
Row 압축 時 77.8% 감소

CPU 118% 조금 증가,
Page 압축 時 35.6% 감소,
Row 압축 時 32.8% 감소

CPU 235% 증가,
Page 압축 時 249.8% 증가,
Row 압축 時 150.8% 증가
* 100만 건 입력 시간 비교

Moor’s law
Multi-core processor
Memory
SSD
Virtualization
Scale-out vs. Scale-up●

② 5개 패러렐 월드로 분할하여 하나의 월드는 동시 접속
6,000을 허용, 합계 3만의 동시 접속을 허용한다
① 과금 인증 시스템은 공통으로 한다
시스템 기본 구조도의 작성○
③ 하나의 월드를 8개의 지역으로 분할한다(8코어)
④ 하나의 월드당 360개의 인스턴스를 준비한다(16코어)
K Online Game System●

Billing System
K Online
World2 W3 W4 W5
WEB
1
5
1
World1
2
6
2
3
7
4
8
3
6
0
Field
Instance
Game DB
CU 6,000名

결제SV
Backend
DBMS
CU 6,000名
Backup
dbsv
DBMS
Backup
logsv
worldsv
authsv
Frontend Internet
loginsv
gmsv1
msgsv
gmsv2
gmsv3
proxy
proxy
proxy
proxy
proxy
cli
cli
cli
공간 분할법을 이용했을 경우의 프로세스 관계도○

Case Study●
Game Client
Web Board

W Publishing Platform●
WAG: Authentication
WPG: Payment
WPIN: Personal Identification Number
WOTP: One Time Password
Game Security○
WPCBG: PCBang
WIRG: Item Register
Game Support○

W Publishing Platform●
WAG
CashDB
WPG
WPCBG
PC Bang
DB
Game
Server
XML-RPC
JSON
AVRO
Channel
Game
Client
AuthDB
JDBC
JDBC
x

XML-RPC is a remote procedure call (RPC) protocol which uses XML
to encode its calls and HTTP as a transport mechanism
JSON-RPC is a remote procedure call protocol encoded in JSON. It is
a very simple protocol (and very similar to XML-RPC), defining only a
handful of data types and commands.
http://en.wikipedia.org/wiki/XML-RPC○
http://en.wikipedia.org/wiki/JSON-RPC○
Remote procedure call (RPC) protocol●
Avro is a remote procedure call and serialization framework
developed within Apache's Hadoop project. It uses JSON for defining
data types and protocols, and serializes data in a compact binary
format.
http://en.wikipedia.org/wiki/Thrift_(protocol)○

W Personal Identification Number●

How to hack Diablo 3,
I’ve OTP?

2channel Authentication system●
Case by Dream Security○

2channel Authentication system●
Compare○
2channel + 2factor 1channel + 2 factor 2channel + 2factor
ARS

SHA-3●
http://csrc.nist.gov/groups/ST/hash/sha-3/Round3/index.html○

SHA-3●
• The NIST hash function competition is an open competition
held by the US National Institute of Standards and
Technology for a new SHA-3 function to replace the older
SHA-1 and SHA-2, which was formally announced in the
Federal Register on November 2, 2007.
• NIST has selected five SHA-3 candidate algorithms to advance
to the third (and final) round:
 BLAKE
 Grøstl (Knudsen et al.)
 JH
 Keccak (Keccak team, Daemen et al.)
 Skein (Schneier et al.)
http://csrc.nist.gov/groups/ST/hash/sha-3/Round3/index.html○

SHA-3●
• NIST noted some factors that figured into its selection as it
announced the finalists:
 Performance: "A couple of algorithms were wounded or eliminated by
very large [hardware gate] area requirement – it seemed that the area
they required precluded their use in too much of the potential
application space."
 Security: "We preferred to be conservative about security, and in some
cases did not select algorithms with exceptional performance, largely
because something about them made us 'nervous,' even though we
knew of no clear attack against the full algorithm."
 Analysis: "NIST eliminated several algorithms because of the extent of
their second-round tweaks or because of a relative lack of reported
cryptanalysis – either tended to create the suspicion that the design
might not yet be fully tested and mature."
 Diversity: The finalists included hashes based on different
constructions, including the HAIFA and sponge hash constructions, and
hashes with different sources of nonlinearity, including S-boxes and the
interaction between addition and XOR.

SHA-3 WINNER●
• NIST announced Keccak as the winner of the SHA-3
Cryptographic Hash Algorithm Competition and the new
SHA-3 hash algorithm in a press release issued on October
2, 2012. Keccak was designed by a team of cryptographers
from Belgium and Italy, they are:
– Guido Bertoni (Italy) of STMicroelectronics,
– Joan Daemen (Belgium) of STMicroelectronics,
– Michaël Peeters (Belgium) of NXP Semiconductors, and
– Gilles Van Assche (Belgium) of STMicroelectronics.

8F Seung Kwang Bldg., 143-8 Samsung-Dong, Gangnam-Gu, Seoul, 135-877, Korea
신 승 민 / Seungmin Shin
TEL: +82-2-2050-4307
E-MAIL: joo @ windysoft.net
감사합니다!
CONTACT ME

[KGC 2012] Online Game Server Architecture Case Study Performance and Security

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Destaque

Destaque (20)

Semelhante a [KGC 2012] Online Game Server Architecture Case Study Performance and Security

Semelhante a [KGC 2012] Online Game Server Architecture Case Study Performance and Security (20)

Mais de Seungmin Shin

Mais de Seungmin Shin (6)

Último

Último (20)

[KGC 2012] Online Game Server Architecture Case Study Performance and Security