1. SERVICE PRODUCTION
Walter Liu
2012/06/11
Confidential | Copyright 2012 Trend
1 12/24/2012
Micro Inc.

2. About me
• Architect in Core Tech WRS
• Trend Micro 2007~ Now
• Chief Technical Director in Netgame Dep.
• Softstar Inc. Taiwan 1998~2007
• Expertise:
• Backend service development and operation

3. It’s coming ……

4. The beginning
Confidential | Copyright 2012 Trend 4
12/24/2012 Micro Inc.

5. Assault - Error 12 !!!
Confidential | Copyright 2012 Trend 5
12/24/2012 Micro Inc.

6. The nightmare Error 37

7. Out of stock in Taiwan D3 packages
• D3 packages are out of stock in largest 3 convenient
stores in Taiwan and all game shops.
• Some people shared they finally got packages in Jibei
island.
• Many people shared they pursued after trucks of
convenient stores to get the packages.
Confidential | Copyright 2012 Trend 7
12/24/2012 Micro Inc.

8. Out of Stock of Taiwan Game Card
Confidential | Copyright 2012 Trend 8
12/24/2012 Micro Inc.

9. Blizzard Korean lower priority of
Taiwan IP
Korean IP
Taiwan IP

10. Questions?

11. What’s the most important
things to users?
Confidential | Copyright 2012 Trend 11
12/24/2012 Micro Inc.

12. What mistakes Blizzard
make in this D3 service
production？
Confidential | Copyright 2012 Trend 1
12/24/2012 Micro Inc. 2

13. Why Quantity Estimation?
Cost Effective
What if it is wrong?

14. Scalability & Elasticity

15. Fail in Scalability
Performance
1200
1000
800
600
400
200
0
0 1 2 3 4 5 6 7 8 9 10 11

16. EPIC Fail in Scalability
Performance
1200
1000
800
600
400
200
0
0 1 2 3 4 5 6 7 8 9 10

17. Ideal Horizontal Scalability
Performance
2500
2000
1500
1000
500
0
0 1 2 3 4 5 6 7 8 9 10 11

18. Elasticity - Unpredictable Traffic

19. Elasticity - Cloud Solution

20. Elasticity – Others
• Elastic Application Architecture.
• Several flexible hardware providers.
• Flexible ISPs and pricing.
• ……

21. Customer Service and Social
Communication
• Bz is doing pretty bad.
• FB Event: Closed-beta account in Taiwan.
• Build the image to help their users. Like,
• No explanation about incidents.
Confidential | Copyright 2012 Trend
VS.
2
12/24/2012 Micro Inc. 1

22. Incident happens
- Especially your service goes production.

23. Avengers Assemble !!!

24. Fast and Responsive Organization
Teams
Effective
Awareness
Communication

25. Fast and Responsive Process
• Incident management
• Problem management

26. Some other practices
• Interlock with related teams at beginning/middle.
• Customer service prepares resources for burst incoming
calls/tickets.
• Customer service prepares training for the new service/product/
• Data Center team gives out advices and plan for your project.
• Recruit a Service Manager
• Fail case: something that not belong to any team.
• Have someone responsible for whole service.
• Update/patch/change SOP
• Fail case: Service changed, but your CS don’t know about it. Your
customers are confused when they call your CS.
• Fail case: Service changed and caused some trouble, but your
service manager said he didn’t decide/say it.

27. Not related to these Diablo 3 failures, but
important to any system
• Availability
• Security
• Easy to administrate
• System Health/Statistics Monitoring
• Easy Deployment
• Easy Configure
• Risk Management
Confidential | Copyright 2012 Trend 2
12/24/2012 Micro Inc. 7

28. 工商服務時間
• Web Reputation Service
• Parental Control and Productivity Control
• 像是Hinet色情守門員，防毒軟體的孩童防護鎖
• 或者公司不希望員工上班看色情、賭博等網站
• Web Threat Protection
• 防止惡意網頁如病毒/木馬下載、釣魚網站等等。
• Advanced Persistent Threat
• 鎖定特定目標
• 假冒信件或者其他
• 低調且緩慢
• 客製化惡意元件
• 安裝遠端控制工具
• 傳送情資
Confidential | Copyright 2012 Trend 2
12/24/2012 Micro Inc. 8

29. WRS – Parental Control

30. WRS – Web Threat Protection

31. Funny Diablo 3 sales on Taobao, China
Confidential | Copyright 2012 Trend 3
12/24/2012 Micro Inc. 1

32. Thank You!

33. Risk Management
• Identify Critical Failure
• Develop a feasible plan to stabilize customer’s
satisfaction.
• Workaround.
• Rollback.
Confidential | Copyright 2012 Trend 3
12/24/2012 Micro Inc. 3

34. Quantity Estimation
• Goal: Cost Effective Quantity Estimation
• For estimating
• For wrong estimation
• What if the estimation is not correct?
• Too few
• Too many
Confidential | Copyright 2012 Trend 3
12/24/2012 Micro Inc. 4

35. Scalability & Elasticity
• Scalability
• Is your application horizontal scalable?
• Elasticity
• Speed of commissioning / decommissioning
• Max amount of resource can be brought in
• Granularity of usage accounting
• Develop the plan for high traffic.
Confidential | Copyright 2012 Trend 3
12/24/2012 Micro Inc. 5

36. Dare to fail
• Fail is inevitable for fast changing application.
(Web, service)
• Fast changing
• Create dare-to-fail process and environment
• Facebook
• Backup plan
• Rollback plan