5월 17일 서울COEX에서 열린 AWS Summit Seoul 2016에서 Bespin Global 이한주님이 발표하신 "AWS를 활용한 Big Data 실전 배치 사례" 발표자료입니다.

1. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
베스핀글로벌 이한주 대표
2016.5.17
AWS를 활용한 Big Data 실전 배치 사례
모니터링, IOT의 Big Data 배치 및 Data Visualization 사례

2. About HanJoo Lee
CEO & Cofounder , BESPIN GLOBAL
GeneralPartner,
1998 Cofounded
IT Infra entrepreneur

3. Built and Operated 12 Datacenters
100,000+ 서버 운영
11 Countries with 12 Datacenters around the world: 1998년 시작
Australia Belgium Canada France Germany Korea Netherlands USA UK India Rumania
Sydney Antwerp Vancouver Paris Frankfurt Seoul Amsterdam Chicago London Mumbai Bucharest
Hannover Tampa / Austin
Fort Lauderdale

4. About Bespin Global
베스핀 글로벌
Managed Service Provider
한국에서 최초로 AWS MSP 인증 받은 회사
한국에는 나와있는 3개 MSP 인증 중 2개가 Bespin
• Cloud 전략 수립
• Cloud Architecture
• Cloud Migration
• Cloud 운영
• Hybrid IT Management
• 한국과 중국에서 140명

5. About Bespin Global
Cloud의 길잡이

6. Data lifecycle in Cloud Infra
클라우드 인프라
데이터 애플리케이션
데이터 입력 가공 분석 활용
• IoT를 통해 세상의 모든
것들을 디지털 데이터
로 수집
• 빅데이터 기술을 통해
과거의 텍스트 중심의
정형 데이터뿐만 아니
라 동화상, 음성 등 비
정형 데이터까지 처리
• 대규모 데이터를 실시
간 분석해 결과를 추출
하고 의사결정 진행
• 디지털 골든크로스로서
인간의 두뇌 이상의 의
사결정과 작업을 컴퓨
터가 스스로 수행
머신러닝
빅데이터
IoT

7. Bespin Global
김성수 상무
Building Big Data
Backend Using
Native AWS
Services
WhaTap
김성조 CTO
Building Big Data
System Using
Proprietary File
System while
Using AWS
N3N
김호민 대표
Building Big Data
Visualization Using
Splunk and AWS
Big Data for SaaS based Monitoring Services

8. Bespin Global: How to build and
Operate Big Data System on AWS?

9. Avoid unplanned Downtime
Monitoring Process
Server
DBMS
Application
MonitoringTarget
분석 DB
Basic Process
Advanced Process
(사후 처리)
Metric
Data 수집 및
모니터링
Big Data
Processing
※ Ca사 모니터링 솔루션 UIM 적용
Threshold Alarm
사전 예방 활동
Scale up
Scale out
(Time Series, Correlation, …)
알람 or 장애 조치
Monitoring Dashboard

10. Building Big Data System on On-Premise
R Analysis
( Time Series / Correlation )
Pre detection
DatabaseDistributed
Messaging System
…..
M N1 N2 Nn
Cluster Computing
Framework
Yarn
…..
HDFS
Time Series Database
N2 Nn…..M N1
Analysis Platform
……
Data Sources
Application
Server
DBMS
LogAgent
…
Applications
BI
Marketing
Advertisement
…
Limited Scalability
Internal IT Resources to Manage Cluster (Tuning and monitoring, etc…)
Upfront capital expense
NoSQL
…
사전 진단/예방
Producer
Stream
Collector
Consumer
Stream

11. Building Big Data System on AWS
Data Sources
Application
Server
DBMS
LogAgent
…
Producer
Stream
Kinesis
Streaming
Data Platform
EMR AWS ElasticSearch
Time Series Database
RDS
Pre detection
KCL
Kinesis S3 Connector
Consumer
S3
Lambda
Archiving Data
EMR
Redshift
RDS
Analysis(BISolutions)
Application
Upsell Analysis
Marketing
Advertisement
…
Elastic and Highly Scalable
Don’t Manage Cluster (and AWS has tuned Services)
Easy to Use and Deploy to Multiple Locations
No upfront capital expense – Pay as you go
Cluster monitoring
Cluster monitoring
R Analysis
( Time Series / Correlation )
R on EMR
Consumer
사전 진단/예방

12. On-Premise vs AWS
R Analysis
( Time Series / Correlation )
Database
Streaming
Data Platform
Kinesis
Time Series Database
Elastic Search
Easy to Use and Managed Console
RDS
Database
Complex to Use Managed Cost+
Cluster Managed Cost (Tuning and monitoring, etc…)+
Elastic Scalability
Don’t Managed Cluster (and AWS has tuned Services)
Upfront capital expense
Pay as you go
Distributed
Messaging System
…..
M N1 N2 Nn
Cluster Computing
Framework
Yarn
…..
Time Series Database
N2 Nn…..M N1
Limited Scalability
EMR
R Analysis
( Time Series / Correlation )
R on EMR

13. WhaTap: Running Cloud Based
Monitoring Service on AWS

14. Monitoring Issue Changes
Testing Open Stable
Application Physical SystemNeeds
Unknown
bottlenecks
Known issues

15. Monitoring Solutions
Needs
Testing Open Stable
APM
(분석, 문제 해결, 실시간)
SMS
(관제, 경고 이벤트)

16. The differences on Cloud
Dedicated Unix
Infra
개별 서버 성능
서버 관리 유연성
서버(노드) 수
시스템 처리 성능
부분 오류 가능성
Cloud Infra
서비스 안정성

17. Monitoring Service(SaaS based)
Master
security group security
1
2
200 Slave
Monitoring Service

18. APM on Premise vs on Cloud
• Data collection per several seconds
• Active Transaction Analysis
• All Transaction Profiling
On Premise Solution
• Data collection per 10 seconds
• Active Transaction Analysis
• Selective Profiling & Integrative Analysis
On Cloud Service

19. 사용자 접속 정보,
트랜잭션, 자원, 튜닝 정보를
하나의 관점으로 비교 분석할 수
있어야 합니다. Archiving
Tech
User (browser)
Transaction
IP, 도시/국가,
접속 매체, OS,
최근 방문자,
액티브 사용자
TPS, Response
time, Error
time, URL, SQL
Resource
CPU, Heap,
Disk, GC
Tuning
History 비교분석
Active Stack 분석
Hit Map 분석
Integrated analysis of perf. data

20. EC2 instance
Data
Server
EBS
Backup (S3)
Staticstics
RDS (MySql)
Scalable APM (AWS based)
EC2 instance
Data
Server
EBS
EC2 instance
Data
Server
EBS
EC2 instance
Data
Server
EBS
Scalable!!

21. Elastic Load
Balancing
Amazon
Route 53
Auto Scaling group
EC2 instance
Data
Server
EBS
Backup (S3)
Project/Tenant Mgmt
RDS (MySql)
Staticstics
RDS (MySql)
ElastiCache
(REDIS)
Elastic Load
Balancing
Amazon
Route 53
Auto Scaling group
Region (Tokyo)
Elastic Load
Balancing
Auto Scaling group
EC2 instance
Data
Server
EBS
Backup (S3)
Staticstics
RDS (MySql)
Region (US.West)
ElastiCache
(REDIS)
Elastic Load
Balancing
Auto Scaling group
Region (Tokyo)
Region (US.West)
Elastic Search
Scalable APM (AWS based)

22. APM to SMS on Cloud
Needs
Testing Open Stable
APM
• 분석, 문제 해결
• 실시간
• 대용량 데이터
SMS
• 관제, 문제 인지
• 경고 이벤트
• 소규모 데이터

23. SUPPORTS
MOBILE
LOWER
COST
SHORTER
IMPLEMENTATION
PERIOD
SMS on Cloud
1 Month
EXISTING MONITORING
CLOUD MONITORING
5 Minutes
$10000 + α
EXISTING MONITORING
CLOUD MONITORING
$100/mon
EXISTING MONITORING
CLOUD MONITORING
In Office
Anywhere

24. WhaTap TokyoWhaTap Virginia
WhaTap Oregon
출처 : https://rctom.hbs.org/
WhaTap can service to global customers

25. Big Data Visualization on AWS

26. N3N: What We Do
N3N provides IOT Visualization for Fortune 500 Companies and Cities Governments.

27. N3N - Physical Visualization
Global
Asia
Korea
Seoul
Suwon
Z: Hierarchies
Daejeon
X: Dependencies Y: Relationships

28. N3N - Logical Visualization
Business Unit
Service
Application
X: Dependencies Y: Relationships
ApplicationData Bases
Z: Hierarchies

29. N3N - Business Impact
QuantitativeBenefits
KPI Before After Result Remarks
MTTD, MTTR 2 days 1.5 days -0.5 day CISCO Stat,
Splunk .conf 2014
Big Data Solution Usage Rate 5% 100% +95%
Improvements in Stability &
availability
+25%
Reduction in Operational Cost +10%
25% 95% 25% 10%

30. Wrap Up

31. Visualization Big data for Biz Decision
Monitoring Log
IOT event Log
System Event Log
AWS event Log
Visualized
Analysis On
AWS

32. Wrap Up
- Big Data on AWS – Monitoring Use Case
- Not only for Monitoring
- Merchandise, Logistics, Turnover, Marketing, etc.
- Best Practice & Consulting from Bespin Global

33. WE ARE HIRING!!!!
• KOREA
• CHINA
• JAPAN
• USA
PLEASE JOIN US

34. 감사합니다!
HELPING YOU ADOPT CLOUD.