2016.4.19

1. e 4 M G G TS 3EC
p fl 5IF 5 p
n
n n h
(). , )1

2. • n n
• 5IF “
• i n i n
• 5IF S Z2 TOP SL P P STWbK P
5, e SKW W SGRT 4 ZW 3EC TD m np

3. ”
6TR 8L L4 5IF4
6TR 8L L 5IF n
• 5WLc F
• 5WLc T P T
• 5WLc EPO STQ
• 5WLc 9 E
• 5WLc D TNUFTRS
• 5WLc 5 L

4. z 4 M G Gd z 3ECd

5. 4 M G G d
n i
• i k n
• n
• e g
n “
• n i k n “

6. 4 M G G d
n i
• i k n
• n
• e g
n “
• n i k n “
6TR 8L L

7. 4 M G G
n n n i
n nAI n fl ”
n m n
n n
B P B P n
n n o CF n i n n
o

8. 4 M G G
n n n i
n nAI n fl ”
n m n
n n
B P B P n
n n o CF n i n n
o
-
” “ l
) i) i) i r
)

9. e ~
fl
n r n
” fl
i
“
n
i
i i
n
w
u “ v n

10. e ~
fl
n r n
” fl
i
“
n
i
i i
n
w
u “ v n
)

11. z 4 M G G2
n i
• hyh n
• 2 AP fTa 2 n 3 /2
• k i
• fl
• i

12. 4 M G G w s ff
s “ i
h t
s“~ n i
h fl t

13. z 3EC nd
n p a l –
4 M G G ~ p e o
•
• n o
•
•
• n lF $ EPO STQ $ 9 E ”
K QT

14. 3EC w e u– m

15. 3EC w e u– m

16. 4 M G G TS 3EC
n
• -( n i n i i
n o
• n k
• i n i
• 6TR 8L L 3

17. 4 M G G TS 3EC e

18. 5WLc F $ LNTP
5WLc T P T
5IF = G
5WLc 8b LW 86
5WLc E8F 5 L
5IF ?LWMOL
7? 5ZZ
e e e
n
5WLc
9 E
5WLc
EPO STQ
5WLc LNST P
?PL T R
5WLc
D TNUFTRS
5IF 8L L CTZP T Pe e

19. 5WLc F $ LNTP
5WLc T P T
5IF = G
5WLc 8b LW 86
5WLc E8F 5 L
5IF ?LWMOL
7? 5ZZ
e e e
n
5WLc
9 E
5WLc
EPO STQ
5WLc LNST P
?PL T R
5WLc
D TNUFTRS
5IF 8L L CTZP T Pe e

20. Amazon S3
e o r e
~ b e e ~b
n z n n
11 111111111w
) 6 fl , fl
e
00(0000000
n
n5
n
n6
n n7
HTTP/
HTTPS
e
TD
3EC
e

21. 3RG TS C-
e
B BK WN L
e
5, B C
C T GMK
9G K G
4C
BK WN L
5QT TS
e
9E
C T GMK
9G K G
e
QGW I
D GSWIT K
e
9QGI K
e
G G
KQ SK
C- 3EC e

22. C- e e e g h
e e
e
C- C-
B
BK WN L
5,

23. QGW I
G BK IK
SGRT 4 BK CN L
e oC-~i– a ~
w p g e h
C-

24. QGW I
G BK IK
SGRT 4 BK CN L
e oC-~i– a ~
w p g e h
C-
e

25. QGW I
G BK IK
SGRT 4 BK CN L
e oC-~i– a ~
w p g e h
C-

26. SGRT 4 BK CN L
e oC-~i– a ~
w p g e h
C-

27. i C- o u–

28. 5WLc F $ LNTP
5WLc T P T
5IF = G
5WLc 8b LW 86
5WLc E8F 5 L
5IF ?LWMOL
7? 5ZZ
e e e
n
5WLc
9 E
5WLc
EPO STQ
5WLc LNST P
?PL T R
5WLc
D TNUFTRS
5IF 8L L CTZP T Pe e

29. 3RG TS SKW W 5
n w e r
e e ~ o
Data
Sources
AWS Endpoint App.1
[Aggregate &
De-
Duplicate]
Data
Sources
Data
Sources
C- e
BK WN L
App.2
[Sliding
Window
Analysis]
Availability
Zone
Shard 1
Shard 2
Shard N
Availability
Zone
Availability
Zone
Amazon Kinesis
e
-3F& ,.N%
5
5,

30. 3RG TS SKW W KNTWK
e e 3RG TS C- 3RG TS BK WN L
1 n n F EPO STQ i n
i “
e 1 i n n i i
n .(
1 n n i n

31. 3RG TS SKW W 3SGQ IW
e e CA ~ y

32. n
• n n
•
• n
• EG6 n
• n 2 ,( 6
• n n 2 ). /(( 6
• EG6 n 2 .(( 0(G6
3 BTQQ

33. )( d-
+/
Dynamo
DBS3 Lambda
S3
HBase
EC2
EC2
EC2
...
SQS
SQS
SQS
...
n
EC2
EC2
EC2
...
e e
SKW W

34. -
Dynamo
DB
S3
HBase
EC2
EC2
EC2
...
EC2
EC2
EC2
...
Kinesis
e e
SKW W
n
7?

35. • - n
• )-- n
• ., n
• - ) T P T n
SKW W
x . )3) (

36. • )- y n
•
SKW W ~

37. ip e
0( fl n n n T P T

38. ip e
0( fl n n n T P T

39. 5WLc F $ LNTP
5WLc T P T
5IF = G
5WLc 8b LW 86
5WLc E8F 5 L
5IF ?LWMOL
7? 5ZZ
e e e
n
5WLc
9 E
5WLc
EPO STQ
5WLc LNST P
?PL T R
5WLc
D TNUFTRS
5IF 8L L CTZP T Pe e

40. Amazon Redshift
3RG TS C-
3RG TS BK WN L 4 e
k n i n
k n

41. Amazon Redshift
n
•
• fl
•
n
• k n
• C R PFD? FD?
• 6= n n

42. • n FD?
• E86 n
• n 9G?
• ”i
•
n n $ n
orderid name price
1 Book 100
2 Pen 50
…
n Eraser 70
orderid name price
1 Book 100
2 Pen 50
…
n Eraser 70
gB 4 Ch gBK WN L h

43. BK WN L
l
e
B C
C-
ee
e
3
4
e
e
e
e
e
e
e
e
e
e4 e

44. ?DD
n i

45. ?DD
http://media.amazonwebservices.com/jp/csd20140909/BZ-03.pdf

46. ?DD
http://media.amazonwebservices.com/jp/csd20140909/BZ-03.pdf

47. yn er
8I )
n i
)
n fl n
i
n
i
i ,(

48. 5WLc F $ LNTP
5WLc T P T
5IF = G
5WLc 8b LW 86
5WLc E8F 5 L
5IF ?LWMOL
7? 5ZZ
e e e
n
5WLc
9 E
5WLc
EPO STQ
5WLc LNST P
?PL T R
5WLc
D TNUFTRS
5IF 8L L CTZP T Pe e

49. 3RG TS QGW I G BK IKg Bh
w e :G TT )C G P
e e ~
5IF n
• 6TR 8L L
• n n
n
•
•
• n
•
• i
• FZ
LO Z
5WLc 9 E
)
E= FZ

50. w wa
aws emr create-cluster !
--name bigdata-handson !
--ami-version 3.2.1 !
--applications Name=Hive !
--instance-groups
InstanceGroupType=MASTER,InstanceCount=1,InstanceType=m1.large
InstanceGroupType=CORE,InstanceCount=2,InstanceType=m1.large !
--log-uri s3:/PATH/TO/LOG/ !
--ec2-attributes SubnetId=subnet-a06474e6,KeyName=YOUR_KEY!
--steps Type=HIVE,Name='Hive program’, Args=[-f,s3://PATH/TO/
QUERY.q] !
--auto-terminate!

51. C- e l
• 8:F n F n
• =ACHG BHGCHG 2 p “
hadoop jar YOUR_JAR.jar !
--src s3://YOUR_BUCKET/logs/ !
--dest s3://YOUR_BUCKET/output/!
hadoop jar YOUR_JAR.jar !
--src s3://YOUR_BUCKET/logs/ !
--desct hdfs:///output/ !
• F n F “
• F n n 8:F “

52. Amazon Elastic MapReduce
• LO Z n n
• WLZ PO NP$ ST_P$ ZTR$ PLWT R
” LO Z
• ST_P FD?
EPO STQ fl
iGE5AF:BE H8: H85:
” r
• n
• i
fl
“
Amazon Redshift
• E86
• FD?
• 6= n
• fl n
•
•

53. SQL on Big Data
10 GigE
(HPC)
COPY
UNLOAD
SQL Clients/BI Tools
128GB RAM
16TB disk
16 cores
JDBC/ODBC
128GB RAM
16TB disk
16 coresCompute
Node
128GB RAM
16TB disk
16 coresCompute
Node
128GB RAM
16TB disk
16 coresCompute
Node
Leader
Node
SQL Clients/BI Tools
JDBC/ODBC
Master
Node
Core/Task
Node
Core/Task
Node
Core/Task
Node
READ
WRITE
EMRFSS3 / EMR / DynamoDB / SSH

54. SQL on EMR
Query EngineApplicatio
n
Storage
YARN
Map
Reduce Tez Spark
Hive
Spark
SQL
Presto
JDBC/ODBC
HiveMetastore
HDFS
EMRFS
Hue
Zeppelin
SELECT COUNT(*)

55. e e
T TRS i5WLc F
fl n 5WLc
9 L TN LZEPO NP
C P i
R 86 n
n “
m C=
L L b TN i5WLc EPO STQ
n i
LM PL NSL T ” 6= n

56. e e
T TRS i5WLc F
fl n 5WLc
9 L TN LZEPO NP
C P i
R 86 n
n “
m C=
L L b TN i5WLc EPO STQ
n i
LM PL NSL T ” 6= n

57. 5WLc F $ LNTP
5WLc T P T
5IF = G
5WLc 8b LW 86
5WLc E8F 5 L
5IF ?LWMOL
7? 5ZZ
e e e
n
5WLc
9 E
5WLc
EPO STQ
5WLc LNST P
?PL T R
5WLc
D TNUFTRS
5IF 8L L CTZP T Pe e

58. Amazon QuickSight
n 6= n
5IF n n
5
LZS
FC=79
n n
n
n

61. 4 +)+
F L OL O 9OT T 2
• ) n 1 n ) r
• ( - 6 FC=79 n )( 6 fl
9 P Z T P 9OT T 2
• F L OL O 9OT T i n
n i i58
• , n )0 n )
r
• ( 0 6 FC=79 n )( 6 fl
x n “

62. 5WLc F $ LNTP
5WLc T P T
5IF = G
5WLc 8b LW 86
5WLc E8F 5 L
5IF ?LWMOL
7? 5ZZ
e e e
n
5WLc
9 E
5WLc
EPO STQ
5WLc LNST P
?PL T R
5WLc
D TNUFTRS
5IF 8L L CTZP T Pe e

63. 3RG TS B C
n i i i i
e c e e e
3F G 3F H
e
” e e
(5 )

64. 3RG TS 3 T G
• 5WLc fl E86 F
• n o n
• n n o bFD? - . r
k n
k
.,G6
fl

68. 9 GS

71. e 4 M G G TS 3EC
n
• -( n i n i i
n o
• n k
•
• 6TR 8L L 3

72. Data Science
Amazon Redshift
ETL処理
~ w
K1 SZKS , +/ n
e eAmazon
Kinesis
Amazon
EMR
http://www.slideshare.net/AmazonWebServices/bdt306-how-hearst-publishing-manages-clickstream-analytics-with-aws

73. i o k
tv w