20190306 AWS Black Belt Online Seminar Amazon EC2 スポットインスタンス

1. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
AWS Webinar
https://amzn.to/JPWebinar https://amzn.to/JPArchive
EC2
2019-03-06
Amazon EC2
[AWS Black Belt Online Seminar]

2. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
( )
/ EC2
EC2
EC2
AWS
• Amazon EC2 Auto Scaling
• AWS Support

3. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
AWS Black Belt Online Seminar
•
•
S
Twitter
#awsblackbelt

4. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
• 2019 3 6
AWS (http://aws.amazon.com/)
• AWS
AWS
•
• AWS does not offer binding price quotes. AWS pricing is publicly available and is subject to
change in accordance with the AWS Customer Agreement available at
http://aws.amazon.com/agreement/. Any pricing information included in this document is
provided only as an estimate of usage charges for AWS services based on certain information
that you have provided. Monthly charges will be based on your actual use of AWS services, and
may vary from the estimates provided.

5. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
• ( ) ( )
•
• ( )
• ( )
• CloudWatch ( )
•
• ( ) :
• ( )
• ( )
• ( )
• ( ) ( )

6. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark

7. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
- Amazon EC2
A 1
0 (
2 /
/
/
nEC
)%
/ /
(
/
o 3 z
/ /
m a1 1

8. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
E 2 C
C
C
C

9. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Amazon EC2
• 9
•
-
HPC/
•
1,000
•
• API
•
•

10. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark

11. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
- 110
• z ig(s lw
)
• vn z
: cu
• cu AWS
tm
•
• 2017 8h26 ra
vCPU 110
( y)
• e
bp
• o ( )
22/1 1 .- . . 1 1 - 2 - / . 11 -
2 1.- - 1 2 .- / 1 1/.2 -12 - 1

12. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
- 110
• i(u mny
)
• z p l
ew
• ew AWS
b vo
•
• 2017 8 26 tc
vCPU 110
(a )
• : : : : g
r h
• ( : )
: s
22/1 1 .- . . 1 1 - 2 - / . 11 -
2 1.- - 1 2 .- / 1 1/.2 -12 - 1

13. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
H L
I NR
5
/ H FGT
0 D
) 2
F 2 I P
2
7 D
2 C D
C D
cA ad
C A
8 3
D
( 2
% I P
D
D /
50877 D
) 2
(
( E 2
Y a % da &

14. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
•
• ( )
•
•

15. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
AWS OSS,
AWS
CloudFormation
AWS
OpsWorks
Amazon
EMR
Amazon
ECS
AWS Data
Pipeline
AWS BatchAuto Scaling
AWS Thinkbox
Deadline

16. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark

17. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
- 2017 11

18. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
(1)
1 ) 4
1- 1 ) 1
. ) . ) ( . )
1 ) 4 (
. ) . ) ( . )

19. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
(2)
1 ) 4
1- 1 ) 1
. ) ( . )
1 ) 4 (
. ) ( . ). ) . )

20. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
(3)
4 .,
(4 4 ,- 4
1 -, 1 -,
4 .,
1 -, 1 -,1 -, 1 -,
T R T )
A

21. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
(4)
1 ) 4
1- 1 ) 1
. ) ( . )
1 ) 4 (
. ) ( . )
10
10
10
10
. )
10
. )
10
$
.
R
$

22. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
(1)
6 4 -
1 )-.0 4
2 .- 2 .-
143
143
2 .-
143
$
20. o m ks i Ra
ce
g lce r y
• h
2 .-
• T p c
e nt ks $ ($
o
.

23. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
(2)
- 67 1 - -
73 7 0276
( 4- 0 .( 4- 0
021
021
( 4- 0
021
$
R T R T c ke
R T R R gr
• n - .( 4- 0 ao
• hm R T
$ $ $ )$
p
• R T c
R l is
ao
.

24. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
(1)
4 -
1 )-. 4
2 .-
$
2 .-2 .-
4 ( C e T T ET T C
e A
• a RT
2 .-C
c C

25. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
(2)
- 7 1 - -
73 7 027
5( 4- 0 .( 4- 0
042
042
( 4- 0
042
)
(
hg R i T
• c - .( 4- 0
R
• R $ $ $
• R $ $
$ $ )$
1. 3
$
e a

26. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
•
• AWS
2
• Amazon EC2
• ( )

27. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark

28. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
• 2
•
•
• CloudWatch Events

29. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
-
•
•
• http://169.254.169.254/latest/meta-data/
• https://docs.aws.amazon.com/ja_jp/AWSEC2/latest/UserGuide/ec2-
instance-metadata.html
•
• http://169.254.169.254/latest/meta-data/spot/instance-action
•
• 5
• https://docs.aws.amazon.com/ja_jp/AWSEC2/latest/UserGuide/spot-
interruptions.html

30. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
{
"version": "0",
"id": "12345678-1234-1234-1234-123456789012",
"detail-type": "EC2 Spot Instance Interruption Warning",
"source": "aws.ec2",
"account": "123456789012",
"time": "yyyy-mm-ddThh:mm:ssZ",
"region": "us-east-2",
"resources": ["arn:aws:ec2:us-east-2:123456789012:instance/i-1234567890abcdef0"],
"detail": {
"instance-id": "i-1234567890abcdef0",
"instance-action": "action"
}
}

31. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
- CloudWatch Events
. / . - . / / . / / / / 22 / /

32. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
- CloudWatch Events

33. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark

34. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
AWS
CLI
(aws ec2 ...) (EC2)
run-instances
(request-spot-
instances)
one-time (sync) /
persistent (sync)
request-spot-
fleet
/
vCPU
( )
request (async) /
maintain (async)
EC2
create-fleet /
vCPU
( )
instant (sync) /
request (async) /
maintain (async)
Auto Scaling
(New!)
- ,
-

35. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
m4.large Spot ASG Min: 1 Max: 10
m5.large Spot ASG Min: 1 Max: 10
c4.xlarge O-D ASG Min: 1 Max: 10
Availability
Zone 1
Availability
Zone 2
Availability
Zone 3
- C AC
ASG
ASG 2
E
- C
C
2ASG E
ASG: Auto Scaling Group
O-D: On Demand Instance

36. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
2 / - E
DE S C C G R
m4.large Spot Instances
m5.large Spot Instances
c4.xlarge On-Demand instances
Availability
Zone 1
Availability
Zone 2
Availability
Zone3
ASG SO
GAI S
C C
G R
Single ASG
ASG: Auto Scaling Group
O-D: On Demand Instance
RI: Reserved Instance

37. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark

38. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
AWS
CLI
(aws ec2 ...) (EC2)
run-instances
(request-spot-
instances)
one-time (sync) /
persistent (sync)
request-spot-
fleet
/
vCPU
( )
request (async) /
maintain (async)
EC2
create-fleet /
vCPU
( )
instant (sync) /
request (async) /
maintain (async)
Auto Scaling
(New!)
- ,
-

39. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark

40. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
(1)

41. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
(2)
• Load balancing:
• Flexible workloads:

42. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
(3)
• vCPU
•

43. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
(4)
•
vCPU
•
•

44. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
(5)
• EC2
•

45. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
(6)
•
•
•

46. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
EC2 Auto Scaling Group (1)
• EC2 EC2 Auto Scaling

47. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
EC2 Auto Scaling Group (2)

48. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
EC2 Auto Scaling Group (3)
•
•

49. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
EC2 Auto Scaling Group (4)
•
• 1.
• 2.
• 3.
•
• 12
•
2
•
30:70
•
• 2 +3
• 7

50. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
< > EC2 Auto Scaling Spot Fleet (1)
EC2 Auto Scaling Spot Fleet
AZ
[1]
1. Total target capacity
2. "Optional On-demand
portion
3.
[2]
1. Desired Capacity
2. "On-Demand base"
3. OD:Spot Ratio
ELB
Launch Template
$latest, $default
[1] https://docs.aws.amazon.com/ja_jp/AWSEC2/latest/UserGuide/spot-fleet.html#on-demand-in-spot
[2] https://docs.aws.amazon.com/ja_jp/autoscaling/ec2/userguide/AutoScalingGroup.html#asg-purchase-options

51. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
• (Termination Policy)
• (Termination
Protection)
•
•
(Entering Standby)
•
(Suspending Processes)
• (Rebalancing
activities)
< > EC2 Auto Scaling Spot Fleet (2) -
•
•
• ( )
( )

52. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
–
1
–
1
1
•
• 1 ( )

53. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
( )
2 =
=
$ $
( (
=
=
))
))
= 4
3 2 1 =

54. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
( )
1
$ $
4
((
((
) 3
= 2 1

55. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
( )
() 1 =3
1= =
1=
=
1=
=
$
2= =
(

56. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
( )
=
3 5
= 3
21
$ $
1 =
3
= 3
((
((
4
) =
3

57. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
( )
3 5
3
) 21
$ $
1 33
4 =
(
3

58. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
( )
() 1 =3
1= =
1=
=
1=
=
$
2= =
(

59. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
• - Amazon EC2 - Amazon EC2 | AWS
https://aws.amazon.com/jp/ec2/spot/pricing/

60. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
(1)
EC2

61. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
(2)
AWS CLI: describe-spot-price-history
$ aws ec2 describe-spot-price-history --instance-types m3.medium
{
"SpotPriceHistory": [
{
"Timestamp": "2019-03-04T21:36:38.000Z",
"AvailabilityZone": "ap-northeast-1a",
"SpotPrice": "0.109600",
"ProductDescription": "SUSE Linux",
"InstanceType": "m3.medium"
},
{
"Timestamp": "2019-03-04T21:36:38.000Z",
"AvailabilityZone": "ap-northeast-1c",
"SpotPrice": "0.109600",
"ProductDescription": "SUSE Linux",
"InstanceType": "m3.medium"
},
:

62. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark

63. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
( )
On Demand Auto-
Scaling group
Output S3
bucket
Availability Zone A
Availability Zone B
Spot Fleet
Upload object
into input S3
bucket
Job SQS
Queue
Auto Scaling group and Spot fleet
scale up based on queue depth and
scale down based on CPU utilization
CW metrics
Workers will
check for jobs
in the queue
Workers will update Job status
(start time, SLA end time, etc) in
DynamoDB
Uploads to S3 will trigger
a Lamda function to put
jobs in SQS and
DynamoDB
EFSEC2 instance
worker fleet
Cost-effective Batch Processing with Amazon EC2 Spot |
AWS Compute Blog
https://aws.amazon.com/blogs/compute/cost-effective-
batch-processing-with-amazon-ec2-spot/

64. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
AWS Batch ( )
JobQueue
submit-job
list-jobs
describe-jobs
cancel-jobs
terminate-jobs
Compute Environment
ECR/Docker
Registry
S3/EFS/NFS
Autoscaling
ECS
Docker Image
JobDefinitio
n
job
AWS Batch

65. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Master
Node (OD
or RI)
S3
EMRFS
Master Fleet
Task
Node
(Spot)
Task
Node
(Spot)
Task
Node
(Spot)
Task Fleet
Instance
Store
EBS
Volume
Instance
Store
EBS
Volume
Instance
Store
EBS
Volume
Core Fleet
HDFS
Instance Fleet
Core Node
(OD or RI)
Core Node
(OD or RI)
Core Node
(OD or RI)
Amazon EMR

66. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Stateless Web
Server (Spot)
Stateless Web
Server (Spot)
Session State Data
Stateless Web Server
(On-Demand)
Elastic Load
Balancing
Amazon EC2
Auto Scaling
ECS
container 1
Stateless Web Server
(On-Demand)
ECS
container 2
ECS
container 1
ECS
container 2
https://github.com/awslabs/ec2-spot-labs/tree/master/workshops/ec2-spot-fleet-web-app

67. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
< > ECS
• c j ECS ga
• ECSS_ _ eh i d E A
(/etc/ecs/ecs.config)
• ECS AMI C
. . / / - - // /
. . / / - : / - /

68. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
< > ECS
( )
• ie do cha
• _ _ _ Draining E
• ( S _)SNS lj
•
→ ilj gC mnA
: . . : / / - - - - - /

69. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
< > ECS
• CloudFormation
•
•
• :
Draining
• SNS E
. - - - - .2 .- / -2/ - 2 / - .-

70. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark

71. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
4+1

72. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
4+1 -
• z ig(s lw
)
• vn z
: cu
• cu AWS
tm
•
• 2017 8h26 ra
vCPU 110
( y)
• e
bp
• o ( )
22/1 1 .- . . 1 1 - 2 - / . 11 -
2 1.- - 1 2 .- / 1 1/.2 -12 - 1

73. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
4+1 -
4 .,
(4 4 ,- 4
1 -, 1 -,
4 .,
1 -, 1 -,1 -, 1 -,
T R T )
A

74. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark

75. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
AWS Auto Scaling
2019 -
AWS Auto Scaling 2019 | http://blog.father.gedow.net/2019/02/05/aws-auto-scaling-2019/
2019

76. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
EC2 Spot Instances Workshops
https://ec2spotworkshops.com/

77. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
( )
• ( ) ( )
•
• ( )
• ( )
• CloudWatch ( )
•
• ( ) :
• ( )
• ( )
• ( )
• ( ) ( )

78. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
( )
• 9
•
•
•
• CloudWatch Events
•
• :
•
•
•
• ( )

79. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Q&A
AWS Japan Blog https://aws.amazon.com/jp/blogs/news/

80. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
https://amzn.to/JPWebinar
3 Black Belt Online Seminar
3/5 ( ) 12:00-13:00 Amazon EC2
3/6 ( ) 18:00-19:00 Amazon EC2
3/12 ( ) 12:00-13:00 AWS Well-Architected Framework
3/13 ( ) 18:00-19:00 Amazon VPC
3/19 ( ) 12:00-13:00 Amazon FSx for Windows File Server/Lustre
3/20 ( ) 18:00-19:00 Amazon EBS
3/26 ( ) 12:00-13:00 Amazon CloudWatch

81. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
AWS AWS
https://amzn.to/JPArchive

82. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
[ ] AWS Innovate
https://amzn.to/AWSInnovateJP

83. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
. / . .- a i m
rh a i m o
• [ W
ld m A ]p
• jc WbnS W
/ : . .
( e)W

84. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
AWS Webinar
https://amzn.to/JPWebinar https://amzn.to/JPArchive