The document discusses various challenges and tradeoffs of moving systems and services to the cloud, noting that autoscaling may be easier than expected but other aspects like monitoring, logging, and managing credentials can become more complex. It also touches on dependencies on cloud providers and "glue" services that connect cloud components as well as best practices for testing, defensive coding, and database choices when building cloud-based systems.

1. It’s not always sunny in the clouds
Me,
teetering
Text

2. Mike Subelsky
@subelsky

3. Mike Subelsky
@subelsky

4. 1 / 1 / 08
08:00:00 ET

5. “Everything needs
to be automated”

6. Autoscaling is the easiest
part

7. Think carefully about
credential management

8. You really could use internal
DNS

9. It's maybe not that cheap

10. Launching servers is not
that fast

11. You will become dependent
on “glue” services

12. You will become dependent
on “glue” services

13. You will depend on a distant
faceless provider

14. Use DVCS

15. You will spend a lot of time
on monitoring

16. Your logs will
runneth over

17. Your logs will
runneth over

18. Write lots of
“in-process tests”

19. Write lots of
“in-process tests”
m = Message.find_last_slug_message
secs = m.created_at - m.date_sent
if secs > 600
ErrorNotifier.fatal(quot;Missed slug deliveryquot;,secs,m)
end

20. Snapshots are slow

21. Snapshots are slow

22. Rails will be the
least of your worries

23. Cloud services involve
subtle-yet-massive
tradeos

24. SQS guarantees delivery at
least once

25. SQS guarantees delivery at
least once
def self.guard_against_repeat_delivery(smtp_message_id)
begin
create!(:smtp_message_id = smtp_message_id)
return true
rescue ActiveRecord::StatementInvalid
end

26. SQS guarantees delivery at
least once
def self.guard_against_repeat_delivery(smtp_message_id)
begin
create!(:smtp_message_id = smtp_message_id)
return true
rescue ActiveRecord::StatementInvalid
rows_updated = raw_update(quot;UPDATE application_message_logs
SET last_attempt_at = UTC_TIMESTAMP(),
WHERE smtp_message_id = #{smtp_message_id} AND
last_attempt_at DATE_SUB(UTC_TIMESTAMP(),
INTERVAL 900 SECOND)quot;)
end

27. SQS guarantees delivery at
least once
def self.guard_against_repeat_delivery(smtp_message_id)
begin
create!(:smtp_message_id = smtp_message_id)
return true
rescue ActiveRecord::StatementInvalid
rows_updated = raw_update(quot;UPDATE application_message_logs
SET last_attempt_at = UTC_TIMESTAMP(),
WHERE smtp_message_id = #{smtp_message_id} AND
last_attempt_at DATE_SUB(UTC_TIMESTAMP(),
INTERVAL 900 SECOND)quot;)
if rows_updated 1
logger.warn(quot;Preventing duplicate msg delivery #{smtp_message_id}quot;)
return false
else
return true
end
end

28. Queue lengths inaccurate
for 1000 items

29. SQS not
necessarily FIFO

30. So you may not
want a cloud queue

31. SimpleDB optimized
for writes, not reads

32. You must code defensively

33. There are no good quot;cloud
sandboxesquot;

34. Pay attention to
MySQL timeouts

35. quot;User account management
is
-not- ideal.quot;
-Deacon
Bradley

36. You are locked-in to your
provider

37. Relational DB may not
be the best choice

38. Is there a benefit?

39. Changes the way you write
code

40. You can start
right away

41. Pretty awesome redundancy

42. Time for an example?

43. US-EAST-1C
o
t Ta
me SQS s
a kin
en e
Fil ars g
3P
S
Postfix MailSender MailReceiver MailParser Master DB
le
Ra
Fi
w
Mb
x
bo
ox
M
Fil
w
e
Ra
S3
US-EAST-1B
to Ta
e SQS
am skin
en e
Fil ars g
S3 P
Postfix MailSender MailReceiver MailParser Master DB
le
Ra
Fi
w
Mb
x
bo
ox
M
Fil
w
e
Ra
S3

44. US-EAST-1C
o
t Ta
me SQS s
a kin
en e
Fil ars g
3P
S
Postfix MailSender MailReceiver MailParser Master DB
le
Ra
Fi
w
Mb
x
bo
ox
M
Fil
w
e
Ra
S3
X XX
US-EAST-1B
to Ta
e SQS
am skin
en e
Fil ars g
S3 P
Postfix MailSender MailReceiver MailParser Master DB
le
Ra
Fi
w
Mb
x
bo
ox
M
Fil
w
e
Ra
S3

45. X
US-EAST-1C
o
t Ta
me SQS s
a kin
en e
Fil ars g
3P
S
Postfix MailSender MailReceiver MailParser Master DB
le
Ra
Fi
w
Mb
x
bo
ox
M
Fil
w
e
Ra
S3
X
US-EAST-1B
to Ta
e SQS
am skin
en e
Fil ars g
S3 P
Postfix MailSender MailReceiver MailParser Master DB
le
Ra
Fi
w
Mb
x
bo
ox
M
Fil
w
e
Ra
S3

46. US-EAST-1C
X
o
t Ta
me SQS s
a kin
en e
Fil ars g
3P
S
Postfix MailSender MailReceiver MailParser Master DB
le
Ra
Fi
w
Mb
x
bo
ox
M
Fil
w
e
Ra
S3
US-EAST-1B
X
to Ta
e SQS
am skin
en e
Fil ars g
S3 P
Postfix MailSender MailReceiver MailParser Master DB
le
Ra
Fi
w
Mb
x
bo
ox
M
Fil
w
e
Ra
S3

47. X
US-EAST-1C
SMTP Cloud
o
t Ta
me SQS s
a kin
en e
Fil ars g
3P
S
Postfix MailSender MailReceiver MailParser Master DB
le
Ra
Fi
w
Mb
x
bo
ox
M
Fil
w
e
Ra
S3
US-EAST-1B
X
to Ta
e SQS
am skin
en e
Fil ars g
S3 P
Postfix MailSender MailReceiver MailParser Master DB
le
Ra
Fi
w
Mb
x
bo
ox
M
Fil
w
e
Ra
S3

49. Me,recovering

50. Questions?
mike@oib.com
@subelsky
subelsky.com