How to manage the Holes In The Serverless Development Lifecycle. Updated for the AWS Seattle User Group Presentation on 1/29/2019.

1. Is Serverless The New Swiss Cheese?
Chase Douglas, CTO, stackery.io
chase@stackery.io
@txase
MANAGING THE HOLES IN THE SERVERLESS DEVELOPMENT LIFECYCLE

2. My Background
• Brought Multi-Touch Input Support To
The Linux Desktop
• Technical Architect of New Relic
Browser
• CTO and Co-founder of Stackery

3. What I’ve Seen
• Business Impeded Because Infrastructure/
Processes Couldn’t Scale
• Engineering Orgs That Don’t Know What
Has Been Deployed Where
• Spent Years Figuring Out Serverless Best
Practices

4. Most Of You Have Built A Serverless App

5. Some Of You Have Deployed An App In
Production

6. If Your Team Made It To Production,
You’ve Probably Learned A Few Things

7. AWS | LAMBDA FEATURES PAGE
AWS Lambda invokes your code only when
needed and automatically scales to support the
rate of incoming requests without requiring you
to configure anything. There is no limit to the
number of requests your code can handle.
The Promise:
SELF-HEALING SERVERLESS APPLICATIONS | PG2

8. AWS | LAMBDA FEATURES PAGE
The Reality:
AWS Lambda invokes your code only when
needed and automatically scales to support the
rate of incoming requests without requiring you
to configure anything. There is no limit to the
number of requests your code can handle.
s
architecture
sometimes
certain
s
es
every
can
but
areproperly
^
(suggested edits)
SELF-HEALING SERVERLESS APPLICATIONS | PG3

9. DOCKER | WHY DOCKER? PAGE
By combining its industry-leading container engine technology,
an enterprise-grade container platform and world-class
services, Docker enables you to bring traditional and cloud
native applications built on Windows Server, Linux and
mainframe into an automated and secure supply chain,
advancing dev to ops collaboration and reducing time to value.
The Promise (Docker):
SELF-HEALING SERVERLESS APPLICATIONS | PG2

10. SMASH COMPANY | DOCKER IS THE
DANGEROUS GAMBLE WHICH WE
WILL REGRET
At the moment, you need more systems
expertise to use Docker, not less. Nearly every
article you’ll read on Docker will show you the
extremely simple use-cases and will ignore the
complexities of using Docker on multi-host
production systems. This gives a false
impression of what it takes to actually use
Docker in production.
The Reality (Docker):
SELF-HEALING SERVERLESS APPLICATIONS | PG2
If you look for any combination of “docker +
version + filesystem + OS” on Google, you’ll find
a trail of issues with various impact going back
all the way to docker birth.
I didn’t manage to find anyone on the planet
using Docker seriously AND successfully AND
without major hassle.
THE HFT GUY | DOCKER IN
PRODUCTION: AN UPDATE

11. Serverless Powers Success

12. What Do Serverless Apps Look Like?
• Web / Mobile Backend
• Image / Video / Text Processing
• Responding To Notifications
• Real-Time Processing
• Cron Jobs
• Queue Processing

13. Web / Mobile Backends

14. Web / Mobile Backends

15. Real-Time Processing

16. Cron Jobs

17. Queue Processing

18. Serverless Development Lifecycle Gaps
• Managing Infrastructure As Code
• Testing
• Monitoring And Instrumentation

19. Managing
Infrastructure As Code

20. I can point and
click!

22. No more
clicky-click for
you

23. Today, we’re going
to solve the hardest
real-world IaC
challenges!

24. Infrastructure As Code Techniques
• PSA: Permission Scoping
• Namespacing Resources
• Parameterization
• Custom Resources
• Using Existing Resources

25. Aside: We’ll be using AWS
SAM/CloudFormation
The same principles
work elsewhere

26. Definitions
Stack: An instance of an app deployed
from an IaC template
Resource: A component of an app (e.g.
function, NoSQL table, PubSub topic)
specified in an IaC template

27. PSA: Use Scoped
Permissions!
(I’m also using this to level-set on referencing
resources in SAM/CloudFormation templates)

28. Access And Permission Management
HTTPS: //GITHUB.COM/SERVERLESS/EXAMPLES/BLOB/MASTER/AWS-NODE-UPLOAD-TO-S3-AND-POSTPROCESS/SERVERLESS.YML
Access To Do Anything In
Every S3 Bucket In AWS
Account!

29. Access And Permission Management
Need To Scope Access To Specific Actions
Need To Scope Access To Specific
Resources

30. Access And Permission Management

31. Namespacing
Resources

32. But What Is Our
Topic Name?
mystack-mytopic-jshdfkjhf8sd8f8
Random

33. Goal: Understand Three Things From
The Resource Name
Application
Environment
Resource

34. Stack Name: Application-Environment
+
=
backend-prod-notifications

35. Application
Environment
Resource
backend-prod-notifications

36. Parameterization
(Or, How To Not Put Your DB
Password In Your Template)

43. But Where Do I Store
All These Passwords?

47. Custom Resources

48. 1. Provision an API
2. Update a DNS record
Custom Resource Example: DNS Record

49. Custom Resource Example: DNS Record

50. Custom Resource Example: DB Migrations
On every deploy,
migrate the database

51. Custom Resource Example: DB Migrations

52. Using Existing
Resources

53. World’s Simplest Serverless App

54. I can easily deploy different
environments for dev/test

56. But What If I Need To Use
An Existing Table In Prod?

58. But Now I Can’t Easily
Provision Tables For Dev/
Test…

60. What If I Need Multiple
Attributes Of The Resource?

62. This Could Get
Tedious…

63. What About Our Custom
Resource Friends?

65. Testing

66. Testing
Serverless Does Not Change Testing!
Serverless Changes How You Run Tests

67. Testing
Unit Tests: Same As Always
Integration Tests: ???
System Tests: ???

68. Testing
System And Integration Tests: Two
Schools Of Thought
A: Always Test In The Cloud
B: Fake Services For Local Testing

69. Testing
Integration Tests In The Cloud
Pros: Faithful representation, possible
today
Cons: Slower, requires cloud access

70. Testing
Integration Tests Locally With Service Fakes
Pros: Faster, does not require cloud access
Cons: Skew in behavior vs cloud, not very well
supported today
Upstream projects are trying to make this possible/
easier (e.g. AWS SAM Local)

71. Testing
Integration Tests: Advice
(For Today Only!)
If application is only API endpoints +
Functions, do local tests
Otherwise, deploy into cloud and test

72. Testing
With The Right Approach, Serverless Is
Just As Testable As Other Architectures

73. Monitoring And
Instrumentation

74. Monitoring And Instrumentation
How We Do It Today
1. Organization picks a set of monitoring tools
2. Ask everyone to always instrument the same way
3. Pray
4. Draconian measures

76. Monitoring And Instrumentation
How We Should Do It
1. Pick a set of monitoring tools
2. Define instrumentation rules centrally
3. Framework auto-instruments every function
4. Cake

77. Monitoring And Instrumentation
How Can A Framework Auto-instrument?

78. Monitoring And Instrumentation

79. Monitoring And Instrumentation

80. Monitoring And Instrumentation
Now Just Update The Handler:
index.handler => instrumented.handler

81. Serverless Development Lifecycle Gaps
• Managing Infrastructure As Code
• Collaboration Mechanisms
• Testing
• Monitoring And Instrumentation

82. How Will You Manage The Gaps?

83. Build All The Things Yourself

84. Build All The Things Yourself

85. High Performance Serverless Teams
CONFIGURATION MANAGEMENT COLLABORATION CONSISTENCY

86. Use Software That Does It For You

88. Thank you!
Chase Douglas, CTO @ Stackery.io
chase@stackery.io
@txase