The introduction of microservices, Kubernetes, and cloud technology has provided many benefits for developers. However, the age-old problem of getting user traffic routed correctly to the API of your backend applications can still be an issue, and may be complicated with the adoption of cloud native approaches: applications are now composed of multiple (micro)services that are built and released by independent teams; the underlying infrastructure is dynamically changing; services support multiple protocols, from HTTP/JSON to WebSockets and gRPC, and more; and many API endpoints require custom configuration of cross-cutting concerns, such as authn/z, rate limiting, and retry policies. A cloud native API gateway is on the critical path of all requests, and also on the critical path for the workflow of any developer that is releasing functionality. Join this session to learn about the underlying technology and the required changes in engineering workflows. Key takeaways will include: A brief overview of the evolution of API gateways over the past ten years, and how the original problems being solved have shifted in relation to cloud native technologies and workflow Two important challenges when using an API gateway within Kubernetes: scaling the developer workflow; and supporting multiple architecture styles and protocols Strategies for exposing Kubernetes services and APIs at the edge of your system Insight into the (potential) future of cloud native API gateways https://qconlondon.com/london2020/presentation/future-cloud-native-api-gateways

1. The Future of Cloud-Native API
Gateways Richard Li

2. 2
Edge: The boundary between
your data center and your
client(s).

3. 3
Thesis: The evolution of the edge
has been driven by application
architecture.

4. 4
~1995

5. Application Architecture in the ‘90s
5

6. Hardware Load Balancer
6
User Systems administrators
Purpose High availability / scalability
Key Features
Load balancing (round robin,
sticky sessions)
Health checks

7. 7
~2000

8. Similar application architecture
8

9. 9
20022001

10. Sidebar: L4 vs L7
10

11. Software Load Balancer
11
User
Systems administrators (“pre
DevOps”)
Purpose High availability / scalability
Key Features
Load balancing
Health checks
Observability

12. 12
~2005

13. 13

14. 14
Ecommerce
0
75
150
225
300
2000 2005 2010 2015 2019
Amazon Revenue,
Billions

15. 15

16. The Application Delivery Controller
16

17. Application Delivery Controllers
17
User Systems administrators
Purpose
High availability and application
acceleration
Key Features
SSL offload, caching,
compression + load balancing

18. 18
https://www.flickr.com/photos/pchow98/5518600886

19. 19
~2010

20. The proliferation of APIs
20
2005: API launched 2008 2009

21. API Gateway (1st Gen)
21
User
Systems administrators & API
developers
Purpose
Expose business APIs to broader
ecosystem (“API management”)
Key Features
L7 routing (e.g., throttling),
Publishing, Dev Portal, Analytics,
Monetization

22. 22
~2015

23. 23

24. Mini-services
24

25. API Gateway (2nd Generation)
25

26. API Gateway (2nd Generation)
26
User
Systems administrators & API
developers
Purpose
Centralize cross-cutting app
concerns
Key Features
Authentication, rate limiting,
monitoring + routing

27. 27
Cloud-native
applications

28. Cloud-Native Microservices
28
• Service proliferation
(“microservices”)
• Built & operated by
independent application
teams
• Fully elastic

29. App Architecture: A Spectrum of Services
• Different locations (K8S, VMs, FaaS)
• Different protocols (gRPC, HTTP,
WebSockets, TCP)
• Different load balancing requirements
(sticky sessions, round robin)
• Different authentication requirements
• …
29

30. Cloud Gateway
30
Real-time Service
Discovery
Need ADC-like traffic management
capabilities: timeouts, retries, rate
limiting, load balancing, caching, …
Need API Gateway-type
management capabilities:
authentication, developer
portal, metrics, …
1
2
3

31. 31
A spectrum of services
means Cloud Gateways
merge:
Load balancers / ADC functionality +
API management +
Service discovery

32. Microservices portend an even bigger change.
32

33. 33
Developers are on call.

34. Microservices: Full Cycle Development
34
• App teams have full responsibility (and
authority) for delivering a service
• Increases agility by accelerating the feedback
loop.
• Netflix post: https://netflixtechblog.com/full-
cycle-developers-at-netflix-a08c31f83249

35. 35
This is a change in workflow.

36. 36
Thesis: The evolution of the edge has
been will be driven by application
architecture and the application
development workflow.

37. 37
We are doing this to go faster than
ever before!

38. 38
The Edge and
Full Cycle
Development

39. The edge plays a crucial role in full cycle
development
39

40. Full Cycle Development: Release
• Release: Giving your customers access to a specific version of a microservice (and distinct
from deployment)
• Cloud Gateways control release through routing
• Canary release: route 5% of incoming traffic to v1.1, while 95% goes to v1.0
• Instant rollback: change the route back!
• Full cycle developers should define a policy for release
• Ramp traffic from 5% to 100% over 24 hours
• Rollback if more than 0.1% requests result in 5xx errors
40

41. Full Cycle Development: Run
• Run: Making sure your service is available
• Cloud Gateways support running a service via metrics and traffic management
• Metrics: Real-time data about application traffic
• Traffic management: Strategies for resilience and availability such as timeouts,
automatic retries, rate limiting, health checks, and load balancing
• Full cycle developers should define a policy for traffic management
• Automatically retry requests twice if there is no response within 100ms
41

42. A policy is a statement of intent.
• In other words, they’re declarative!
• If you're using Kubernetes, you're writing policies
already.
42

43. Edge Policies
• Policy for traffic management & release
• Designed for full-cycle developers
• Decentralized (one policy per microservice)
• Managed by the development teams (i.e.,
“self-service API Gateway”)
• Operations teams set global policies &
defaults
43
—
apiVersion: cloudgateway.io/v2
kind: EdgePolicy
metadata:
name: target
spec:
prefix: /target/
service: backend
retry_policy:
retry_on: “5xx"
num_retries: 2
release:
strategy: canary
window: 24h
increment: “5%”

44. Cloud Gateways and Full Cycle Development
44
Traffic management policy
+ metrics
Release policy

45. Full cycle development: Code
45
• Your app consists of hundreds of
microservices with an API Gateway,
databases, message queues, and more
• How do you set up your dev environment?
• Can’t run locally — not enough CPU/
memory, and it’s complicated
• Prefer not to run in cloud — slow dev
loop, can't use IDE, etc.
• So what can you do?

46. Route traffic to laptop
46
• Run microservice you're coding on
locally, using your IDE & preferred
toolset
• Run everything else in cloud
• Specify specific requests get sent to
your laptop
• “Developer requests” (e.g., requests
with a specific auth header)
• “Shadow 1% of requests”

47. Full Cycle Development: Deploy
• The deployment cycle for Kubernetes is complicated
• Why expose app dev teams to this abstraction?
47

48. Function-As-A-Service / Serverless
• FaaS: focus on application logic
• Servers, deployment, routing are automatically
handled for the developer
• Application logic lives in a code repository
• The Cloud Gateway provides a natural way to deploy
functions
48

49. Routing from a GitHub repository
49
• GitHub repository contains application code &
deployment metadata
• Cloud Gateway automatically reads GitHub
and deploys code
• Associate release policy with the repository
(service)
—
apiVersion: cloudgateway.io/v2
kind: Project
metadata:
name: example
spec:
host: example.com
prefix: /
githubRepo: dw/example
githubToken: XYZABC
release:
strategy: canary
window: 24h
increment: “5%”

50. Cloud Gateways and Full Cycle Development
50
Traffic management policy
+ metrics
Release policy
Route to laptop
Route from Code

51. Simplifying full-cycle development
51
—
apiVersion: cloudgateway.io/v2
kind: Project
metadata:
name: example
spec:
host: example.com
prefix: /
githubRepo: dw/example
githubToken: XYZABC
release:
strategy: canary
window: 24h
increment: “5%”
—
apiVersion: cloudgateway.io/v2
kind: EdgePolicy
metadata:
name: example-policy
spec:
prefix: /
retry_policy:
retry_on: “5xx"
num_retries: 2

52. Recap
• Microservices changes not just the architecture but the workflow of software applications
• The edge is a critical component of that workflow
• The edge must be managed by developers, and not just operations
• Full cycle developers can use the edge to go much, much faster
• This doesn’t seem like an “API Gateway” any more … cloud gateway? Edge stack?
52

53. 53
Getting started

54. How do I get started?
54
LESS: Locate, Empower, Strangler Fig, Share.
1. Locate a self-contained
edge business function.
2. Empower a full cycle
development team.
3. Strangler Fig your
monolith.
4. Share best practices.

55. Locate
55
• Scope by business capability, not technical
domain (e.g., “recommendation engine” vs
“machine learning algorithm”)
• Start at the edge and work your way in;
foundational services present more
complexity
• Ensure a clear and compelling business
justification for the effort
A self-contained edge business function

56. Empower
56
• Full-cycle development teams must function
independently, so take advantage of this!
• Give team the autonomy to iterate and
experiment to figure out what works best
• Think of a “spinoff”
A full cycle development team

57. Strangler Fig
57
Apply the pattern to your monolith
• Incrementally adopt microservices
without impacting customer
adoption
• Deploy Gateway in front of your
monolith
• As functionality migrates from
monolith to microservices, update
Gateway to route to the
microservices

58. Share
58
• Perhaps the biggest challenge with full cycle
development is training.
• The first full cycle dev team should share
best practices with the next team.
• Create a platform team that codifies best
practices as part of the platform.
Best practices

59. 59
Recap

60. Edge has evolved in response to app arch & workflow
60
• REST API Gateways
• Zuul, Kong, Apigee
1995 2000 2005 2020
• Hardware load
balancers
• F5 BIG/ip (1996)
• Software load
balancers
• HAProxy, NGINX
• Application Delivery
Controllers
• F5, Citrix Netscaler
2010
• Cloud Gateways
• Ambassador/Envoy,
?
Web 2.0, Web Services, & SOA
Monolithic web applications
REST-ful web APIs
Microservices
Cloud-scale
services

61. 61
Cloud-native is a new application …
• Architecture (microservices)
• Development workflow (full cycle development)

62. 62
Locate an edge service.
Empower a full cycle dev team.
Strangler Fig your monolith.
Share best practices.
Start with LESS

63. Questions?
• Richard Li / Twitter: @rdli
• We have a distributed engineering team and we’re hiring!
• https://www.datawire.io/careers/
• Work on both open source & commercial products:
• Telepresence (https://www.telepresence.io), a CNCF project for
local dev with Kubernetes
• Ambassador (https://www.getambassador.io), a cloud-native
gateway for Kubernetes built on Envoy Proxy
63