This document discusses using Akka Streams and Akka HTTP for large scale production applications. It covers why Akka Streams is useful for high performance stream processing. Operationalizing services requires managing the service lifecycle, monitoring pipelines, and ensuring resiliency. Common requirements for production services include managing the service lifecycle, logging, monitoring, and implementing resiliency through techniques like circuit breakers and timeouts. The squbs project provides tools for operationalizing Akka HTTP services at scale through components for lifecycle management, monitoring integration, and resiliency patterns.

1. Large Scale Production Applications
Taking Akka Streams & Akka HTTP to
Akara Sucharitakul, PayPal

2. Why Akka
Streams???
High performance, pure async,
stream processing
Conforms to reactive streams
Simple, yet powerful GraphDSL
allows clear stream topology
declaration
Unifying construct in processing
pipeline

3. Operationalizing your Service
• Service Lifecycle
• Pipelines
• Gates
Monitoring your Service
Now, eat your cake, too!

4. Operationalization Infrastructure

5. Production at Scale… Production What???
• One service is no service
• Services never come alone, they come in
flocks, herds, domains, organizations
• Services come in different platforms,
implementation languages, technologies
• To be manageable in the big scheme of
things, they have to walk like a duck and
quack like a duck
• Three basic requirement sets for
services:
• Service Lifecycle
• Manageability
• Resiliency

6. Service Lifecycle Requirements
• Observing and acting upon service instance
state changes
• Exposing lifecycle state to external
infrastructure
• Manage internal state transitions without data
loss

7. Manageability Requirements
• Logging
• Uniformity of logs across services
• Monitoring
• Collect metrics, consistently across
services and technologies
• Tracing/Correlation
• Within services
• Across services
• Troubleshooting
• Log bad requests
• Log timeouts
• Intrusion detection
• Suspicious calls
• Metering
• How many times are you allowed to call
• Authentication
• Organization’s policies and mechanisms
• Authorization
• Organization’s policies and mechanisms

8. Resiliency Requirements
• End-to-end back-pressure
• Timeouts
• Retries
• Circuit breakers

9. A Simple End-to-End HTTP Stream
Re
q
Res
p
Re
q
Res
p

10. End-to-End Streams
Why???
Provides back-pressure through all
the components
No known places like unbounded
buffers or mailboxes to OOM
Process what you can take
End-to-End Streams, End-to-End
Resiliency

11. Managing the Service Lifecycle
• Extended Content Verification (ECV)
• Enables/disables load balancer traffic
• Provided through admin UI
• Traffic enabled when “Active”
• Perpetual Stream
• Starts when ”Active”
• Stops incoming and drains stream at
“Stopping”
• Stopping is the hard part
• Without losing data in async systems
Starting
Active
Stopping
Stopped

12. The Pipeline
• Provided by Infra, not part of application logic
• Application can override/add/remove
• Separation of concerns
• Allows standardization of request/response handling across large number of applications
• Similar architecture for client and server side
• Pipeline components:
BidiFlow[RequestContext, RequestContext, RequestContext, RequestContext, NotUsed]
RequestContext: [Request, Option[Try[Response]], Attributes]
• Allows processing request, response, or even short-cutting the request from biz logic
• Pipeline assembly by putting BidiFlow components together (atop)
• Fully utilizes Akka Streams fusing

13. Server-Side Pipeline
Re
q
Res
p
Pipelin
e

14. Client-Side Pipeline
Re
q
Res
p
Pipelin
e
Composite
Client

15. Common
Pipeline Components
Logging
Monitoring
Tracing/Correlation
Request/Response Capture
(Troubleshooting)
Intrusion Detection
Authn/Authz

16. Stage Gates Separates one part of flow from another
Synchronous internal state
Custom BidiFlow keeps the gate’s state
synchronous and encapsulated
Best known implementation of
resiliency components
Can be used as pipeline component

17. Gated Flow
Main
Flow
Gated
Flow
Gat
e

18. Common Gates
Circuit
Breaker
Timeout Retry
Monitor
Counters
• Lag
• Message
Loss
• In/out Ratio

19. State Sharing Some gates share state
Gates to shared resource
• Shared local/remote actor
• Shared service/database
Create explicit state holder like
CircuitBreakerState
Share state holder between
materializations

20. Monitoring Akka HTTP Services

21. What to Monitor
• Requests rate
• Open Connections – current number of active stream materializations
• New Connection rate – HTTP stream materialization rate
• In-flight requests
• Stream collapses, and reasons – Client dropping connections, etc.
• Http response codes
• Standard system and network statistics

22. Monitoring Internet-facing services
• Akka HTTP is not the most tolerant
• Routing API makes even stricter assumptions about the
requests
• Pipeline handlers for internet-facing services:
• Request sanitizer – also capture stats on non-compliant requests
• Request logger – capture requests that failed the sanitize
Be tolerant with others and strict with yourself

23. Now, eat your cake, too!

24. squbs is not… A framework by its own
A programming model – use Akka
All or nothing – Components/patterns
can mostly be used independently

25. squbs
Akka for large
scale deployments
Bootstrap
Lifecycle management
Loosely-coupled module system
Integration hooks for logging,
monitoring, ops integration

26. squbs
Akka for large
scale deployments
JSON console
HttpClient with pluggable resolver and
monitoring/logging hooks
Test tools and interfaces
Goodies:
- Activators and G8 templates
for Scala & Java
- Programming patterns and helpers for
Akka and Akka Stream Use cases…, and
growing

27. squbs Components available on Alpakka
PersistentBuffer
BroadcastBuffer
Stream Circuit Breaker
Stream Deduplicator
Stream Timeout
Stream Retry
http://developer.lightbend.com/docs/alpakka/current/external-components.html

28. What’s Next? squbs 1.0 – on Akka 2.5, Scala 2.12
Refined documentation & API
More goodies, components
Better monitoring & stats
Beyond squbs 1.0 – Operationalizing
distributed patterns at large scale!!!

29. Summary
• Akka Streams & Akka HTTP for high throughput, high burst services
• Resilient: Back-pressure keeps system stable under load
• Operationalization is a big deal!
• Provide the right hooks and tools to understand a running system
• Lifecycle hooks and pipeline allow standards and separation of concern
• Resiliency gates/components essential to building resilient systems
• squbs: Eat your cake, too!
• Functionality without sacrificing performance
• Provides operationalization hooks and components for Akka HTTP/Akka
Streams

30. Q&A – Feedback Appreciated
Join us on – link from https://github.com/paypal/squbs
@squbs, @S_Akara