This presentation was made by Mangesh Patankar (Developer Advocate - IBM Cloud) as part of Container Conference 2018: www.containerconf.in. "How do we make microservices resilient and fault-tolerant? How do we enforce policy decisions, such as fine-grained access control and rate limits? How do we enable timeouts/retries, health checks, etc.? A service-mesh architecture attempts to resolve these issues by extracting the common resiliency features needed by a microservices framework away from the applications and frameworks and into the platform itself. Istio provides an easy way to create this service mesh."

1. Make Java Microservices Resilient with Istio
IBM Code Patterns:
http://developer.ibm.com/code
Mangesh Patankar
- Developer Advocate
mapatank@in.ibm.com
@MangeshPatank

2. Agenda
• Evolution Of Microservice
• Relation of Micro services with Container
Orchestrator (K8s)
• Why Service Mesh (Istio)?
• Features of Istio
• Demo

3. Evolution of Microservices

4. Monolithic
Application
to
Microservices
Application

5. Microservices
Application
Scaling

6. Microservices
Application
Update

7. Typically microservices are encapsulated inside containers…
One:One relationship between a microservice and a container
Everyone’s container journey starts with one container….
IBM Cloud - Container Service

8. At first the growth is easy to handle….
IBM Cloud - Container Service

9. Microservices
Application
Interactions
- Simple

10. But soon it is overwhelming…we need container
and microservices management
IBM Cloud - Container Service

11. Enter Container Orchestrator
IBM Cloud Container Service

12. Slide Title Goes Here
Container Stack
Physical InfrastructureLayer 1
Virtual InfrastructureLayer 2
Operating SystemLayer 3
Container EngineLayer 4
Orchestration/Scheduling
Service Model
Layer 5
Development Workflow
Opinionated Containers
Layer 6

13. Kubernetes

14. Slide Title Goes Here
What is Kubernetes?
• Container orchestrator
• Runs and manages containers
• Supports multiple cloud and bare-metal environments
• Inspired and informed by Google's experiences and internal
systems(Borg, Omega)
• 100% Open source, written in Go
• Manage applications, not machines
• Rich ecosystem of plug-ins for scheduling, storage, networking

15. Intelligent Scheduling Self-healing Horizontal scaling
Service discovery & load balancing Automated rollouts and rollbacks Secret and configuration management
IBM Cloud- Container Service

16. Slide Title Goes Here
Kubernetes Architecture
API
UI
CLI
Kubernetes
Master
Worker Node 1
Worker Node 2
Worker Node 3
Worker Node n
Registry
• Etcd
• API Server
• Controller Manager
Server
• Scheduler Server

17. Slide Title Goes Here
Simplified Kubernetes Cluster

18. Kubernetes is great for
Microservices…
Why do we need a Service
mesh and what is it?

19. Kubernetes Service Vs Service Mesh
What a k8s Service component cannot do
• To get more control of the traffic that goes to this API
• To support many API versions
• Do canary deployments
• Watch and keep track of each request that comes in
• L7 metrics
• Traffic Control - Splitting
• Rate limiting
• Resiliency & Efficiency - Circuit breaking
• Visibility
• Security
• Policy Enforcement …….

20. • Lightweight sidecars
to manage traffic
between services
• Sidecars can do
much more
than just load
balancing!
How to build a
‘Service Mesh’ ?

21. Service Mesh
• Linkerd
• Conduit
• Istio
• ….

22. Istio

23. Istio
Concepts
• Pilot - Configures Istio deployments
and propagate configuration to the
other components of the system.
Routing and resiliency rules go here
• Mixer - Responsible for policy
decisions and aggregating telemetry
data from the other components in the
system using a flexible plugin
architecture
• Proxy – Based on Envoy, mediates
inbound and outbound traffic for all
Istio-managed services. It enforces
access control and usage policies, and
provides rich routing, load balancing,
and protocol conversion.

24. Istio
Concepts

25. Istio
Concepts

26. Istio
Architecture

27. What is a ‘Service Mesh’ ?
A network for services, not bytes
" Resiliecny and efficiency
● Traffic Control
● Visibility
● Security
● Policy Enforcement

28. • Istio adds fault tolerance to your application
without any changes to code
• Resilience features
❖ Timeouts
❖ Retries with timeout budget
❖ Circuit breakers
❖ Health checks
❖ AZ-aware load balancing w/
automatic failover
❖ Control connection pool size and
request load
❖ Systematic fault injection
Resiliency

29. Code Pattern
- Kubernetes,
Microservices
and Istio

30. Slide Title Goes Here
IBM Cloud Kubernetes Container Service - IKS

31. IBM Code Pattern
http://developer.ibm.com/code
DEMO

32. Slide Title Goes Here
Twelve-factor apps make a strong case for designing and implementing your microservices for failure. What that means is with the
proliferation of microservices, failure is inevitable, and applications should be fault-tolerant. Istio, a service mesh, can help make your
microservices resilient without changing application code.
Developer Works Code: https://developer.ibm.com/code/patterns/make-java-microservices-resilient-with-istio/
Github: https://github.com/IBM/resilient-java-microservices-with-istio
IBM Code
Leverage Istio to create resilient and fault tolerant Microservices

34. MS-A
Istio Ingress
Envoy
User Input
EnvoyIstio Pilot
Circuit Breaker (
X Max Conn,
Y Max Pending)
Administrator
Set Destination Policy
N requests N requests
Reached maximum connec2ons –
put the incoming requests in pending state
Reached maximum pending
requests - eject all the incoming
requests.
MS-B

35. MS-AEnvoy
MS-B Pod 2
(Broken)
Istio Pilot
Circuit BreakerAdministrator
Set Destination Policy 503
Load Balancing Pool for MS-B
MS-B Pod 1
(Working)
Eject X minutes
1 2 3
MS-BEnvoy
Istio Ingress
User Input
N requests N requests
Envoy MS-B

36. MS-B Pod
Istio Pilot
Timeout
X seconds delay
Fault Injection
504 error
Administrator
Set Route Rule
MS-AEnvoy
Envoy MS-B
Istio Ingress N requests
User Input
N requests

37. What is a ‘Service Mesh’ ?
A network for services, not bytes
● Resiliency & Efficiency
● Traffic Control
● Visibility
● Security
● Policy Enforcement

38. • // A simple traffic splitting rule
• destination:
serviceB.example.cluster.local
• match:
source:
serviceA.example.cluster.local
route:
- tags:
version: v1.5
• env: us-prod
• weight: 99
• - tags:
version: v2.0-alpha
• env: us-staging
• weight: 1
svcA
Envoy
Pod
Service
A
svcB
Envoy
ServiceB
http://serviceB.example
Pod Labels:
version: v1.5
env: us-prod
svcB
Envoy
Pod Labels:
version: v2.0-
alpha, env:us-
staging
serviceB.example.cluster.loc
Traffic routing
rules
99%
1%
Rules API
Istio-Manager
Traffic Splitting

39. © IBM Corporation / ConfidentialIBM Cloud I Internal Usage Only
svcA
Service A
svcB
Service B
version: v1
Pod 1
Pod 2
Pod 3
Content-based traffic steering
svcA
Service A
svcB
Service B
version: v1
Pod 1
Pod 2
Pod 3
svcB’
version: canary
Pod 4
• // Content-based traffic steering rule
• destination: serviceB.example.cluster.local
match:
httpHeaders:
user-agent:
regex: ^(.*?;)?(iPhone)(;.*)?$
precedence: 2
route:
- tags:
version: canary
Traffic Steering

40. • Monitoring & tracing should not be an
afterthought in the infrastructure
• Goals
• Metrics without instrumenting apps
• Consistent metrics across fleet
• Trace flow of requests across services
• Portable across metric backend
providers
Istio Zipkin tracing dashboard
Istio - Grafana dashboard w/ Prometheus backend
Visibility

41. • Mixer collects metrics emitted by Envoys
• Adapters in the Mixer normalize and
forward to monitoring backends
• Metrics backend can be swapped at
runtime
Pod
Service
A
svcB
Envoy
Service
B
API: /svcB
Latency: 10ms
Status Code: 503
Src: 10.0.0.1
Dst: 10.0.0.2
…...
Prometheu
s
InfluxDB
Prometheus
Adapter
InfluxDB
Adapter
Custom
Adapter
Mixer
Prometheu
ss
InfluxDB
InfluxDB Custom
backend
Metric Flow
Envoy
svcA

42. • Application do not have to deal
with generating spans or
correlating causality
• Envoys generate spans
• Applications need to *forward*
context headers on outbound
calls
• Envoys send traces to Mixer
• Adapters at Mixer send traces to
respective backends svcA
Envoy
Pod
Service
A
svcB
Envoy
Service
B
Trace Headers
X-B3-TraceId
X-B3-SpanId
X-B3-ParentSpanId
X-B3-Sampled
X-B3-Flags
svcC
Envoy
Service
C
Span
s
Span
s
Prometheu
s
InfluxDB
Zipkin
Adapter
Stackdriver
Adapter
Custom
Adapter
Mixer
Prometheu
sZipkin
InfluxDB
Stackdriver Custom
backend
Visibility : Tracing

43. Slide Title Goes Here
Microservices and containers have changed application design and deployment patterns. They have also introduced new challenges, such as
service discovery, routing, failure handling, and visibility to microservices. Kubernetes can handle multiple container-based workloads, including
microservices, but when it comes to more sophisticated features like traffic management, failure handling, and resiliency, a microservices mesh
like Istio is required.
Developer Works Code: https://developer.ibm.com/code/patterns/manage-microservices-traffic-using-istio/
Github: https://github.com/IBM/microservices-traffic-management-using-istio
IBM Code:
Manage micro services traffic using Istio on Kubernetes

45. Slide Title Goes Here
Microservices and containers have changed application design and deployment patterns. They have also introduced new challenges, such as
service discovery, routing, failure handling, and visibility to microservices. Kubernetes can handle multiple container-based workloads, including
microservices, but when it comes to more sophisticated features like traffic management, failure handling, and resiliency, a microservices mesh
like Istio is required.
Developer Works Code: https://developer.ibm.com/code/journey/manage-microservices-traffic-using-istio/
Github: https://github.com/IBM/microservices-traffic-management-using-istio
IBM Code
Manage micro services traffic using Istio on Kubernetes

46. Summary
• Micro-service – Evolution and need
• Framework supporting micro-service deployment
• Scaling
• Service Mesh
• Frameworks for Service Mesh

47. Thank you!