A look at different ways to achieve multicluster communication and when it makes sense to use a service mesh.

1. Multiclusterk8s: How a
Service MeshCan/Can’t Help
Andrew Jenkins, CTO
@notthatjenkins

2. Aspen Mesh Survey at KubeCon 2019 Europe
Multiple Independent Prod
Dev/Test/Stage
Multiple x-comm Prod
Multiple
(85%)
(10%) Other
(5%) One

3. Service Mesh
Service Mesh Control Plane
App A
Proxy
App B
Proxy
Service A Service B
ObservabilitySecurityTraffic
Management
k8s apiserver
Cluster

4. Service Mesh
Service Mesh Control Plane
App A
Proxy
App B
Proxy
Service A Service B
k8s apiserver
Cluster A
Service Mesh Control Plane
App A
Proxy
App B
Proxy
Service A Service B
k8s apiserver
Cluster B

5. Service Mesh
App A
Proxy
App B
Proxy
Service A Service B
k8s apiserver
Cluster A
App A
Proxy
App B
Proxy
Service A Service B
k8s apiserver
Cluster B
Service Mesh Control Plane

6. Service Mesh
App A
Proxy
App B
Proxy
Service A Service B
k8s apiserver
Cluster A
App A
Proxy
App B
Proxy
Service A Service B
k8s apiserver
Cluster B
Service Mesh Control Plane

7. Service Mesh
Service Mesh Control Plane
App A
Proxy
App B
Proxy
Service A Service B
k8s apiserver
Cluster A
Service Mesh Control Plane
App A
Proxy
App B
Proxy
Service A Service B
k8s apiserver
Cluster B
Higher Level

8. There are many reasons to want to run multiple clusters…
* Blast-radius (a problem in one cluster doesn't kill the whole system)
* Environment isolation (dev, test, prod)
* Reliability (a zone or region outage does not bring down the app)
* Latency (run the app as close to customers as possible)
* Scale (the app is too big to fit in a single cluster)
* Provider diversity (for regulatory, geographic, data gravity, or
other reasons)
* Jurisdiction (keep user data in-country)
* Upgrade scope (upgrade infra for some parts of your app but not all of
it)
* Avoid the need for in-place cluster upgrades
* Performance isolation (teams don't want to feel each other)
* Security isolation (sensitive data or untrusted code)
* Organizational isolation (teams have different management domains)
* Cost isolation (teams get different bills)
Tim Hockin, Re: Proposing Submariner as a sig-multicluster

9. What does the Internet have
to teach us?

10. NarrowPurpose Diversity
…what itruns on
…what runs on it
...how big it is
Any system with an IP addresscan send
packetsto any other system with anIP
address
Internet
IEEE 802.3
IEEE 802.5
IEEE 802.11
RFC1577
RFC2549

11. ~100GB/wk 1983
~100GB/day 1992
~100GB/hour 1997
~100GB/second2002
~100GB/ 50ms 2007
~100GB/ms 2019

12. Scalable Evolutionary

13. Example

14. B
D
C
A

15. B
D
C
A
To Hops Via
AJ 1 -
To Hops Via
Cat 1 -
Routing InformationProtocol(RIP)
Bellman-Ford

16. B
D
C
A
To Hops Via
AJ 1 -
To Hops Via
Cat 1 -
To Hops Via
Cat 2 D

17. B
D
C
A
To Hops Via
AJ 1 -
To Hops Via
Cat 1 -
To Hops Via
Cat 2 D
To Hops Via
Cat 3 C

18. B
D
C
A
To Hops Via
AJ 1 -
To Hops Via
Cat 1 -
To Hops Via
Cat 2 D
To Hops Via
Cat 3 C
AJ 2 A

19. B
D
C
A
To Hops Via
AJ 1 -
To Hops Via
Cat 1 -
To Hops Via
Cat 2 D
To Hops Via
Cat 3 2 C D
AJ 2 A

20. B
D
C
A
To Hops Via
AJ 1 -
Cat 3 B
To Hops Via
Cat 1 -
To Hops Via
Cat 2 D
To Hops Via
Cat 3 2 C D
AJ 2 A

21. Scalable Evolutionary

22. B
D
C
A
E
FG
H
I

23. B
D
C
A
E
FG
H
I
AS4037
AS717
AS2310
Border GatewayProtocol(BGP)

24. B
D
C
A
E
FG
H
I
AS4037
AS717
AS2310
RIP
RIP
RIP OSPF

25. OK, what does this have to
do with k8s?

26. There are many reasons to want to run multiple clusters…
* Blast-radius (a problem in one cluster doesn't kill the whole system)
* Environment isolation (dev, test, prod)
* Reliability (a zone or region outage does not bring down the app)
* Latency (run the app as close to customers as possible)
* Scale (the app is too big to fit in a single cluster)
* Provider diversity (for regulatory, geographic, data gravity, or
other reasons)
* Jurisdiction (keep user data in-country)
* Upgrade scope (upgrade infra for some parts of your app but not all of
it)
* Avoid the need for in-place cluster upgrades
* Performance isolation (teams don't want to feel each other)
* Security isolation (sensitive data or untrusted code)
* Organizational isolation (teams have different management domains)
* Cost isolation (teams get different bills)
Tim Hockin, Re: Proposing Submariner as a sig-multicluster

27. Scalable Evolutionary

28. Unified Management – Configurethem all inoneplace
Unified Trust – Crypto trusttraceable back to onecommonroot
Heterogenous Network – Clusters can have overlappingor non-routableinternal IPs
Independent Fault Domain – If Cluster A blows up,Cluster B is still OK
Inter-Cluster Mesh Traffic –Inter-cluster traffic is still Service Mesh traffic
To Multicluster, or Not to Multicluster: Inter-cluster Communication Using a Service Mesh

29. UnifiedManagement UnifiedTrust Heterogenous Network
Independent Fault
Domain
Inter-clusterMesh
Traffic
Independent ✓ ✓
Common Management ✓ ✓ ✓
Flat Network ✓ ✓ ✓
Split Horizon ✓ ✓ ✓ ✓
Cluster-AwareService
Routing
✓ ✓ ✓ ✓
To Multicluster, or Not to Multicluster: Inter-cluster Communication Using a Service Mesh

30. Istio: Multicluster Deployments
Split Horizon
✓ Unified Mgmt
✓ Unified Trust
✓ Hetero Network
✘Fault Domain
✓ X-Cluster Mesh

31. Istio: Multicluster Deployments
Cluster-Aware Service Routing
✘ Unified Mgmt
✓ Unified Trust
✓ Hetero Network
✓ Fault Domain
✓ X-Cluster Mesh

32. Recap

33. Aspen Mesh Survey at KubeCon 2019 Europe
Multiple Independent Prod
Dev/Test/Stage
Multiple x-comm Prod
Multiple
(85%)
(10%) Other
(5%) One

34. There are many reasons to want to run multiple clusters…
* Blast-radius (a problem in one cluster doesn't kill the whole system)
* Environment isolation (dev, test, prod)
* Reliability (a zone or region outage does not bring down the app)
* Latency (run the app as close to customers as possible)
* Scale (the app is too big to fit in a single cluster)
* Provider diversity (for regulatory, geographic, data gravity, or
other reasons)
* Jurisdiction (keep user data in-country)
* Upgrade scope (upgrade infra for some parts of your app but not all of
it)
* Avoid the need for in-place cluster upgrades
* Performance isolation (teams don't want to feel each other)
* Security isolation (sensitive data or untrusted code)
* Organizational isolation (teams have different management domains)
* Cost isolation (teams get different bills)
Tim Hockin, Re: Proposing Submariner as a sig-multicluster

35. Unified Management – Configurethem all inoneplace
Unified Trust – Crypto trusttraceable back to onecommonroot
Heterogenous Network – Clusters can have overlappingor non-routableinternal IPs
Independent Fault Domain – If Cluster A blows up,Cluster B is still OK
Inter-Cluster Mesh Traffic –Inter-cluster traffic is still Service Mesh traffic
To Multicluster, or Not to Multicluster: Inter-cluster Communication Using a Service Mesh

36. Thank You

37. Speaker Name
Title

39. Section Title
Goes Here

43. Unique, live, never-to-be repeated entertainment &
experiences
created by the magical interactions of many.

44. Slide Title
Goes Here
Observability Security Insights

45. Config data toEnvoys TLS certs toEnvoys Monitors K8s fornew pods toinject Envoys
Mixer Sidecar InjectorPilot
IstioControlPlane
Ingress
Gateway
Egress
GatewayEnvoy
Container
Flask
Python
Container
SERVICEA
SERVICEA
Envoy
Container
Flask
Python
Container
SERVICEA
SERVICEA
Envoy
Container
Flask
Python
Container
SERVICEA
SERVICEA
Policy, quota &telemetry
Citadel
Managing Microservices
with Istio