5. Introduction
Seed quotes..
“ ’Network is reliable’ - a fallacy of the distributed system. ”
“ A distributed system is one in which the failure of a computer you didn’t
even know existed can render your own computer unusable. ” - Leslie Lamport
“ Never attribute to malice that which is adequately explained by stupidity.
” - Hanlon’s Razor
“ Never attribute to Byzantine failure which can be explained by an ill
node(s) ” - Me
Raghavendra Prabhu (Percona) Corpus collapsum 31 October, 2014 5 / 58
6. Introduction
Seed quotes..
“ ’Network is reliable’ - a fallacy of the distributed system. ”
“ A distributed system is one in which the failure of a computer you didn’t
even know existed can render your own computer unusable. ” - Leslie Lamport
“ Never attribute to malice that which is adequately explained by stupidity.
” - Hanlon’s Razor
“ Never attribute to Byzantine failure which can be explained by an ill
node(s) ” - Me
Raghavendra Prabhu (Percona) Corpus collapsum 31 October, 2014 5 / 58
7. Introduction
Seed quotes..
“ ’Network is reliable’ - a fallacy of the distributed system. ”
“ A distributed system is one in which the failure of a computer you didn’t
even know existed can render your own computer unusable. ” - Leslie Lamport
“ Never attribute to malice that which is adequately explained by stupidity.
” - Hanlon’s Razor
“ Never attribute to Byzantine failure which can be explained by an ill
node(s) ” - Me
Raghavendra Prabhu (Percona) Corpus collapsum 31 October, 2014 5 / 58
8. Introduction
Seed quotes..
“ ’Network is reliable’ - a fallacy of the distributed system. ”
“ A distributed system is one in which the failure of a computer you didn’t
even know existed can render your own computer unusable. ” - Leslie Lamport
“ Never attribute to malice that which is adequately explained by stupidity.
” - Hanlon’s Razor
“ Never attribute to Byzantine failure which can be explained by an ill
node(s) ” - Me
Raghavendra Prabhu (Percona) Corpus collapsum 31 October, 2014 5 / 58
15. Introduction
Actors
▶ Jenkins
♦ Build flow and CI
▶ Storage
♦ Why
▶ “Others”
Raghavendra Prabhu (Percona) Corpus collapsum 31 October, 2014 9 / 58
16. Details
But why
▶ The ’P’ in CAP
▶ WAN scalability
▶ Real Reason - fun!
▶ Tolerance to latency variance
Raghavendra Prabhu (Percona) Corpus collapsum 31 October, 2014 10 / 58
17. Details
But why
▶ The ’P’ in CAP
▶ WAN scalability
▶ Real Reason - fun!
▶ Tolerance to latency variance
Raghavendra Prabhu (Percona) Corpus collapsum 31 October, 2014 10 / 58
18. Details
But why
▶ The ’P’ in CAP
▶ WAN scalability
▶ Real Reason - fun!
▶ Tolerance to latency variance
Raghavendra Prabhu (Percona) Corpus collapsum 31 October, 2014 10 / 58
19. Details
But why
▶ The ’P’ in CAP
▶ WAN scalability
▶ Real Reason - fun!
▶ Tolerance to latency variance
Raghavendra Prabhu (Percona) Corpus collapsum 31 October, 2014 10 / 58
20. Details
But why
▶ Failures in warehouses.
▶ Not quorum, but consensus.
▶ Real world networks and synchronous replication
- Delay
- Partition
Raghavendra Prabhu (Percona) Corpus collapsum 31 October, 2014 11 / 58
27. Details
Where did it start
▶ Bug! https://bugs.launchpad.net/galera/+bug/1274192
▶ Loss of PC
▶ Crash
▶ HA goal
Raghavendra Prabhu (Percona) Corpus collapsum 31 October, 2014 18 / 58
56. Details
Docker
▶ Why not virtualize
♦ Occam
♦ Namespaces
▶ Simplicity
♦ Network
♦ One application per node
Raghavendra Prabhu (Percona) Corpus collapsum 31 October, 2014 27 / 58
57. Details
Docker
▶ Portability
- See same qualitative behavior that I do.
▶ Reproducibility
- Makes it determinstic
▶ Configurable and CI
- Byproducts
Raghavendra Prabhu (Percona) Corpus collapsum 31 October, 2014 28 / 58
58. Details
Docker
▶ QEMU and Docker
▶ Scalability
♦ Performance
♦ Feature
▶ Abstraction of channels
Raghavendra Prabhu (Percona) Corpus collapsum 31 October, 2014 29 / 58
59. Details
Container Networking
▶ Linking didn’t help
▶ Dnsmasq to rescue!
♦ Hosts file and volumes
♦ SIGHUP and refresh
▶ More elegant methods
Swarm
Raghavendra Prabhu (Percona) Corpus collapsum 31 October, 2014 30 / 58
62. Details
Method I
▶ Qdisc is detached after load
▶ Objective
- Time to recover of full cluster
▶ Done with a larger subset
Raghavendra Prabhu (Percona) Corpus collapsum 31 October, 2014 33 / 58
63. Details
Method II
▶ Qdisc is kept till the end
▶ Objective
- Formation of primary component
▶ Comparatively smaller set
Raghavendra Prabhu (Percona) Corpus collapsum 31 October, 2014 34 / 58
64. Details
Observations
▶ Post sanity types
- Why
▶ Which method is more pertinent
▶ State transfer issues
- Beginning
- During re-emergence
Raghavendra Prabhu (Percona) Corpus collapsum 31 October, 2014 35 / 58
65. Details
Observations
▶ Direct load to affected nodes
▶ Logs
- journalctl
- Streaming?
Raghavendra Prabhu (Percona) Corpus collapsum 31 October, 2014 36 / 58
66. Details
Other noises
▶ Aim
▶ Fsync
- libeatmydata
- Variance
▶ Correlation with network
▶ How with Docker
Raghavendra Prabhu (Percona) Corpus collapsum 31 October, 2014 37 / 58
73. Details
Eviction
▶ Aim
▶ Quorum required
- Why? - Not shoot each other - Non-PC nodes also.
Raghavendra Prabhu (Percona) Corpus collapsum 31 October, 2014 44 / 58
74. Details
Eviction
▶ Aim
▶ Quorum required
- Why? - Not shoot each other - Non-PC nodes also.
Raghavendra Prabhu (Percona) Corpus collapsum 31 October, 2014 44 / 58
75. Details
Eviction
▶ EVS version and upgrade
▶ TODO!
- Ingress only - Follow here.
▶ Credits to Teemu Ollakka, Yan Zhang and Alex Yurchenko from codership.
Raghavendra Prabhu (Percona) Corpus collapsum 31 October, 2014 45 / 58
76. Details
Coredumps with Docker
▶ Breakdown of abstraction
▶ Lack of isolation
▶ What was done
- Volumes
- core_pattern & sysctl
- suid and ulimit
Raghavendra Prabhu (Percona) Corpus collapsum 31 October, 2014 46 / 58
77. Details
WAN Segments
▶ How they work
▶ Random allocation
▶ Joiner starvation
▶ Simulates data center
▶ Donor selection
Raghavendra Prabhu (Percona) Corpus collapsum 31 October, 2014 47 / 58
79. Epilogue
Code: todo
▶ Docker automated builds
▶ Orchestration
▶ Docker
♦ Injection
♦ Signal proxying
Raghavendra Prabhu (Percona) Corpus collapsum 31 October, 2014 49 / 58
80. Epilogue
Code: todo
▶ Use Hoare’s channels - Go!
▶ Run it bare - CoreOS
▶ Overlay with etcd/fleet/libswarm
Raghavendra Prabhu (Percona) Corpus collapsum 31 October, 2014 50 / 58
86. Epilogue
Further Reading
▶ Byzantine fault tolerance
- Reaching agreement in presence of faults
▶ The Network is Reliable
▶ NetEm
▶ Latency: The New Web Performance Bottleneck
▶ Galera
▶ Auto eviction code
▶ Don’t Settle for Eventual Consistency
Raghavendra Prabhu (Percona) Corpus collapsum 31 October, 2014 56 / 58
87. Epilogue
About
▶ /me: Raghavendra Prabhu, Product Lead, Percona XtraDB Cluster, Percona.
▶ Slides will be at slideshare and owncloud
▶ Keybase.io: rdprabhu
▶ About.me: raghavendra.prabhu
▶ Keybase.io: rdprabhu
▶ Presentation under CC BY-SA 4.0
Raghavendra Prabhu (Percona) Corpus collapsum 31 October, 2014 57 / 58