O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.
Corpus collapsum 
Partition tolerance of Galera in a noisy high load 
environment 
Highload++ 2014 
Raghavendra Prabhu 
 ...
The Title?
Our Cluster
Split brain
Seed quotes.. 
“ ’Network is reliable’ - a fallacy of the distributed system. ” - Peter 
Deutsch 
“ A distributed system i...
Seed quotes.. 
“ ’Network is reliable’ - a fallacy of the distributed system. ” - Peter 
Deutsch 
“ A distributed system i...
Seed quotes.. 
“ ’Network is reliable’ - a fallacy of the distributed system. ” - Peter 
Deutsch 
“ A distributed system i...
Seed quotes.. 
“ ’Network is reliable’ - a fallacy of the distributed system. ” - Peter 
Deutsch 
“ A distributed system i...
The fallacies 
▶ The network is reliable 
▶ Latency is zero 
▶ Bandwidth is infinite 
▶ The network is secure 
9 / 92
The fallacies 
▶ Topology doesn’t change 
▶ There is one administrator 
▶ Transport cost is zero 
▶ The network is homogen...
20000 feet view
Actors 
▶ Database - WSREP/PXC 
▶ Plugin - Galera 
▶ Traffic control 
♦ Traffic Control - tc 
♦ NetEm 
12 / 92
Actors 
▶ Database - WSREP/PXC 
▶ Plugin - Galera 
▶ Traffic control 
♦ Traffic Control - tc 
♦ NetEm 
13 / 92
Actors 
▶ Database - WSREP/PXC 
▶ Plugin - Galera 
▶ Traffic control 
♦ Traffic Control - tc 
♦ NetEm 
14 / 92
Actors 
▶ Containers - Docker 
▶ Load 
♦ Generators - Sysbench, RQG 
▶ Network 
♦ Dnsmasq 
♦ nsenter 
15 / 92
Actors 
▶ Containers - Docker 
▶ Load 
♦ Generators - Sysbench, RQG 
▶ Network 
♦ Dnsmasq 
♦ nsenter 
16 / 92
Actors 
▶ Jenkins 
♦ Build flow and CI 
▶ Storage 
♦ Why 
17 / 92
But why 
▶ The ‘P’ in CAP 
▶ WAN scalability 
▶ Real Reason - fun! 
▶ Tolerance to latency variance 
18 / 92
But why 
▶ The ‘P’ in CAP 
▶ WAN scalability 
▶ Real Reason - fun! 
▶ Tolerance to latency variance 
19 / 92
But why 
▶ The ‘P’ in CAP 
▶ WAN scalability 
▶ Real Reason - fun! 
▶ Tolerance to latency variance 
20 / 92
But why 
▶ The ‘P’ in CAP 
▶ WAN scalability 
▶ Real Reason - fun! 
▶ Tolerance to latency variance 
21 / 92
But why 
▶ Failures in warehouses. 
▶ Not quorum, but consensus. 
▶ Real world networks and synchronous replication 
- Del...
Galera
Galera 
▶ Data-centric approach 
▶ EVS 
▶ Causality and Synchronous 
▶ Latency 
24 / 92
Where did it start
Where did it start 
▶ Bug! https://bugs.launchpad.net/galera/+bug/1274192 
▶ Loss of PC 
▶ Crash 
▶ HA goal 
29 / 92
One can bring the whole down
The Flow
Basic Flow 
Jenkins Build image(s?) Start Dnsmasq Bootstrap 
nsenter/netem Pre-sanity SST/Others Load/Sysbench 
32 / 92
Basic Flow 
Jenkins Build image(s?) Start Dnsmasq Bootstrap 
nsenter/netem Pre-sanity SST/Others Load/Sysbench 
33 / 92
Basic Flow 
Jenkins Build image(s?) Start Dnsmasq Bootstrap 
nsenter/netem Pre-sanity SST/Others Load/Sysbench 
34 / 92
Basic Flow 
Jenkins Build image(s?) Start Dnsmasq Bootstrap 
nsenter/netem Pre-sanity SST/Others Load/Sysbench 
35 / 92
Basic Flow 
Jenkins Build image(s?) Start Dnsmasq Bootstrap 
nsenter/netem Pre-sanity SST/Others Load/Sysbench 
36 / 92
Basic Flow 
Jenkins Build image(s?) Start Dnsmasq Bootstrap 
nsenter/netem Pre-sanity SST/Others Load/Sysbench 
37 / 92
Basic Flow 
Jenkins Build image(s?) Start Dnsmasq Bootstrap 
nsenter/netem Pre-sanity SST/Others Load/Sysbench 
38 / 92
Basic Flow 
Jenkins Build image(s?) Start Dnsmasq Bootstrap 
nsenter/netem Pre-sanity SST/Others Load/Sysbench 
39 / 92
Basic Flow 
RR sysbench 
Detach/Keep 
Post sanity Core trace 
Sanity check Reconciliation 
Examine! 
Cleanup Collect logs ...
Basic Flow 
RR sysbench 
Detach/Keep 
Post sanity Core trace 
Sanity check Reconciliation 
Examine! 
Cleanup Collect logs ...
Basic Flow 
RR sysbench 
Detach/Keep 
Post sanity Core trace 
Sanity check Reconciliation 
Examine! 
Cleanup Collect logs ...
Basic Flow 
RR sysbench 
Detach/Keep 
Post sanity Core trace 
Sanity check Reconciliation 
Examine! 
Cleanup Collect logs ...
Basic Flow 
RR sysbench 
Detach/Keep 
Post sanity Core trace 
Sanity check Reconciliation 
Examine! 
Cleanup Collect logs ...
Basic Flow 
RR sysbench 
Detach/Keep 
Post sanity Core trace 
Sanity check Reconciliation 
Examine! 
Cleanup Collect logs ...
Basic Flow 
RR sysbench 
Detach/Keep 
Post sanity Core trace 
Sanity check Reconciliation 
Examine! 
Cleanup Collect logs ...
Basic Flow 
RR sysbench 
Detach/Keep 
Post sanity Core trace 
Sanity check Reconciliation 
Examine! 
Cleanup Collect logs ...
Cluster Resilience 
48 / 92
Parameters 
▶ Sysbench 
▶ Segments 
▶ Reconciliation period 
▶ Number of nodes with loss 
49 / 92
Parameters 
▶ Sysbench 
▶ Segments 
▶ Reconciliation period 
▶ Number of nodes with loss 
50 / 92
Parameters 
▶ Sysbench 
▶ Segments 
▶ Reconciliation period 
▶ Number of nodes with loss 
51 / 92
Parameters 
▶ Sysbench 
▶ Segments 
▶ Reconciliation period 
▶ Number of nodes with loss 
52 / 92
Parameters 
▶ NetEm 
▶ Detach qdisc or keep it? 
▶ Fsync 
▶ Shutdown 
53 / 92
Parameters 
▶ NetEm 
▶ Detach qdisc or keep it? 
▶ Fsync 
▶ Shutdown 
54 / 92
Parameters 
▶ NetEm 
▶ Detach qdisc or keep it? 
▶ Fsync 
▶ Shutdown 
55 / 92
Parameters 
▶ NetEm 
▶ Detach qdisc or keep it? 
▶ Fsync 
▶ Shutdown 
56 / 92
Parameters
Containers!
Docker 
▶ Why not virtualize 
♦ Occam 
♦ Namespaces 
▶ Simplicity 
♦ Network 
♦ One application per node 
59 / 92
Docker 
▶ Portability 
- See same qualitative behavior that I do. 
▶ Reproducibility 
- Makes it determinstic 
▶ Configura...
Docker 
▶ QEMU and Docker 
▶ Scalability 
♦ Performance 
♦ Feature 
▶ Abstraction of channels 
61 / 92
Container Networking 
▶ Linking didn’t help 
▶ Dnsmasq to rescue! 
♦ Hosts file and volumes 
♦ SIGHUP and refresh 
▶ More ...
Noise 
▶ Initial setup 
- Bridge 
- Egress only 
- IFB 
▶ Present state 
▶ NetEm 
- tc qdisc buckets 
- packet loss, delay...
Testing methods
Method I 
▶ Qdisc is detached after load 
▶ Objective 
- Time to recover of full cluster 
▶ Done with a larger subset 
65 ...
Method II 
▶ Qdisc is kept till the end 
▶ Objective 
- Formation of primary component 
▶ Comparatively smaller subset 
66...
Observations 
▶ Post sanity types 
- Why 
- Whom 
▶ Which method is more pertinent 
▶ State transfer issues 
- Beginning 
...
Observations 
▶ Direct load to affected nodes 
▶ Logs 
- journalctl 
- Streaming? 
68 / 92
Other noises 
▶ Aim 
▶ Fsync 
- libeatmydata 
- Variance 
▶ Correlation with network 
▶ How with Docker 
- LD_PRELOAD 
69 ...
System Load
Load generation 
▶ Sysbench 
- Generation 
- Reconnect on partition 
▶ Sockets chosen 
- Load on affected nodes 
▶ Distrib...
Load generation 
▶ Nature of data/load 
- DDL 
▶ RQG in future 
- Fuzz testing 
72 / 92
The Fix
Strike Out!
Eviction 
▶ STONITH 
▶ Permanent eviction 
▶ ’N’ strikes & out! 
- Timers - evs parameters 
- wsrep_evs_delayed and wsrep_...
Eviction 
▶ Aim 
▶ Quorum required 
- Keep majority of the group live or to avoid eviction. 
- Why? - Not shoot each other...
Eviction 
▶ Aim 
▶ Quorum required 
- Keep majority of the group live or to avoid eviction. 
- Why? - Not shoot each other...
Eviction 
▶ EVS version and upgrade 
▶ TODO! 
- Ingress only 
- Follow here. 
▶ Credits to Teemu Ollakka, Yan Zhang and Al...
Coredumps with Docker 
▶ Breakdown of abstraction 
▶ Lack of isolation 
▶ What was done 
- Volumes 
- core_pattern & sysct...
WAN Segments 
▶ How they work 
▶ Random allocation 
▶ Joiner starvation 
▶ Simulates data center 
▶ Donor selection 
80 / ...
The code 
▶ Github: https://github.com/percona/pxc-docker 
▶ Jenkins: http://jenkins.percona.com/job/PXC-5.6-netem/ 
▶ Con...
Code: todo 
▶ Docker automated builds 
▶ Orchestration 
- Kubernetes, Mesos et.al. 
▶ Docker 
♦ Injection 
♦ Signal proxyi...
Code: todo 
▶ Poll and exit during reconciliation 
- Instrument v/s number of nodes 
▶ Use actual channels 
▶ Run it bare ...
Future work
Future work 
▶ Fault injection 
♦ Memory 
- Poisoned memory/madvise 
♦ Disk 
- libeatmydata 
- Opposite 
- ENOSPC 
85 / 92
Fault injection 
▶ CPU 
- NUMA? 
- Hotplug 
▶ More network 
- corruption, duplication, reordering, rate-limit 
- Better di...
More Chaos
Future work 
▶ Disturb cluster more! 
- Membership changes 
* Manual eviction 
* Pull the cord! 
- Corrupt nodes 
▶ Consis...
Further Reading 
▶ Jepsen 
▶ Byzantine fault tolerance 
- Reaching agreement in presence of faults 
▶ The Network is Relia...
About 
▶ /me: Raghavendra Prabhu, Product Lead, Percona XtraDB Cluster, Percona. 
▶ Slides will be at slideshare. 
▶ About...
Image Credits 
▶ http://galeracluster.com/documentation-webpages/ 
▶ http://www.thelastdragontribute.com/40th-anniversary-...
Spaseeba & Da sveedaneeya!
Corpus collapsum: Partition tolerance of Galera in a noisy high load environment
Corpus collapsum: Partition tolerance of Galera in a noisy high load environment
Corpus collapsum: Partition tolerance of Galera in a noisy high load environment
Próximos SlideShares
Carregando em…5
×

Corpus collapsum: Partition tolerance of Galera in a noisy high load environment

536 visualizações

Publicada em

This is the talk given at Highload++ 2014 in Moscow, Russia. The topic was partition tolerance testing of Galera in a noisy high load environment with NetEm and Docker.

Publicada em: Engenharia
  • Seja o primeiro a comentar

Corpus collapsum: Partition tolerance of Galera in a noisy high load environment

  1. 1. Corpus collapsum Partition tolerance of Galera in a noisy high load environment Highload++ 2014 Raghavendra Prabhu  raghavendra.d.prabhu@gmail.com Percona  raghavendra.prabhu@percona.com  randomsurfer  wnohang.net  rdprabhu  ronin13
  2. 2. The Title?
  3. 3. Our Cluster
  4. 4. Split brain
  5. 5. Seed quotes.. “ ’Network is reliable’ - a fallacy of the distributed system. ” - Peter Deutsch “ A distributed system is one in which the failure of a computer you didn’t even know existed can render your own computer unusable. ” - Leslie Lamport “ Never attribute to malice that which is adequately explained by stupidity. ” - Hanlon’s Razor “ Never attribute to Byzantine failure which can be explained by an ill node(s) ” - Me 5 / 92
  6. 6. Seed quotes.. “ ’Network is reliable’ - a fallacy of the distributed system. ” - Peter Deutsch “ A distributed system is one in which the failure of a computer you didn’t even know existed can render your own computer unusable. ” - Leslie Lamport “ Never attribute to malice that which is adequately explained by stupidity. ” - Hanlon’s Razor “ Never attribute to Byzantine failure which can be explained by an ill node(s) ” - Me 6 / 92
  7. 7. Seed quotes.. “ ’Network is reliable’ - a fallacy of the distributed system. ” - Peter Deutsch “ A distributed system is one in which the failure of a computer you didn’t even know existed can render your own computer unusable. ” - Leslie Lamport “ Never attribute to malice that which is adequately explained by stupidity. ” - Hanlon’s Razor “ Never attribute to Byzantine failure which can be explained by an ill node(s) ” - Me 7 / 92
  8. 8. Seed quotes.. “ ’Network is reliable’ - a fallacy of the distributed system. ” - Peter Deutsch “ A distributed system is one in which the failure of a computer you didn’t even know existed can render your own computer unusable. ” - Leslie Lamport “ Never attribute to malice that which is adequately explained by stupidity. ” - Hanlon’s Razor “ Never attribute to Byzantine failure which can be explained by an ill node(s) ” - Me 8 / 92
  9. 9. The fallacies ▶ The network is reliable ▶ Latency is zero ▶ Bandwidth is infinite ▶ The network is secure 9 / 92
  10. 10. The fallacies ▶ Topology doesn’t change ▶ There is one administrator ▶ Transport cost is zero ▶ The network is homogeneous 10 / 92
  11. 11. 20000 feet view
  12. 12. Actors ▶ Database - WSREP/PXC ▶ Plugin - Galera ▶ Traffic control ♦ Traffic Control - tc ♦ NetEm 12 / 92
  13. 13. Actors ▶ Database - WSREP/PXC ▶ Plugin - Galera ▶ Traffic control ♦ Traffic Control - tc ♦ NetEm 13 / 92
  14. 14. Actors ▶ Database - WSREP/PXC ▶ Plugin - Galera ▶ Traffic control ♦ Traffic Control - tc ♦ NetEm 14 / 92
  15. 15. Actors ▶ Containers - Docker ▶ Load ♦ Generators - Sysbench, RQG ▶ Network ♦ Dnsmasq ♦ nsenter 15 / 92
  16. 16. Actors ▶ Containers - Docker ▶ Load ♦ Generators - Sysbench, RQG ▶ Network ♦ Dnsmasq ♦ nsenter 16 / 92
  17. 17. Actors ▶ Jenkins ♦ Build flow and CI ▶ Storage ♦ Why 17 / 92
  18. 18. But why ▶ The ‘P’ in CAP ▶ WAN scalability ▶ Real Reason - fun! ▶ Tolerance to latency variance 18 / 92
  19. 19. But why ▶ The ‘P’ in CAP ▶ WAN scalability ▶ Real Reason - fun! ▶ Tolerance to latency variance 19 / 92
  20. 20. But why ▶ The ‘P’ in CAP ▶ WAN scalability ▶ Real Reason - fun! ▶ Tolerance to latency variance 20 / 92
  21. 21. But why ▶ The ‘P’ in CAP ▶ WAN scalability ▶ Real Reason - fun! ▶ Tolerance to latency variance 21 / 92
  22. 22. But why ▶ Failures in warehouses. ▶ Not quorum, but consensus. ▶ Real world networks and synchronous replication - Delay - Partition 22 / 92
  23. 23. Galera
  24. 24. Galera ▶ Data-centric approach ▶ EVS ▶ Causality and Synchronous ▶ Latency 24 / 92
  25. 25. Where did it start
  26. 26. Where did it start ▶ Bug! https://bugs.launchpad.net/galera/+bug/1274192 ▶ Loss of PC ▶ Crash ▶ HA goal 29 / 92
  27. 27. One can bring the whole down
  28. 28. The Flow
  29. 29. Basic Flow Jenkins Build image(s?) Start Dnsmasq Bootstrap nsenter/netem Pre-sanity SST/Others Load/Sysbench 32 / 92
  30. 30. Basic Flow Jenkins Build image(s?) Start Dnsmasq Bootstrap nsenter/netem Pre-sanity SST/Others Load/Sysbench 33 / 92
  31. 31. Basic Flow Jenkins Build image(s?) Start Dnsmasq Bootstrap nsenter/netem Pre-sanity SST/Others Load/Sysbench 34 / 92
  32. 32. Basic Flow Jenkins Build image(s?) Start Dnsmasq Bootstrap nsenter/netem Pre-sanity SST/Others Load/Sysbench 35 / 92
  33. 33. Basic Flow Jenkins Build image(s?) Start Dnsmasq Bootstrap nsenter/netem Pre-sanity SST/Others Load/Sysbench 36 / 92
  34. 34. Basic Flow Jenkins Build image(s?) Start Dnsmasq Bootstrap nsenter/netem Pre-sanity SST/Others Load/Sysbench 37 / 92
  35. 35. Basic Flow Jenkins Build image(s?) Start Dnsmasq Bootstrap nsenter/netem Pre-sanity SST/Others Load/Sysbench 38 / 92
  36. 36. Basic Flow Jenkins Build image(s?) Start Dnsmasq Bootstrap nsenter/netem Pre-sanity SST/Others Load/Sysbench 39 / 92
  37. 37. Basic Flow RR sysbench Detach/Keep Post sanity Core trace Sanity check Reconciliation Examine! Cleanup Collect logs 40 / 92
  38. 38. Basic Flow RR sysbench Detach/Keep Post sanity Core trace Sanity check Reconciliation Examine! Cleanup Collect logs 41 / 92
  39. 39. Basic Flow RR sysbench Detach/Keep Post sanity Core trace Sanity check Reconciliation Examine! Cleanup Collect logs 42 / 92
  40. 40. Basic Flow RR sysbench Detach/Keep Post sanity Core trace Sanity check Reconciliation Examine! Cleanup Collect logs 43 / 92
  41. 41. Basic Flow RR sysbench Detach/Keep Post sanity Core trace Sanity check Reconciliation Examine! Cleanup Collect logs 44 / 92
  42. 42. Basic Flow RR sysbench Detach/Keep Post sanity Core trace Sanity check Reconciliation Examine! Cleanup Collect logs 45 / 92
  43. 43. Basic Flow RR sysbench Detach/Keep Post sanity Core trace Sanity check Reconciliation Examine! Cleanup Collect logs 46 / 92
  44. 44. Basic Flow RR sysbench Detach/Keep Post sanity Core trace Sanity check Reconciliation Examine! Cleanup Collect logs 47 / 92
  45. 45. Cluster Resilience 48 / 92
  46. 46. Parameters ▶ Sysbench ▶ Segments ▶ Reconciliation period ▶ Number of nodes with loss 49 / 92
  47. 47. Parameters ▶ Sysbench ▶ Segments ▶ Reconciliation period ▶ Number of nodes with loss 50 / 92
  48. 48. Parameters ▶ Sysbench ▶ Segments ▶ Reconciliation period ▶ Number of nodes with loss 51 / 92
  49. 49. Parameters ▶ Sysbench ▶ Segments ▶ Reconciliation period ▶ Number of nodes with loss 52 / 92
  50. 50. Parameters ▶ NetEm ▶ Detach qdisc or keep it? ▶ Fsync ▶ Shutdown 53 / 92
  51. 51. Parameters ▶ NetEm ▶ Detach qdisc or keep it? ▶ Fsync ▶ Shutdown 54 / 92
  52. 52. Parameters ▶ NetEm ▶ Detach qdisc or keep it? ▶ Fsync ▶ Shutdown 55 / 92
  53. 53. Parameters ▶ NetEm ▶ Detach qdisc or keep it? ▶ Fsync ▶ Shutdown 56 / 92
  54. 54. Parameters
  55. 55. Containers!
  56. 56. Docker ▶ Why not virtualize ♦ Occam ♦ Namespaces ▶ Simplicity ♦ Network ♦ One application per node 59 / 92
  57. 57. Docker ▶ Portability - See same qualitative behavior that I do. ▶ Reproducibility - Makes it determinstic ▶ Configurable and CI - Byproducts 60 / 92
  58. 58. Docker ▶ QEMU and Docker ▶ Scalability ♦ Performance ♦ Feature ▶ Abstraction of channels 61 / 92
  59. 59. Container Networking ▶ Linking didn’t help ▶ Dnsmasq to rescue! ♦ Hosts file and volumes ♦ SIGHUP and refresh ▶ More elegant methods - Swarm 62 / 92
  60. 60. Noise ▶ Initial setup - Bridge - Egress only - IFB ▶ Present state ▶ NetEm - tc qdisc buckets - packet loss, delay, corruption, duplication, reordering - nsenter ▶ Future - Docker exec 63 / 92
  61. 61. Testing methods
  62. 62. Method I ▶ Qdisc is detached after load ▶ Objective - Time to recover of full cluster ▶ Done with a larger subset 65 / 92
  63. 63. Method II ▶ Qdisc is kept till the end ▶ Objective - Formation of primary component ▶ Comparatively smaller subset 66 / 92
  64. 64. Observations ▶ Post sanity types - Why - Whom ▶ Which method is more pertinent ▶ State transfer issues - Beginning - During re-emergence 67 / 92
  65. 65. Observations ▶ Direct load to affected nodes ▶ Logs - journalctl - Streaming? 68 / 92
  66. 66. Other noises ▶ Aim ▶ Fsync - libeatmydata - Variance ▶ Correlation with network ▶ How with Docker - LD_PRELOAD 69 / 92
  67. 67. System Load
  68. 68. Load generation ▶ Sysbench - Generation - Reconnect on partition ▶ Sockets chosen - Load on affected nodes ▶ Distribution of Load - RR with socat - Native sysbench support - HAProxy? 71 / 92
  69. 69. Load generation ▶ Nature of data/load - DDL ▶ RQG in future - Fuzz testing 72 / 92
  70. 70. The Fix
  71. 71. Strike Out!
  72. 72. Eviction ▶ STONITH ▶ Permanent eviction ▶ ’N’ strikes & out! - Timers - evs parameters - wsrep_evs_delayed and wsrep_evs_evict_list 75 / 92
  73. 73. Eviction ▶ Aim ▶ Quorum required - Keep majority of the group live or to avoid eviction. - Why? - Not shoot each other - Non-PC nodes also. 76 / 92
  74. 74. Eviction ▶ Aim ▶ Quorum required - Keep majority of the group live or to avoid eviction. - Why? - Not shoot each other - Non-PC nodes also. 77 / 92
  75. 75. Eviction ▶ EVS version and upgrade ▶ TODO! - Ingress only - Follow here. ▶ Credits to Teemu Ollakka, Yan Zhang and Alex Yurchenko from codership. 78 / 92
  76. 76. Coredumps with Docker ▶ Breakdown of abstraction ▶ Lack of isolation ▶ What was done - Volumes - core_pattern & sysctl - suid and ulimit 79 / 92
  77. 77. WAN Segments ▶ How they work ▶ Random allocation ▶ Joiner starvation ▶ Simulates data center ▶ Donor selection 80 / 92
  78. 78. The code ▶ Github: https://github.com/percona/pxc-docker ▶ Jenkins: http://jenkins.percona.com/job/PXC-5.6-netem/ ▶ Contributions/testing welcome! ▶ Dependencies - Sysbench 81 / 92
  79. 79. Code: todo ▶ Docker automated builds ▶ Orchestration - Kubernetes, Mesos et.al. ▶ Docker ♦ Injection ♦ Signal proxying 82 / 92
  80. 80. Code: todo ▶ Poll and exit during reconciliation - Instrument v/s number of nodes ▶ Use actual channels ▶ Run it bare - CoreOS etc. ▶ Overlay with etcd/fleet/libswarm 83 / 92
  81. 81. Future work
  82. 82. Future work ▶ Fault injection ♦ Memory - Poisoned memory/madvise ♦ Disk - libeatmydata - Opposite - ENOSPC 85 / 92
  83. 83. Fault injection ▶ CPU - NUMA? - Hotplug ▶ More network - corruption, duplication, reordering, rate-limit - Better distribution - Other shaping 86 / 92
  84. 84. More Chaos
  85. 85. Future work ▶ Disturb cluster more! - Membership changes * Manual eviction * Pull the cord! - Corrupt nodes ▶ Consistency voting 88 / 92
  86. 86. Further Reading ▶ Jepsen ▶ Byzantine fault tolerance - Reaching agreement in presence of faults ▶ The Network is Reliable ▶ NetEm ▶ Latency: The New Web Performance Bottleneck ▶ Galera Cluster Documentation ▶ Auto eviction code ▶ Don’t Settle for Eventual Consistency ▶ Extended Virtual Synchrony 89 / 92
  87. 87. About ▶ /me: Raghavendra Prabhu, Product Lead, Percona XtraDB Cluster, Percona. ▶ Slides will be at slideshare. ▶ About.me: raghavendra.prabhu ▶ Keybase.io: rdprabhu ▶ Presentation under CC BY-SA 4.0 90 / 92
  88. 88. Image Credits ▶ http://galeracluster.com/documentation-webpages/ ▶ http://www.thelastdragontribute.com/40th-anniversary-death-of-bruce-lee/ ▶ https://upload.wikimedia.org/wikipedia/commons/6/60/Corpus_callosum.png ▶ http://www.thebarrow.org/Neurological_Services/Epilepsy/204354 ▶ https://flic.kr/p/9J6GNu ▶ https://secure.flickr.com/photos/brewbooks/7780990192 ▶ https://www.flickr.com/photos/kwerfeldein/2649294869 ▶ https://secure.flickr.com/photos/mindmob/51951632 ▶ https://secure.flickr.com/photos/arenamontanus/2227769907 ▶ https://www.flickr.com/photos/markop/477199204 ▶ http://galeracluster.com/wp-content/uploads/2013/10/galera_replication1.png ▶ https://www.flickr.com/photos/gcwest/281385801 ▶ https://www.flickr.com/photos/opethdamna/360934079 ▶ http://digital-amphetamine.deviantart.com/art/Sky-82555664 ▶ http://highload.co/i/logo.png ▶ https://flic.kr/p/xTT8n ▶ https://www.flickr.com/photos/29233640@N07/13466208953 ▶ https://www.flickr.com/photos/bob_in_thailand/9782777742/ 91 / 92
  89. 89. Spaseeba & Da sveedaneeya!

×