Jitsi Meet: our tale of blood, sweat, tears and love

Jitsi Meet
Our tale of blood, sweat, tears and love.
Open Tech Will Save Us 2020

"Calm waters" by Abevil is licensed under CC BY-NC-ND 2.0
It’s calm before the storm

📍
📍
📍
📍
📍
📍
📍

Getting Ready
"hope" by siette is licensed under CC BY-NC-ND 2.0

The Italian Flood
"Mexico City - Tourist Bus Ride" by CarlosVanVegas is licensed under CC BY 2.0

🔥
🔥
🔥
🔥
🔥
🔥
🔥
🔥

🔥
🔥
🔥
🔥
🔥
🔥
🔥
🔥
🔥

What went wrong?
"Collapse" by Jon Scally is licensed under CC BY-NC-ND 2.0

Panic Mode ON
😱😱😱🙀🙀🙀

Tens of millions of 
dropped packets

Users getting 504s 
left and right

Tons of ⭐ reviews on 
mobile stores

Hunders of support emails 
and forum posts

ulimit -n
Too small! We used prlimit to raise it in production.

Ephemeral port exhaustion
Add more IP addresses to our load balancers and HAProxys.

Inter-server traﬃc
Having so many servers was killing us, 
we made each JVB connect only to its shard.

This lasted for days
Standup turned into a Situation Room.

Reacting to Feedback
"File:Feedback Terminal With A Touchscreen.jpg" by Littokki is licensed under CC BY-SA 4.0

We’ll always have Paris
"File:Eiffel Tower - Tour Eiffel (34377053374).jpg" by Nan Palmero from San Antonio, TX, USA is licensed under CC BY 2.0

📍
📍
📍
📍
📍
📍
📍
📍

AWS Global Accelerator
Instrumental to balance the load across regions.

The numbers (so far)
"XYZ" by Marina Lewandowska is licensed under CC BY-NC-ND 4.0

1000s of servers
Instead of 10s

Optimize our use of resources
Optimize for transit cost by moving JVBs out of AWS

Move away from 
HAProxy stick tables
Split-brains are unavoidable with the current scheme.

Address more feedback
Keep it coming!

Jitsi Meet: our tale of blood, sweat, tears and love

Recomendados