How we built an open video conferencing service to help people stay connected during corona
You can watch the Youtube Recording here (german):
https://t.co/cg7bGKDOjB?amp=1
2. awlnx
● Senior Network Engineer @AS51324
● Twitter @awlnx
Krombel
● IT Consultant
● Twitter @kr0mbel
Who are we?
3. Jitsi to close the social gap during corona
• An upcoming Freifunk Meeting was about to get canceled because of
Corona
• In the past we used NextCloud talk but we had issues (max 6. users per
conference)
■ We needed something different for at least 20 - 30 people
■ jitsi seemed easy and straightforward to install
● “apt install jitsi-meet” => Done ¯_(ツ)_/¯
4. Debian package wins over Docker
• We had some issues with docker-proxy in the past
• docker image didn’t look up to date
• With scaling in mind
■ Planned using dedicated hardware for Jitsi anyway
■ Debian packages are available
5. First test - looks promising
• testing with some people of Freifunk Munich
• Worked like a charm, with a handful of people
• The idea was born:
■ Maybe other people have the same problem?
■ Why not open it for the public?
■ Maybe teachers, healthcare workers, etc need that, too?
6. We need insights! Let’s monitor!
• How many users are on the platform?
• What’s the impact on the Hardware/VMs?
• Do we have bottlenecks? If so: Where?
7. We have to scale!
• One server cannot take the load => Bottleneck CPU on Videobridge
• Scale horizontally
→ Videobridges need dedicated hardware and we may need many!
→ Looking for sponsors
9. The project gains attention
• Interviews
• BR, DigitalCourage, Focus Online, Stimme, SZ, …
• diverse Listen
10. Problems start - Prosody
• Change network_backend to “epoll” - No more 1024 connections as a limit
11. Don’t tune only one part of the stack - nginx
• 502 all over the place
• Raise the number of nginx workers
• Raise max open files limit
12. More problems - JVB Version 1
• CPU Usage …
• Upgrade to JVB Version 2
→ Bottlenecks moved to context switches and PPS
13. Firefox breaks user experience
• Users experience problems especially in large conferences (25-40 people)
• Add warning for Firefox users … it’s mostly ignored … so problems still exists
• Deactivate Firefox support and release FreifunkMeet app
→https://github.com/freifunkMUC/jitsi-meet-electron/releases
14. Upgrades sometimes break stuff
• apt update && apt upgrade
• Configs get wiped … good to have a testing environment
• Once again this time … ar x … scp and git diff ...
15. Lessons learned
• Dedicated Hardware… non-shared resources (NOT the 2€/month VM )
• Know your audience and react to behaviour
• Know your technical limitations
• Check, test, fix, tune and repeat
• Monitoring, Monitoring, Monitoring
16. Thank you very much!
A special thanks to all who helped
writing/translating the FAQs.
Also thank you to all the users and of
course our supporters:
