A presentation I gave at AOL about Mozilla's crash reporting project Socorro; about how Mozilla works; and about how to become more open. The slides don't have a lot of detail, see the blog post for more: http://www.laurathomson.com/2011/02/being-open/

1. about:mozilla
Laura Thomson
laura@mozilla.com
February 10th, 2011

2. I usually work from the general to the specific, but today’s
going to be a bit different.

3. I usually work from the general to the specific, but today’s
going to be a bit different.
There’s an agenda, but it’s your agenda, so feel free to
interrupt with questions at any time.

4. I usually work from the general to the specific, but today’s
going to be a bit different.
There’s an agenda, but it’s your agenda, so feel free to
interrupt with questions at any time.
I’m going to give you a tour of my work, and of Mozilla, and
what it means to be open by default.

5. Webtools at Mozilla
Webtools is:
Socorro, the crash reporting system
Perf-o-matic, performance graphing
AUS, the automatic update service
Elmo, tools for our localization community
...and whatever else is needed

6. We exist to help ship
the most stable
fastest
safest
most widely available
Firefox.

7. Let’s take a look at Socorro.

8. What is it?

12. “Socorro has a lot of moving parts”
...
“I prefer to think of them as dancing parts”

13. A different type of scaling:
Typical webapp: scale to millions of users without
degradation of response time
Socorro: less than a hundred users, terabytes of data.

14. Basic law of scale still applies:
The bigger you get, the more spectacularly you fail

15. Some numbers
At peak we receive 3000 crashes per minute
3 million per day
Median crash size 100k -> 150k
30TB stored in HBase and growing every day

16. What can we do?
Does betaN have more (null signature) crashes than other
betas?
Analyze differences between Flash versions x and y crashes
Detect duplicate crashes
Detect explosive crashes
Email victims of a malware-related crash

17. As with any large system, regardless of the dimension of
scaling, problem solving involves two things:
system thinking
empirical evidence

18. Stability
Recent challenge, solved by:
basic stability work (research, restarts)
architectural improvements
better build and release processes
better insight into systems
move to a new datacenter

19. Data center move
Worse than moving house:
Four months of planning, ten people, a hundred new
machines, forty smoke test boxes
Audits and system improvements
Opportunity to iron out many problems and weaknesses

20. It’s all open
Get the code:
http://code.google.com/p/socorro
Read/file/fix bugs:
https://bugzilla.mozilla.org/
Call in for the weekly meetings:
https://wiki.mozilla.org/Breakpad/Status_Meetings
Join us in IRC:
irc.mozilla.org #breakpad

21. Open by Default
This is the Mozilla Way:
Open by default
Open in all but a handful of cases
The good, the bad, and the ugly: all of it happens in public
discourse

22. Starting open is easier
Open from the beginning is easier than retroactively open.
Why?
licensing
dependencies
shame
culture

23. Becoming open is possible
How open do we want to be?
What licensing are we going to use?
How will we encourage, accept, and mediate contributions
from non-employees, or non-official project members?
How will projects be governed?

24. Open Works For Mozilla
Open means honest
Open means participation: stuff gets done by volunteers
Open leads to autonomy and trust
Open leads to pushing decision making to the edges
Open leads to a certain degree of anarchy at Mozilla... “we
are fiercely unconventional.”

25. Firefox answers to no-one but you.

26. Questions?
Ask me (almost) anything
Ask me in email: laura@mozilla.com