The document discusses ensuring software stability in production environments. It emphasizes the importance of non-functional requirements like logging, administrative tools, and monitoring. Tests that simulate real-world failures are also recommended, as well as following architectural best practices for stability. Features should be introduced gradually with on/off switches and thorough testing of upgrades and rollbacks. Overall the document stresses prioritizing stability over new features through practices like timeouts, circuit breakers, and decoupling layers.

1. Software Instability
Renato Lucindo
@rlucindo -- lucindo.github.com
FFFFUUUU Conf 2010 -- ffffuuuu.me

2. What kind of software?
Web Applications
Kinda Big
Distributed Systems

3. Production-ready Software
works
stable
maintainable
documented
scalable

4. Real life Production Software
it runs

5. Anatomy of a Real life Software
focus on features
poor logs
no administrative tools
no information
poorly tested

6. Why?
Feature centric
"What're the odds of that happening?"
Deadlines
ReReReReReReprioritization
Over confidence
Inexperience
Naiveness
Over excitement

7. The Clients...
Product Owner?
Manager?
Stakeholder?
Who will deal with your mess?
OPERATIONS

8. Building and Running a System

9. What to do?

10. Non-functional Requirements
decent logs
with context (user, host, operation, ...)
automatic reload (configs, templates, ...)
administrative tools
plenty of it
counters and statistics
charts
operations requirements

11. Tests that matter
Fully automated functional test
No mocks
Log replay
High concurrency test
Bullet-proof tests
The -STOP test
Load test (iozone, gcc test)
Kill test

12. How to add a new feature
On/Off switch
Tests with different versions
Full system functional tests
Upgrade test
Downgrade (rollback) test

13. Architecture 101
Cache
Queue
Partition

14. Stability Patterns
Use Timeouts
Circuit Breaker
Bulkheads
Steady State
Fail Fast
Handshaking
Test Harness
Decoupling Middleware

15. Use your head!

16. What to expect

17. CAP Theorem

18. Failures
Hardware failure
Network failure
Electricity
Natural disasters
Code failure

19. Bugs

20. Human Errors

21. Your cluelessness