The slide deck used for performance tuning basics for an app in managed environment. Uses ThoughtWorks Studios CI/CD server named Go as an example application.
Goes into finding performance problems, pitfalls, anti-patterns, problem patterns for typical performance issues and war-stories extracted from Go performance tuning work.
The Codex of Business Writing Software for Real-World Solutions 2.pptx
Performance Tuning 101
1. Problem Approach Nature of fixes War Stories
Performance Tuning 101
An introduction to performance tuning an application in managed
environment
Pavan KS1 Janmejay Singh2
1 mail: itspanzi@gmail.com
blog: http://itspanzi.blogspot.com
2 mail: singh.janmejay@gmail.com
blog: http://codehunk.wordpress.com
Rootconf Bangalore, May 2012
2. Problem Approach Nature of fixes War Stories
Gist
Don’t be out there with a gun looking for problems.
3. Problem Approach Nature of fixes War Stories
Gist
Don’t be out there with a gun looking for problems.
Do not go overboard, know your end goal.
4. Problem Approach Nature of fixes War Stories
Gist
Don’t be out there with a gun looking for problems.
Do not go overboard, know your end goal.
Intution tends to work very poorly. Belive proof, not gut feel.
5. Problem Approach Nature of fixes War Stories
Gist
Don’t be out there with a gun looking for problems.
Do not go overboard, know your end goal.
Intution tends to work very poorly. Belive proof, not gut feel.
Understand your environment. Know what to measure.
6. Problem Approach Nature of fixes War Stories
Gist
Don’t be out there with a gun looking for problems.
Do not go overboard, know your end goal.
Intution tends to work very poorly. Belive proof, not gut feel.
Understand your environment. Know what to measure.
Round pegs for round holes. Use appropriate tools.
7. Problem Approach Nature of fixes War Stories
Gist
Don’t be out there with a gun looking for problems.
Do not go overboard, know your end goal.
Intution tends to work very poorly. Belive proof, not gut feel.
Understand your environment. Know what to measure.
Round pegs for round holes. Use appropriate tools.
Jumps places, unlike functional bugs, look beyond local
impact.
8. Problem Approach Nature of fixes War Stories
Gist
Don’t be out there with a gun looking for problems.
Do not go overboard, know your end goal.
Intution tends to work very poorly. Belive proof, not gut feel.
Understand your environment. Know what to measure.
Round pegs for round holes. Use appropriate tools.
Jumps places, unlike functional bugs, look beyond local
impact.
Requires mechanical approach, attack worst problem first.
9. Problem Approach Nature of fixes War Stories
Gist
Don’t be out there with a gun looking for problems.
Do not go overboard, know your end goal.
Intution tends to work very poorly. Belive proof, not gut feel.
Understand your environment. Know what to measure.
Round pegs for round holes. Use appropriate tools.
Jumps places, unlike functional bugs, look beyond local
impact.
Requires mechanical approach, attack worst problem first.
Prefer correctness over performance.
10. Problem Approach Nature of fixes War Stories
Gist
Don’t be out there with a gun looking for problems.
Do not go overboard, know your end goal.
Intution tends to work very poorly. Belive proof, not gut feel.
Understand your environment. Know what to measure.
Round pegs for round holes. Use appropriate tools.
Jumps places, unlike functional bugs, look beyond local
impact.
Requires mechanical approach, attack worst problem first.
Prefer correctness over performance.
Reasoning every fix (of paramount importance in
multithreaded app).
11. Problem Approach Nature of fixes War Stories
Gist
Don’t be out there with a gun looking for problems.
Do not go overboard, know your end goal.
Intution tends to work very poorly. Belive proof, not gut feel.
Understand your environment. Know what to measure.
Round pegs for round holes. Use appropriate tools.
Jumps places, unlike functional bugs, look beyond local
impact.
Requires mechanical approach, attack worst problem first.
Prefer correctness over performance.
Reasoning every fix (of paramount importance in
multithreaded app).
Do one fix at a time, allows easy reasoning and
verification, can be reverted.
12. Problem Approach Nature of fixes War Stories
Gist
Don’t be out there with a gun looking for problems.
Do not go overboard, know your end goal.
Intution tends to work very poorly. Belive proof, not gut feel.
Understand your environment. Know what to measure.
Round pegs for round holes. Use appropriate tools.
Jumps places, unlike functional bugs, look beyond local
impact.
Requires mechanical approach, attack worst problem first.
Prefer correctness over performance.
Reasoning every fix (of paramount importance in
multithreaded app).
Do one fix at a time, allows easy reasoning and
verification, can be reverted.
Don’t jump in the lake without knowing whats waiting
inside.
13. Problem Approach Nature of fixes War Stories
Context
App in reference: ThoughtWorks Studios - Go (CI/CD
suite)
14. Problem Approach Nature of fixes War Stories
Context
App in reference: ThoughtWorks Studios - Go (CI/CD
suite)
Java (backend) and JRuby on Rails (frontend) webapp.
15. Problem Approach Nature of fixes War Stories
Context
App in reference: ThoughtWorks Studios - Go (CI/CD
suite)
Java (backend) and JRuby on Rails (frontend) webapp.
Managed environment.
16. Problem Approach Nature of fixes War Stories
Context
App in reference: ThoughtWorks Studios - Go (CI/CD
suite)
Java (backend) and JRuby on Rails (frontend) webapp.
Managed environment.
GNU/Linux
17. Problem Approach Nature of fixes War Stories
Context
App in reference: ThoughtWorks Studios - Go (CI/CD
suite)
Java (backend) and JRuby on Rails (frontend) webapp.
Managed environment.
GNU/Linux
Approach and learning is independent of tech-stack.
18. Problem Approach Nature of fixes War Stories
Outline
1 Problem
Identify scope
Acceptance criteria
2 Approach
How to analyze it
Measure
Fix
3 Nature of fixes
Synchronization
GC
DB
Thrashing
Pitfalls
4 War Stories
GC
Locking
Caching
19. Problem Approach Nature of fixes War Stories
Outline
1 Problem
Identify scope
Acceptance criteria
2 Approach
How to analyze it
Measure
Fix
3 Nature of fixes
Synchronization
GC
DB
Thrashing
Pitfalls
4 War Stories
GC
Locking
Caching
20. Problem Approach Nature of fixes War Stories
Don’t go looking for performance issues
Doing it proactively is a wild goose chase!
Varying parameters and running into a real problem is improbable, if
not impossible.
21. Problem Approach Nature of fixes War Stories
Remember...
Don’t be out there with a gun looking for problems.
Its easier to reproduce and fix a performance issue when you have
seen it happen in a real setup. That way, you also know for sure, it is
real.
22. Problem Approach Nature of fixes War Stories
My app is slow, it doesn’t respond...
In case of Go, this could mean,
The dashboard is slow
23. Problem Approach Nature of fixes War Stories
My app is slow, it doesn’t respond...
In case of Go, this could mean,
The dashboard is slow
Javascript is slow
24. Problem Approach Nature of fixes War Stories
My app is slow, it doesn’t respond...
In case of Go, this could mean,
The dashboard is slow
Javascript is slow
Builds taking too long to schedule
25. Problem Approach Nature of fixes War Stories
My app is slow, it doesn’t respond...
In case of Go, this could mean,
The dashboard is slow
Javascript is slow
Builds taking too long to schedule
Worker processes picking up work too late
26. Problem Approach Nature of fixes War Stories
My app is slow, it doesn’t respond...
In case of Go, this could mean,
The dashboard is slow
Javascript is slow
Builds taking too long to schedule
Worker processes picking up work too late
SCM changes reflect after a long time
27. Problem Approach Nature of fixes War Stories
My app is slow, it doesn’t respond...
In case of Go, this could mean,
The dashboard is slow
Javascript is slow
Builds taking too long to schedule
Worker processes picking up work too late
SCM changes reflect after a long time
Process is hogging resources(requires 16 cores, 8 gig ram
for a basic setup)
28. Problem Approach Nature of fixes War Stories
Outline
1 Problem
Identify scope
Acceptance criteria
2 Approach
How to analyze it
Measure
Fix
3 Nature of fixes
Synchronization
GC
DB
Thrashing
Pitfalls
4 War Stories
GC
Locking
Caching
29. Problem Approach Nature of fixes War Stories
Benchmark
How slow is slow?
30. Problem Approach Nature of fixes War Stories
Benchmark
How slow is slow?
How fast is fast?
31. Problem Approach Nature of fixes War Stories
Benchmark
How slow is slow?
How fast is fast?
Benchmark using realistic setup.
32. Problem Approach Nature of fixes War Stories
Benchmark
How slow is slow?
How fast is fast?
Benchmark using realistic setup.
Use a repeatable and automated setup.
33. Problem Approach Nature of fixes War Stories
Remember...
Do not go overboard.
Know your end goal.
34. Problem Approach Nature of fixes War Stories
Outline
1 Problem
Identify scope
Acceptance criteria
2 Approach
How to analyze it
Measure
Fix
3 Nature of fixes
Synchronization
GC
DB
Thrashing
Pitfalls
4 War Stories
GC
Locking
Caching
35. Problem Approach Nature of fixes War Stories
Trust in testing
What if I start at fix?
36. Problem Approach Nature of fixes War Stories
Trust in testing
What if I start at fix?
What if I stop at fix?
37. Problem Approach Nature of fixes War Stories
Trust in testing
What if I start at fix?
What if I stop at fix?
Why do I need to iterate?
38. Problem Approach Nature of fixes War Stories
Trust in testing
What if I start at fix?
What if I stop at fix?
Why do I need to iterate?
How long should I iterate?
39. Problem Approach Nature of fixes War Stories
Remember...
Intution tends to work very poorly.
Belive proof, not gut feel.
40. Problem Approach Nature of fixes War Stories
Outline
1 Problem
Identify scope
Acceptance criteria
2 Approach
How to analyze it
Measure
Fix
3 Nature of fixes
Synchronization
GC
DB
Thrashing
Pitfalls
4 War Stories
GC
Locking
Caching
41. Problem Approach Nature of fixes War Stories
What to measure
Easy and obvious:
Page response time
42. Problem Approach Nature of fixes War Stories
What to measure
Easy and obvious:
Page response time
Not so easy:
App throughput
Enqueue/Dequeue time for message queues
CPU/IO churn
Database performance
Lock contention
Managed environment specific parameters(Memory usage,
GC churn etc).
Environmental issues(too many processes, too low
memory, very slow IO).
43. Problem Approach Nature of fixes War Stories
Remember...
Understand your environment.
Know what to measure.
44. Problem Approach Nature of fixes War Stories
Tools to measure
User load
ab
htperf
45. Problem Approach Nature of fixes War Stories
Tools to measure
User load
ab
htperf
App throughput: measure app-level
churn(creation/processing/deletion/delivery) counts
46. Problem Approach Nature of fixes War Stories
Tools to measure
User load
ab
htperf
App throughput: measure app-level
churn(creation/processing/deletion/delivery) counts
Enqueue/Dequeue time: Use broker’s management
console/logs/statistics.
47. Problem Approach Nature of fixes War Stories
Tools to measure
User load
ab
htperf
App throughput: measure app-level
churn(creation/processing/deletion/delivery) counts
Enqueue/Dequeue time: Use broker’s management
console/logs/statistics.
CPU/IO churn: Load Avg, iostat, klogd, inotify.
48. Problem Approach Nature of fixes War Stories
Tools to measure
User load
ab
htperf
App throughput: measure app-level
churn(creation/processing/deletion/delivery) counts
Enqueue/Dequeue time: Use broker’s management
console/logs/statistics.
CPU/IO churn: Load Avg, iostat, klogd, inotify.
Database performance: Profiler.
49. Problem Approach Nature of fixes War Stories
Tools to measure
User load
ab
htperf
App throughput: measure app-level
churn(creation/processing/deletion/delivery) counts
Enqueue/Dequeue time: Use broker’s management
console/logs/statistics.
CPU/IO churn: Load Avg, iostat, klogd, inotify.
Database performance: Profiler.
Lock contention: Profiler.
50. Problem Approach Nature of fixes War Stories
Tools to measure
User load
ab
htperf
App throughput: measure app-level
churn(creation/processing/deletion/delivery) counts
Enqueue/Dequeue time: Use broker’s management
console/logs/statistics.
CPU/IO churn: Load Avg, iostat, klogd, inotify.
Database performance: Profiler.
Lock contention: Profiler.
Managed env specific params: GC logs, GC profiling.
51. Problem Approach Nature of fixes War Stories
Tools to measure
User load
ab
htperf
App throughput: measure app-level
churn(creation/processing/deletion/delivery) counts
Enqueue/Dequeue time: Use broker’s management
console/logs/statistics.
CPU/IO churn: Load Avg, iostat, klogd, inotify.
Database performance: Profiler.
Lock contention: Profiler.
Managed env specific params: GC logs, GC profiling.
Environmental issues: ps, dmesg, /proc/<pid>/status,
/proc/net/dev, syslog, messages.
52. Problem Approach Nature of fixes War Stories
Remember...
Round pegs for round holes.
Use appropriate tools. Configure them correctly with least possible
overhead.
53. Problem Approach Nature of fixes War Stories
Outline
1 Problem
Identify scope
Acceptance criteria
2 Approach
How to analyze it
Measure
Fix
3 Nature of fixes
Synchronization
GC
DB
Thrashing
Pitfalls
4 War Stories
GC
Locking
Caching
54. Problem Approach Nature of fixes War Stories
How to carry out fixes
Understand libraries: read fine print in documentation, or
better yet, read code.
55. Problem Approach Nature of fixes War Stories
How to carry out fixes
Understand libraries: read fine print in documentation, or
better yet, read code.
Be mechanical.
56. Problem Approach Nature of fixes War Stories
How to carry out fixes
Understand libraries: read fine print in documentation, or
better yet, read code.
Be mechanical.
Use exhaggerated load to maginify the problem.
57. Problem Approach Nature of fixes War Stories
How to carry out fixes
Understand libraries: read fine print in documentation, or
better yet, read code.
Be mechanical.
Use exhaggerated load to maginify the problem.
Use the same load profile to test your fix.
58. Problem Approach Nature of fixes War Stories
How to carry out fixes
Understand libraries: read fine print in documentation, or
better yet, read code.
Be mechanical.
Use exhaggerated load to maginify the problem.
Use the same load profile to test your fix.
Be rigorous.
59. Problem Approach Nature of fixes War Stories
How to carry out fixes
Understand libraries: read fine print in documentation, or
better yet, read code.
Be mechanical.
Use exhaggerated load to maginify the problem.
Use the same load profile to test your fix.
Be rigorous.
Mob pair on complicated fixes, especially fixes involving
data mutation across threads.
60. Problem Approach Nature of fixes War Stories
How to carry out fixes
Understand libraries: read fine print in documentation, or
better yet, read code.
Be mechanical.
Use exhaggerated load to maginify the problem.
Use the same load profile to test your fix.
Be rigorous.
Mob pair on complicated fixes, especially fixes involving
data mutation across threads.
Attack one and only one problem.
61. Problem Approach Nature of fixes War Stories
How to carry out fixes
Understand libraries: read fine print in documentation, or
better yet, read code.
Be mechanical.
Use exhaggerated load to maginify the problem.
Use the same load profile to test your fix.
Be rigorous.
Mob pair on complicated fixes, especially fixes involving
data mutation across threads.
Attack one and only one problem.
Having said all that, correctness always comes first.
62. Problem Approach Nature of fixes War Stories
Remember...
Requires mechanical approach
Attack worst problem first
Reason every fix
Reasoning is of paramount importance in multithreaded app
Do one fix at a time
Allows easy reasoning and verification, can be reverted
Prefer correctness over performance
63. Problem Approach Nature of fixes War Stories
Outline
1 Problem
Identify scope
Acceptance criteria
2 Approach
How to analyze it
Measure
Fix
3 Nature of fixes
Synchronization
GC
DB
Thrashing
Pitfalls
4 War Stories
GC
Locking
Caching
64. Problem Approach Nature of fixes War Stories
Lock granularity and lockless
When you fix lock contention somewhere, ensure it hasn’t
moved elsewhere, making overall performance worse.
65. Problem Approach Nature of fixes War Stories
Lock granularity and lockless
When you fix lock contention somewhere, ensure it hasn’t
moved elsewhere, making overall performance worse.
More than one shared variable mandates synchronization
usually, watch out!
66. Problem Approach Nature of fixes War Stories
Lock granularity and lockless
When you fix lock contention somewhere, ensure it hasn’t
moved elsewhere, making overall performance worse.
More than one shared variable mandates synchronization
usually, watch out!
Atomic operations and globaly unique objects(like interned
symbols) are your friends.
67. Problem Approach Nature of fixes War Stories
Remember...
Jumps places, unlike functional bugs
Look beyond local impact of a fix
68. Problem Approach Nature of fixes War Stories
Outline
1 Problem
Identify scope
Acceptance criteria
2 Approach
How to analyze it
Measure
Fix
3 Nature of fixes
Synchronization
GC
DB
Thrashing
Pitfalls
4 War Stories
GC
Locking
Caching
69. Problem Approach Nature of fixes War Stories
Garbage Collection
When you hear unresponsiveness is intermittent, be
alarmed!
It probably is GC
It is likely not easy to fix
Root cause may be you or a library
70. Problem Approach Nature of fixes War Stories
Garbage Collection
When you hear unresponsiveness is intermittent, be
alarmed!
It probably is GC
It is likely not easy to fix
Root cause may be you or a library
Scan GC logs
71. Problem Approach Nature of fixes War Stories
Garbage Collection
When you hear unresponsiveness is intermittent, be
alarmed!
It probably is GC
It is likely not easy to fix
Root cause may be you or a library
Scan GC logs
Profile to see if majority of allocations are from just a
handful of calls
If its several calls with each contributing very little to the pie,
don’t even try fixing it.
72. Problem Approach Nature of fixes War Stories
Garbage Collection
When you hear unresponsiveness is intermittent, be
alarmed!
It probably is GC
It is likely not easy to fix
Root cause may be you or a library
Scan GC logs
Profile to see if majority of allocations are from just a
handful of calls
If its several calls with each contributing very little to the pie,
don’t even try fixing it.
Understand GC ergonomics and each algorithm(if you
have alternatives) in detail.
Tweek your max/min permitted allocation judiciously,
malloc/free are expensive system calls.
73. Problem Approach Nature of fixes War Stories
Outline
1 Problem
Identify scope
Acceptance criteria
2 Approach
How to analyze it
Measure
Fix
3 Nature of fixes
Synchronization
GC
DB
Thrashing
Pitfalls
4 War Stories
GC
Locking
Caching
74. Problem Approach Nature of fixes War Stories
SQL Database
Optimize connection pool size.
Expand is very expensive.
Is dispatched on caller thread(unless your connection pool
is super-human).
75. Problem Approach Nature of fixes War Stories
SQL Database
Optimize connection pool size.
Expand is very expensive.
Is dispatched on caller thread(unless your connection pool
is super-human).
Watch out for slow queries, it can snowball.
76. Problem Approach Nature of fixes War Stories
SQL Database
Optimize connection pool size.
Expand is very expensive.
Is dispatched on caller thread(unless your connection pool
is super-human).
Watch out for slow queries, it can snowball.
Understand your DB settings, MVCC, table-locking
scenarios, row-locking scenarios, transaction isolation
tradeoffs.
77. Problem Approach Nature of fixes War Stories
SQL Database
Optimize connection pool size.
Expand is very expensive.
Is dispatched on caller thread(unless your connection pool
is super-human).
Watch out for slow queries, it can snowball.
Understand your DB settings, MVCC, table-locking
scenarios, row-locking scenarios, transaction isolation
tradeoffs.
Contd...
78. Problem Approach Nature of fixes War Stories
SQL Database contd...
Explain and Analyze are your friends.
79. Problem Approach Nature of fixes War Stories
SQL Database contd...
Explain and Analyze are your friends.
Understanding implications of file-system IO helps realize
table scans can be horrible
DB runs with fixed page cache.
Table scan can wipe it all cold.
Use indexes to avoid table scan.
Indexes are order sensitive.
Tweek inner queries to reduce selectivity.
80. Problem Approach Nature of fixes War Stories
SQL Database contd...
Explain and Analyze are your friends.
Understanding implications of file-system IO helps realize
table scans can be horrible
DB runs with fixed page cache.
Table scan can wipe it all cold.
Use indexes to avoid table scan.
Indexes are order sensitive.
Tweek inner queries to reduce selectivity.
Every DB may be subtly different(joins can be more
expensive than materialzed views(if you have em))
81. Problem Approach Nature of fixes War Stories
SQL Database contd...
Explain and Analyze are your friends.
Understanding implications of file-system IO helps realize
table scans can be horrible
DB runs with fixed page cache.
Table scan can wipe it all cold.
Use indexes to avoid table scan.
Indexes are order sensitive.
Tweek inner queries to reduce selectivity.
Every DB may be subtly different(joins can be more
expensive than materialzed views(if you have em))
Denormalization can help crack tough nuts.
82. Problem Approach Nature of fixes War Stories
Outline
1 Problem
Identify scope
Acceptance criteria
2 Approach
How to analyze it
Measure
Fix
3 Nature of fixes
Synchronization
GC
DB
Thrashing
Pitfalls
4 War Stories
GC
Locking
Caching
83. Problem Approach Nature of fixes War Stories
Context Switch Thrashing
Too few cores, too many processes will obviously hurt.
84. Problem Approach Nature of fixes War Stories
Context Switch Thrashing
Too few cores, too many processes will obviously hurt.
Too many locks will cause too many sleeps, hence too
much ctx switching.
85. Problem Approach Nature of fixes War Stories
Context Switch Thrashing
Too few cores, too many processes will obviously hurt.
Too many locks will cause too many sleeps, hence too
much ctx switching.
Ctx switch may schedule a different process, causing CPU
caches and TLB to get cold.
86. Problem Approach Nature of fixes War Stories
Context Switch Thrashing
Too few cores, too many processes will obviously hurt.
Too many locks will cause too many sleeps, hence too
much ctx switching.
Ctx switch may schedule a different process, causing CPU
caches and TLB to get cold.
Too frequent IO will cause io-wait, hence switch.
Use of swap can completely kill performance. Swap is for
emergency, not for regular usage.
Over logging can hurt, be careful!
Concept of buffered IO exists for a reason, use it!
Use memory mapped IO if it makes sense.
87. Problem Approach Nature of fixes War Stories
Context Switch Thrashing
Too few cores, too many processes will obviously hurt.
Too many locks will cause too many sleeps, hence too
much ctx switching.
Ctx switch may schedule a different process, causing CPU
caches and TLB to get cold.
Too frequent IO will cause io-wait, hence switch.
Use of swap can completely kill performance. Swap is for
emergency, not for regular usage.
Over logging can hurt, be careful!
Concept of buffered IO exists for a reason, use it!
Use memory mapped IO if it makes sense.
Assigning very low max memory limit can cause frequent
GCs, allocate enough memory so its your functional thread
that works and not GC threads.
88. Problem Approach Nature of fixes War Stories
Outline
1 Problem
Identify scope
Acceptance criteria
2 Approach
How to analyze it
Measure
Fix
3 Nature of fixes
Synchronization
GC
DB
Thrashing
Pitfalls
4 War Stories
GC
Locking
Caching
89. Problem Approach Nature of fixes War Stories
Common Pitfalls
Prejudice
SSL is bad
Using asynchronicity to avoid fixing root cause
90. Problem Approach Nature of fixes War Stories
Common Pitfalls
Prejudice
SSL is bad
Using asynchronicity to avoid fixing root cause
Lack of testing: In Go context, myth of 150 agent limit.
91. Problem Approach Nature of fixes War Stories
Common Pitfalls
Prejudice
SSL is bad
Using asynchronicity to avoid fixing root cause
Lack of testing: In Go context, myth of 150 agent limit.
Lack of setup understanding: Misconception that 64 bit
JVM is always bad.
92. Problem Approach Nature of fixes War Stories
Common Pitfalls
Prejudice
SSL is bad
Using asynchronicity to avoid fixing root cause
Lack of testing: In Go context, myth of 150 agent limit.
Lack of setup understanding: Misconception that 64 bit
JVM is always bad.
Atomic operations and globaly unique objects(like interned
symbols) are your friends.
93. Problem Approach Nature of fixes War Stories
Common Pitfalls
Prejudice
SSL is bad
Using asynchronicity to avoid fixing root cause
Lack of testing: In Go context, myth of 150 agent limit.
Lack of setup understanding: Misconception that 64 bit
JVM is always bad.
Atomic operations and globaly unique objects(like interned
symbols) are your friends.
Unchartred waters
ERB garbage churn
Strange JVM allocations
Single threaded database
JRuby class hirarchy modification(runtime level) locks.
94. Problem Approach Nature of fixes War Stories
Know what you are getting into...
Know what you are getting into...
95. Problem Approach Nature of fixes War Stories
Gist, to reiterate...
Do not go overboard, know your end goal.
Intution tends to work very poorly. Belive proof, not gut feel.
Understand your environment. Know what to measure.
Round pegs for round holes. Use appropriate tools.
Jumps places, unlike functional bugs, look beyond local
impact.
Requires mechanical approach, attack worst problem first.
Prefer correctness over performance.
Reasoning every fix (of paramount importance in
multithreaded app).
Do one fix at a time, allows easy reasoning and
verification, can be reverted.
Don’t jump in the lake without knowing whats waiting
inside.
96. Problem Approach Nature of fixes War Stories
Outline
1 Problem
Identify scope
Acceptance criteria
2 Approach
How to analyze it
Measure
Fix
3 Nature of fixes
Synchronization
GC
DB
Thrashing
Pitfalls
4 War Stories
GC
Locking
Caching
97. Problem Approach Nature of fixes War Stories
Garbage churn issues
Debug log statements
98. Problem Approach Nature of fixes War Stories
Garbage churn issues
Debug log statements
ERB rendering
99. Problem Approach Nature of fixes War Stories
Garbage churn issues
Debug log statements
ERB rendering
Pooling objects with significant memory footprint
XML parser factory
Buffered reader
100. Problem Approach Nature of fixes War Stories
Garbage churn issues
Debug log statements
ERB rendering
Pooling objects with significant memory footprint
XML parser factory
Buffered reader
Stream being dereferenced into a string
101. Problem Approach Nature of fixes War Stories
Garbage churn issues
Debug log statements
ERB rendering
Pooling objects with significant memory footprint
XML parser factory
Buffered reader
Stream being dereferenced into a string
Thread locals to the rescue(used for caching big reusable
objects)
102. Problem Approach Nature of fixes War Stories
Outline
1 Problem
Identify scope
Acceptance criteria
2 Approach
How to analyze it
Measure
Fix
3 Nature of fixes
Synchronization
GC
DB
Thrashing
Pitfalls
4 War Stories
GC
Locking
Caching
103. Problem Approach Nature of fixes War Stories
Contention and reasoning issues
Locking on interned strings generated for context sensitive
locking
104. Problem Approach Nature of fixes War Stories
Contention and reasoning issues
Locking on interned strings generated for context sensitive
locking
Death of synchronized methods
105. Problem Approach Nature of fixes War Stories
Contention and reasoning issues
Locking on interned strings generated for context sensitive
locking
Death of synchronized methods
Copy on to stack(using lock), mutate locklessly, copy back
to shared memory(using lock)
106. Problem Approach Nature of fixes War Stories
Contention and reasoning issues
Locking on interned strings generated for context sensitive
locking
Death of synchronized methods
Copy on to stack(using lock), mutate locklessly, copy back
to shared memory(using lock)
Use read/write lock when appropriate
107. Problem Approach Nature of fixes War Stories
Contention and reasoning issues
Locking on interned strings generated for context sensitive
locking
Death of synchronized methods
Copy on to stack(using lock), mutate locklessly, copy back
to shared memory(using lock)
Use read/write lock when appropriate
Use atomic operations for lockess concurrency
108. Problem Approach Nature of fixes War Stories
Contention and reasoning issues
Locking on interned strings generated for context sensitive
locking
Death of synchronized methods
Copy on to stack(using lock), mutate locklessly, copy back
to shared memory(using lock)
Use read/write lock when appropriate
Use atomic operations for lockess concurrency
Concurrent data-structure != thread-safety
109. Problem Approach Nature of fixes War Stories
Contention and reasoning issues
Locking on interned strings generated for context sensitive
locking
Death of synchronized methods
Copy on to stack(using lock), mutate locklessly, copy back
to shared memory(using lock)
Use read/write lock when appropriate
Use atomic operations for lockess concurrency
Concurrent data-structure != thread-safety
Take difficult design tradeoffs to keep code requiring a
class of lock in one place
110. Problem Approach Nature of fixes War Stories
Contention and reasoning issues
Locking on interned strings generated for context sensitive
locking
Death of synchronized methods
Copy on to stack(using lock), mutate locklessly, copy back
to shared memory(using lock)
Use read/write lock when appropriate
Use atomic operations for lockess concurrency
Concurrent data-structure != thread-safety
Take difficult design tradeoffs to keep code requiring a
class of lock in one place
Double check locking can be very helpful in frequently
called code
111. Problem Approach Nature of fixes War Stories
Outline
1 Problem
Identify scope
Acceptance criteria
2 Approach
How to analyze it
Measure
Fix
3 Nature of fixes
Synchronization
GC
DB
Thrashing
Pitfalls
4 War Stories
GC
Locking
Caching
112. Problem Approach Nature of fixes War Stories
Caching
Eager caching is dangerous(using miss => load => prime
=> hit works better in practice)
113. Problem Approach Nature of fixes War Stories
Caching
Eager caching is dangerous(using miss => load => prime
=> hit works better in practice)
Testing cache with boundry conditions
Test to ensure cache-hit
Ensure boundry conditions are well understood, overflow is
judiciously defined/used
114. Problem Approach Nature of fixes War Stories
Caching
Eager caching is dangerous(using miss => load => prime
=> hit works better in practice)
Testing cache with boundry conditions
Test to ensure cache-hit
Ensure boundry conditions are well understood, overflow is
judiciously defined/used
LRU eviction is the right(only?) solution sometimes
115. Problem Approach Nature of fixes War Stories
Caching
Eager caching is dangerous(using miss => load => prime
=> hit works better in practice)
Testing cache with boundry conditions
Test to ensure cache-hit
Ensure boundry conditions are well understood, overflow is
judiciously defined/used
LRU eviction is the right(only?) solution sometimes
Double check cache invalidation flows, its hard, sometimes
bleeds across classes.
116. Problem Approach Nature of fixes War Stories
Caching
Eager caching is dangerous(using miss => load => prime
=> hit works better in practice)
Testing cache with boundry conditions
Test to ensure cache-hit
Ensure boundry conditions are well understood, overflow is
judiciously defined/used
LRU eviction is the right(only?) solution sometimes
Double check cache invalidation flows, its hard, sometimes
bleeds across classes.
Be vigilant when using hirarchical caches(no
associative-arrays or maps should be cached), its easy to
run memory out with this.
117. Problem Approach Nature of fixes War Stories
Caching contd...
Do not cache uncommited data or invalidate cache before
commit(transactional caches may come in handy for this).
118. Problem Approach Nature of fixes War Stories
Caching contd...
Do not cache uncommited data or invalidate cache before
commit(transactional caches may come in handy for this).
Ensure transactions use cached data iff no other
transaction modifies it or ensure transactionally consistent
invalidation.
119. Problem Approach Nature of fixes War Stories
Caching contd...
Do not cache uncommited data or invalidate cache before
commit(transactional caches may come in handy for this).
Ensure transactions use cached data iff no other
transaction modifies it or ensure transactionally consistent
invalidation.
Understand your caching library, too low overflow limit may
cause IO issues.
120. Problem Approach Nature of fixes War Stories
Caching contd...
Do not cache uncommited data or invalidate cache before
commit(transactional caches may come in handy for this).
Ensure transactions use cached data iff no other
transaction modifies it or ensure transactionally consistent
invalidation.
Understand your caching library, too low overflow limit may
cause IO issues.
Use one and only one cache instance(its easy to end up in
a memory runout with this). We had view fragments, urls,
domain objects and view models cached in the same
instance.
121. Problem Approach Nature of fixes War Stories
Outline
1 Problem
Identify scope
Acceptance criteria
2 Approach
How to analyze it
Measure
Fix
3 Nature of fixes
Synchronization
GC
DB
Thrashing
Pitfalls
4 War Stories
GC
Locking
Caching
122. Problem Approach Nature of fixes War Stories
Javascript
JS dom frequent offset/size calls hurt
123. Problem Approach Nature of fixes War Stories
Javascript
JS dom frequent offset/size calls hurt
Framework provided foreach loops are bad, use native for
loops.
124. Problem Approach Nature of fixes War Stories
Javascript
JS dom frequent offset/size calls hurt
Framework provided foreach loops are bad, use native for
loops.
Using beautiful framework provided methods like filter, find
and select can have detrimental consequences.
125. Problem Approach Nature of fixes War Stories
Javascript
JS dom frequent offset/size calls hurt
Framework provided foreach loops are bad, use native for
loops.
Using beautiful framework provided methods like filter, find
and select can have detrimental consequences.
Avoid multiple lookups for the same element, use a loaded
copy as far as possible.
126. Problem Approach Nature of fixes War Stories
Javascript
JS dom frequent offset/size calls hurt
Framework provided foreach loops are bad, use native for
loops.
Using beautiful framework provided methods like filter, find
and select can have detrimental consequences.
Avoid multiple lookups for the same element, use a loaded
copy as far as possible.
Avoid CSS selectors in frequently called code.
127. Problem Approach Nature of fixes War Stories
Javascript
JS dom frequent offset/size calls hurt
Framework provided foreach loops are bad, use native for
loops.
Using beautiful framework provided methods like filter, find
and select can have detrimental consequences.
Avoid multiple lookups for the same element, use a loaded
copy as far as possible.
Avoid CSS selectors in frequently called code.
Avoid too much DOM churn.
128. Problem Approach Nature of fixes War Stories
Javascript
JS dom frequent offset/size calls hurt
Framework provided foreach loops are bad, use native for
loops.
Using beautiful framework provided methods like filter, find
and select can have detrimental consequences.
Avoid multiple lookups for the same element, use a loaded
copy as far as possible.
Avoid CSS selectors in frequently called code.
Avoid too much DOM churn.
Use tools like Chromium dev toolset to nail leaks and
profile methods.
129. Problem Approach Nature of fixes War Stories
Javascript
JS dom frequent offset/size calls hurt
Framework provided foreach loops are bad, use native for
loops.
Using beautiful framework provided methods like filter, find
and select can have detrimental consequences.
Avoid multiple lookups for the same element, use a loaded
copy as far as possible.
Avoid CSS selectors in frequently called code.
Avoid too much DOM churn.
Use tools like Chromium dev toolset to nail leaks and
profile methods.
Auto-refreshing dashboards should refresh only visible and
changed area.
130. Problem Approach Nature of fixes War Stories
Questions?
Questions?