Performance Tuning: Pulling a Rabbit From a Hat George Barnett of Atlassian

1. 1

2. Pulling a rabbit from a hat
How to make your apps run faster
George Barnett, Performance Engineer, Atlassian
2
2

3. Why are we here?
• How to make your Atlassian software run faster
• What doesnʼt work
• What Atlassian does to improve performance
• No development HOWTOs or code!
3
3

4. Who can improve performance?
• Atlassian
• YOU! (Thanks for watching!)
4
4

5. Performance Engineering
• Isolate performance issues before they become headaches.
• Make sure these issues get fixed.
5
5

6. Simple code lifecycle
Developer writes code
commit
Unit & Functional tests Reporting
tests pass
Software artefact
Performance Testing Reporting
6
6

7. Performance Testing Scripts
• Apache JMeter
• In beta for JIRA
• Ask support@atlassian.com
• Available for Confluence
• http://confluence.atlassian.com/display/DOC/Performance+Testing+Scripts
7
7

8. Performance Testing
• Real world data and load
• Public Atlassian instances
• Performance test suite
• JMeter & scripts
• Automation
• Maven & Bamboo
8
8

9. Performance Testing
• Daily performance tests
• Most products & libraries
• Weekly soak tests
• Run for much longer than daily tests
• Reporting for developers
• Build screens, dashboards, notifications
9
9

10. Reporting
10
10

11. Does it work?
Jira 3.13 Jira 4.0 Confluence 2.10 Confluence 3.0
90 600
68 450
45 300
23 150
0 0
Browse Issue (ms) Browse Page (ms)
11
11

12. Upgrade your Atlassian Software!
• Dozens of Performance issues fixed
• Release criteria includes performance
12
12

13. You are here
• How I did the testing
• Hardware & Software
• Must have improvements
• Upgrade Java, Hardware, Remove VMWare and tune Garbage Collection
• Worth doing
• Gigabit vs 100mbit, Heap size tuning, use an SSD
• Sadly, not useful
• Tune MySQL, Replace MySQL
13
13

14. The contenders
• Dell PE 850 - “WALL-E”
• Xeon X3220 2.4Ghz Quad Core
• Quad Core, 2 x 4Mb L3
• 8Gb DDR2 RAM
• 2 x 7.2K SAS Disks
• 1 x Intel X25-E 32GB SSD
• ~$1500
14
14

15. The contenders
• Dell PE 2950 - “Johnny 5”
• 2 x Xeon E5405 2.0Ghz
• Quad Core, 2 x 6M L3
• 32Gb RAM
• 2 x 72Gb 15K Drives
• ~$4000
15
15

16. The contenders
• Dell PE R610 - “EVE”
• 2 x Xeon E5520 2.2Ghz
• Quad Core, 8M “SmartCache”
• 32Gb Ram
• 2 x 146G 15K drives
• ~$4000
16
16

17. The Workload
• JIRA 4, MySQL 5, RHEL 5
• ~70,000 Issues, ~80 projects
• ~30 reqs/sec, JMeter 2.3.4 (patched, JSR-223)
• Traffic modelled on http://jira.atlassian.com with higher writes
17
17

18. About the graphs
• Most show Average Response Time in Milliseconds
• Includes time to generate AND deliver the response
• Does not include any browser time
• Arrows show if lower or higher is better!
18
18

19. You are here
• How I did the testing
• Hardware & Software
• Must have improvements
• Upgrade Java, Hardware, Remove VMWare and tune Garbage Collection
• Worth doing
• Gigabit vs 100mbit, Heap size tuning, use an SSD
• Sadly, not useful
• Tune MySQL, Replace MySQL
19
19

20. Must. Have.
20
20

21. Upgrade Java 1.5 to 1.6
• Advances in compiler / VM technology
• Produces better runtime code
• Able to use modern instructions, eg SSE
• Able to use modern hardware, eg NUMA
21
21

22. Java 1.5 vs 1.6 (Johnny 5)
Java 1.5.0_18 Java 1.6.0_16
1500
1125
750
375
0
Average Response Time (ms)
6.3X
22
22

23. Java 1.5 vs 1.6 (Johnny 5)
Java 1.5.0_18 Java 1.6.0_16
380
285
190
95
0
Average CPU (%)
26%
23
23

24. Upgrade Java to 1.6
• 3 x HotSpot VM updates in 1.6.0_XX
• Advancing technology
• Compressed OOPS on 64bit
• Escape Analysis
• G1 Garbage Collector
• NUMA
24
24

25. Java 1.6 vs 1.6 (Eve)
Java 1.6.0_7 Java 1.6.0_16
148
146
143
141
138 7%
Average Response Time (ms)
25
25

26. Hardware Upgrades
• Hyper-Threading
• QuickPath Interconnect
• Nehalem E5520 (Eve) have 3X on die DDR3 Memory Controllers
• Memory throughput improvements - good for Java!
Technology Speed
DDR2-800 (dual channel) 6.4 GB/sec
DDR3-1333 (dual channel) 21.3 GB/sec
26
26

27. FSB architecture
• Shared bus
• 1333 MT/s = 1333M Transfers / sec
• 4 transfers/clock = 266Mhz.
RAM
CPU
FSB
North
Bridge
CPU South
I/O: SATA, USB, etc
Bridge
27
27

28. Quick Path Interconnect
• 3.2Ghz, 6.4 GT/sec
• 25.6 GB/sec
CPU
RAM
QPI IOH
RAM CPU
28
28

29. Hyper Threading
Thread 1
CPU
Thread 2
Thread 1 runs until it hits a cache miss.
Thread 2 runs on the same CPU while T1 is awaiting data
29
29

30. Hardware Upgrades
2.4Ghz Xeon (Wall-E) 2.0Ghz Xeon (Johnny 5) 2.2Ghz Xeon (Eve)
2000
1500
1000
500
0 8-12X
Average Response Time (ms)
30
30

31. Improved GC Throughput
2.4Ghz Xeon (Wall-E) 2.0Ghz Xeon (Johnny 5) 2.2Ghz Xeon (Eve)
30000
22500
15000
7500
0 2-3X
GC Throughput (MB/sec)
31
31

32. Garbage Collection
• Javaʼs Virtual Machine manages memory
• New objects are continually created during runtime
• GC is the process of collecting dead objects to reclaim memory
32
32

33. Tune Garbage Collection
• Getting your GC tuning wrong is bad
• How bad?
• Are the defaults sane?
33
33

34. Tune Garbage Collection
CMS ParNew ParOld
4000
3000
2000
1000
0
Wall-E (ms) Johnny 5 (ms)
Average Response Time
Eve (ms)
2X
34
34

35. Tune Garbage Collection
CMS ParNew ParOld
500
375
250
125
0
Johnny 5 (ms)
Average Response Time
Eve (ms)
2X
35
35

36. Tune Garbage Collection
• Defaults are sane
• Parallel New + Parallel Old is fastest
• CMS prioritises low pauses over CPU usage
• CMS does not compact - can lead to heap fragmentation
36
36

37. VMWare
• Does being in a VM hurt?
• What if the risks are managed?
37
37

38. Remove VMWare
Real ESX 3.5 ESX 4i
170
128
85
43
0
Average Response Time (ms)
43%
38
38

39. Remove VMWare
• On average ~30% slower
• Under extreme resource starvation, up to 100% slower
• Avoid VMWare for high traffic instances
• See the Atlassian docs for recommendations
• http://confluence.atlassian.com/display/DOC/Running+Confluence+in+a+Virtualised+Environment
• http://confluence.atlassian.com/display/JIRA/Running+JIRA+in+a+Virtualised+Environment
39
39

40. Upgrade your browsers!
• Use a modern browser
• IE8, Safari 4 or Firefox 3.6
• Pick the browser thats right for you
• Avoid IE6 (unsupported and slow)
• http://sixrevisions.com/infographics/performance-comparison-of-major-web-browsers/
40
40

41. You are here
• How I did the testing
• Hardware & Software
• Must have improvements
• Upgrade Java, Hardware, Remove VMWare and tune Garbage Collection
• Worth doing
• Gigabit vs 100mbit, Heap size tuning, use an SSD
• Sadly, not useful
• Tune MySQL, Replace MySQL
41
41

42. Maybe.
42
42

43. Gigabit Network vs 100Mbit
• Gigabit is faster, sure
• But 100Mb is OK if youʼre not doing big transfers, RIGHT?!
43
43

44. Use Gigabit
100mb/sec 1000mb/sec
1900
1825
1750
1675
1600
Average Response Time (ms)
10%
44
44

45. Use Gigabit
• Lower RTT on Gigabit
• Unlikely to hit throughput limit
• If you have enough CPU power, keep the DB on localhost
45
45

46. Use Gigabit
100Mbit 1000Mbit
200
188
175
163
150
ReIndex Time (seconds) 26%
46
46

47. Heap size tuning
• Memory starvation is bad, but can we drown the system with too much
memory
47
47

48. Heap size - Wall-E
1.5G 3G 6G
2000
1500
1000
500
0
Average Response Time
23%
48
48

49. Heap size - Johnny 5
1.5G 3G 6G 15G
290
218
145
73
0
Average Response Time 49%
49
49

50. Heap size - Eve
1.5G 3G 6G 15G
170
128
85
43
0
Average Response Time 28%
50
50

51. Heap size tuning
• Heap too small - bad
• Heap too big - not bad!
• Bigger heaps give the JVM more room to work
• Diminishing returns from really big heaps
51
51

52. Buy an SSD
• SSDs are FAST!
• SSDs are EXPENSIVE!
• SSDs are all the rage!
• DBAs are finally happy (almost!)
52
52

53. Buy an SSD
SATA SSD
800
600
400
200
0
Browse Issue Create Issue 33%
53
53

54. Buy an SSD
• Fast writes
• 0.1ms Avg Service Time
• Most improvement comes from SSD on the DB
• Writing to the Lucene index means re-opening IndexReaders
• Much faster on SSD
• Lucene Search Term Cache fits in OS buffers
54
54

55. You are here
• How I did the testing
• Hardware & Software
• Must have improvements
• Upgrade Java, Hardware, Remove VMWare and tune Garbage Collection
• Worth doing
• Gigabit vs 100mbit, Heap size tuning, use an SSD
• Sadly, not useful
• Tune MySQL, Replace MySQL
55
55

56. Meh.
56
56

57. Replace MySQL with MySQL
• MySQL 5.0.45 - RHEL 5
• MySQL 5.0.84 Percona B18.
• Percona HighPerf patches fix contention points
• Improved I/O
• Better scaling
57
57

58. Replace MySQL with MySQL
MySQL 5.0.45 Percona MySQL 5.0.85
1687
1515
1344
1172
1000 0.4%
Average Response Time (ms)
58
58

59. Replace MySQL with MySQL
MySQL 5.0.45 Percona MySQL 5.0.84
25
19
13
6
0 0%
Average CPU(%)
59
59

60. Replace MySQL with MySQL
• JIRA doesnʼt do enough DB queries
• JIRA uses Lucene for many queries
• Percona Highperf MySQL really shines at high query rates
• Try this if you share your DB server with other apps!
60
60

61. Tune MySQL
• InnoDB-Heavy-4G.conf
sort_buffer_size = 8M
join_buffer_size = 8M
read_buffer_size = 2M
read_rnd_buffer_size = 16M
thread_concurrecy = 16
61
61

62. Tune MySQL
Defaults Tuned
1710
1533
1355
1178
1%
1000
Average Response Time (ms)
62
62

63. Tune MySQL
MyISAM ALL Engines
join_buffer_size - Large joins
without Indexes
key_cache_age_threshold,
key_cache_block_size,
sort_buffer_size - Used by
key_cache_division_limit,
ORDER BY / GROUP BY
key_buffer_size
read_buffer_size,
query_cache_size
read_rnd_buffer_size
query_cache_limit
bulk_insert_buffer_size
query_cache_min_res_unit
63
63

64. Tune MySQL
• Be sure to set innodb_buffer_pool_size
• Note! innodb_log_file_size will affect shutdown speed
64
64

65. Remember!
• Upgrade!
• Atlassian software
• Java to 1.6
• Your server
• Use Parallel Garbage Collection with a BIG heap
• Remove VMWare if you can
• Use 1000mbit network
• Buy an SSD for write workloads
65
65

66. A final word
• Donʼt resist change but TEST EVERYTHING!
66
66

67. Thanks! Questions?
• Come chat to me in the hallway!
• Charlie Lounge, 10 June, 3pm - 4pm
• http://blogs.atlassian.com/developer
Flickr Attributions:
Siesta in Denmark - http://www.flickr.com/photos/jakecaptive/27123533/
Day 11 - http://www.flickr.com/photos/ivanomak/4057632458/
Blue Streak - http://www.flickr.com/photos/jettajet/2061715292/
67
67