Presented by Gregg Donovan, Senior Software Engineer, Etsy.com, Inc. Understanding the impact of garbage collection, both at a single node and a cluster level, is key to developing high-performance, high-availability Solr and Lucene applications. After a brief overview of garbage collection theory, we will review the design and use of the various collectors in the JVM. At a single-node level, we will explore GC monitoring -- how to understand GC logs, how to monitor what % of your Solr request time is spend on GC, how to use VisualGC, YourKit, and other tools, and what to log and monitor. We will review GC tuning and how to measure success. At a cluster-level, we will review how to design for partial availability -- how to avoid sending requests to a GCing node and how to be resilient to mid-request GC pauses.For application development, we will review common memory leak scenarios in custom Solr and Lucene application code and how to detect them.

1. Senior Software Engineer,
LIVING WITH GARBAGE
Gregg Donovan

2. 3.5Years Solr & Lucene at Etsy.com
3 years Solr & Lucene atTheLadders.com

4. 8+ million members

5. 20 million items

6. 800k+ active sellers

7. 8+ billion pageviews per month

15. CodeAsCraft.etsy.com

16. Understanding GC
Monitoring GC
Debugging Memory Leaks
Design for Partial Availability

18. public class BuzzwordDetector {
static String[] prefixes = { "synergy", "win-win" };
static String[] myArgs = { "clown synergy", "gorilla win-wins", "whamee" };
public static void main(String[] args) {
args = myArgs;
int buzzwords = 0;
for (int i = 0; i < args.length; i++) {
String lc = args[i].toLowerCase();
for (int j = 0; j < prefixes.length; j++) {
if (lc.contains(prefixes[j])) {
buzzwords++;
}
}
}
System.out.println("Found " + buzzwords + " buzzwords");
}
}

19. New():
ref <- allocate()
if ref = null /* Heap is full */
collect()
ref <- allocate()
if ref = null /* Heap is still full */
error "Out of memory"
return ref
atomic collect():
markFromRoots()
sweep(HeapStart, HeapEnd)
From Garbage Collection Handbook

20. markFromRoots():
initialise(worklist)
for each fld in Roots
ref <- *fld
if ref != null && not isMarked(ref)
setMarked(ref)
add(worklist, ref)
mark()
initialise(worklist):
worklist <- empty
mark():
while not isEmpty(worklist)
ref <- remove(worklist) /* ref is marked */
for each fld in Pointers(ref)
child <- *fld
if (child != null && not isMarked(child)
setMarked(child)
add(worklist, child)
From Garbage Collection Handbook

21. Trivia:Who invented the first
GC and Mark-and-Sweep?

22. Weak Generational
Hypothesis

23. Where do objects in common Solr
application live?
AtomicReaderContext?
SolrIndexSearcher?
SolrRequest?

24. GC Terminology:
Concurrent vs Parallel

25. JVM Collectors

26. Serial

27. Trivia: How does System.identityHashCode
() work?

28. Throughput

29. CMS

30. Garbage First (G1)

31. Continuously Concurrent Compacting Collector (C4)

32. IBM, Dalvik, etc.?

33. Why Throughput?

34. Questions so far?

35. Monitoring

36. GC time per Solr request

37. ...
import java.lang.management.*;
...
public static long getCollectionTime() {
long collectionTime = 0;
for (GarbageCollectorMXBean mbean : ManagementFactory.getGarbageCollectorMXBeans()) {
collectionTime += mbean.getCollectionTime();
}
return collectionTime;
}
Available via JMX

38. Visual GC

40. export GC_DEBUG="-verbose:gc
-XX:+PrintGCDateStamps
-XX:+PrintHeapAtGC
-XX:+PrintGCApplicationStoppedTime
-XX:+PrintGCApplicationConcurrentTime
-XX:+PrintAdaptiveSizePolicy
-XX:AdaptiveSizePolicyOutputInterval=1
-XX:+PrintTenuringDistribution
-XX:+PrintGCDetails
-XX:+PrintCommandLineFlags
-XX:+PrintSafepointStatistics
-Xloggc:/var/log/search/gc.log"

41. 2013-04-08T20:14:00.162+0000: 4197.791: [Full GCAdaptiveSizeStart: 4206.559 collection: 213
PSAdaptiveSizePolicy::compute_generation_free_space limits: desired_promo_size: 9927789154
promo_limit: 8321564672 free_in_old_gen: 4096 max_old_gen_size: 22190686208 avg_old_live:
22190682112
AdaptiveSizePolicy::compute_generation_free_space limits: desired_eden_size: 9712028790
old_eden_size: 8321564672 eden_limit: 8321564672 cur_eden: 8321564672 max_eden_size: 8321564672
avg_young_live: 7340911616
AdaptiveSizePolicy::compute_generation_free_space: gc time limit gc_cost: 1.000000 GCTimeLimit: 98
PSAdaptiveSizePolicy::compute_generation_free_space: costs minor_time: 0.167092 major_cost:
0.965075 mutator_cost: 0.000000 throughput_goal: 0.990000 live_space: 29859940352 free_space:
16643129344 old_promo_size: 8321564672 old_eden_size: 8321564672 desired_promo_size: 8321564672
desired_eden_size: 8321564672
AdaptiveSizeStop: collection: 213
[PSYoungGen: 8126528K->7599356K(9480896K)] [ParOldGen: 21670588K->21670588K(21670592K)] 29797116K-
>29269944K(31151488K) [PSPermGen: 58516K->58512K(65536K)], 8.7690670 secs] [Times: user=137.36
sys=0.03, real=8.77 secs]
Heap after GC invocations=213 (full 210):
PSYoungGen total 9480896K, used 7599356K [0x00007fee47ab0000, 0x00007ff0dd000000,
0x00007ff0dd000000)
eden space 8126528K, 93% used [0x00007fee47ab0000,0x00007ff0177ef080,0x00007ff037ac0000)
from space 1354368K, 0% used [0x00007ff037ac0000,0x00007ff037ac0000,0x00007ff08a560000)
to space 1354368K, 0% used [0x00007ff08a560000,0x00007ff08a560000,0x00007ff0dd000000)
ParOldGen total 21670592K, used 21670588K [0x00007fe91d000000, 0x00007fee47ab0000,
0x00007fee47ab0000)
object space 21670592K, 99% used [0x00007fe91d000000,0x00007fee47aaf0e0,0x00007fee47ab0000)
PSPermGen total 65536K, used 58512K [0x00007fe915000000, 0x00007fe919000000,
0x00007fe91d000000)
object space 65536K, 89% used [0x00007fe915000000,0x00007fe918924130,0x00007fe919000000)
}

42. GC Log Analyzers?
GCHisto
GCViewer
garbagecat

43. Graphing with Logster
github.com/etsy/logster

45. GC Dashboard
github.com/etsy/dashboard

47. YourKit.com

48. Designing for Partial Availability

49. JVMTI GC Hook?

50. How can a client ignore GC-ing hosts?

51. Server lies to clients about availability
TCP socket receive buffer
TCP write buffer

52. “Banner” protocol
1. Connect via TCP
2.Wait ~1-10ms
3. Either receive magic four byte header or try another host
4. Only send query after receiving header from server

53. 0xC0DEA5CF

54. What if GC happens
mid-request?

55. Backup requests

56. Jeff Dean: Achieving Rapid
Response Time in Large
Online Services

57. Solr sharding?
Right now, only as fast as the slowest shard.

58. “Make a reliable whole
out of unreliable parts.”

59. Memory Leaks

60. Solr API hooks for
custom code
QParserPlugin SearchComponent
SolrRequestHandler SolrEventListener
SolrCache ValueSourceParser
etc.FieldType

61. PSA: Are you sure you
need custom code?

62. CoreContainer#getCore()
RefCounted<SolrIndexSearcher>

63. SolrIndexSearcher generation marking with
YourKit triggers

65. Questions so far?

66. Miscellaneous Topics

67. System.gc()?

68. -XX:+UseCompressedOops

69. -XX:+UseNUMA

70. Paging

71. #!/usr/bin/env bash
# This script is designed to be run every minute by cron.
host=$(hostname -s)
psout=$(ps h -p `cat /var/run/etsy-search.pid` -o min_flt,maj_flt 2>/dev/null)
min_flt=$(echo $psout | awk '{print $1}') # minor page faults
maj_flt=$(echo $psout | awk '{print $2}') # major page faults
epoch_s=$(date +%s)
echo -e "search_memstats.$host.etsy-search.min_fltt${min_flt:-0}t$epoch_s" | nc
graphite.etsycorp.com 2003
echo -e "search_memstats.$host.etsy-search.maj_fltt${maj_flt:-0}t$epoch_s" | nc
graphite.etsycorp.com 2003

72. Solution 1: Buy more RAM
Ideally enough RAM to:
Keep index in OS file buffers
AND ensure no paging ofVM memory
AND whatever else happens on the box
~$5-10/GB

73. echo “0” > /proc/sys/vm/swappiness

74. mlock()/mlockall()

75. echo “-17” > /proc/$PID/oom_adj
Mercy from the OOM Killer

76. Huge Pages

77. -XX:+AlwaysPreTouch

78. Possible Future Directions

79. Many small VMs instead of one large VM
microsharding

80. In-memory Lucene codecs
I.e. custom DirectPostingsFormat

81. Off-heap memory with sun.misc.Unsafe?

82. Try G1 again

83. Try C4 again

84. Resources

85. gchandbook.org

87. bit.ly/mmgcb
Mark Miller’s GC Bootcamp

88. bit.ly/giltene
GilTene: Understanding Java
Garbage Collection

89. bit.ly/cpumemory
Ulrich Drepper: What Every Programmer Should
Know About Memory

90. github.com/pingtimeout/jvm-options

91. Read the JVM Source
(Not as scary as it sounds.)
hg.openjdk.java.net/jdk7/jdk7

92. Mechanical Sympathy Google Group
bit.ly/mechsym

93. Questions?
Thanks for coming!

94. CONTACT
Gregg Donovan
gregg@etsy.com