The document discusses strategies for addressing Metaspace OutOfMemoryErrors (OOM) when redeploying web applications. It recommends monitoring Tomcat logs and memory usage, analyzing heap dumps to find classloader leaks and duplicate classes, and making configuration changes like closing Quartz schedulers and Log4j contexts. Specific issues addressed include leaks caused by third-party JARs like log4j-web and Xerces, and weak references in WeakHashMaps. Stress testing and troubleshooting on a production system found the root cause was an outdated Xerces JAR producing classloader leaks.
3. Leon Chen
○ Master of NTHU CS
○ More than 20 years in Java/JEE, as programmer, architect,
developer leader and consultant, in finance/telecom domain.
■ WT Microelectronics
■ Oracle
■ Ericsson
■ Shin Kong Life Insurance Co., Ltd.
■ Smarteam Co., Ltd.
■ Industrial Technology Research Institute
3
7. Before Java 8: PermGen
○ Put in PermGen space of heap
○ Class metadata
○ Method of a class (include bytecodes)
○ Names of the classes
○ Constant pool Information
○ Objects array and type arras associated with a class
○ Internal objects crated by the JVM
○ Information used for optimization by the compilers (JITs)
7
8. After Java 8: Metaspace
○ Remove PermGen space
○ Use native memory
○
○ … OS
○ MaxMetaspaceSize
● -XX:MaxMetaspaceSize=1G
8
13. Monitoring: Watch Tomcat’s Log
○ Watch system out / catalina.out during undeploy/redeploy
web application
13
07-Sep-2019 21:25:15.168 WARNING [localhost-startStop-2]
org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
web application [test-web] appears to have started a thread named [Log4j2-TF-4-
Scheduled-2] but has failed to stop it. This is very likely to create a memory leak.
Stack trace of thread:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
50. Heap Dump and Analyze
50
Path To GC Roots |
Exclude all phantom/weak/soft etc. references
!
51. Heap Dump and Analyze
51
Path To GC Roots | with all references
52. Reference
○ Strong reference
Car car = new Car ();
○ Weak reference
WeakReference<Car> weakCar = new WeakReference<>(car);
○ Soft reference
SoftReference<Car> softCar = new SoftReference<>(car);
○ Phantom reference
PhantomReference<Car> phaCar = new PhantomReference<>(car, referenceQueue);
52
53. Weak Reference
○ Look dude, I am creating this object as a weak reference. Even
though I need it, feel free to garbage collect it if you run out of
memory. I know this object can be GC'd any time and am
prepared to deal with it.
- http://www.programmr.com/blogs/what-every-java-developer-should-know-strong-and-weak-
references
○ Don’t care Weak References
53
54. Soft References
○ All soft references to softly-reachable objects are guaranteed to have
been cleared before the virtual machine throws an
OutOfMemoryError.
● https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/lang/ref/SoftReference.html
○ That’s a lie. It was true when soft references were first introduced in
java 1.2, but from java 1.3.1 the jvm property
-XX:SoftRefLRUPolicyMSPerMB was introduced. It defaults to 1000
(milliseconds), meaning that if there’s only 10MB available heap, the
garbage collector will free references that have been used more than
10s ago.
- https://blog.shiftleft.io/understanding-jvm-soft-references-for-great-good-and-building-a-cache-244a4f7bb85d 54
55. Phantom Reference
○ Use Cases:
● It can be used instead of a finalize method, guaranteeing that the
object is not resurrected during finalization
● Detect exactly when an object has been removed from memory
○ If the garbage collector determines at a certain point in time that the
referent of a phantom reference is phantom reachable (neither
strongly, softly, nor weakly reachable, it has been finalized, and some
phantom reference refers to it), then at that time or at some later
time it will enqueue the reference..
https://stackoverflow.com/questions/53822132/java-phantomreference-vs-finalize
https://en.wikipedia.org/wiki/Phantom_reference 55
ReferenceQueue<Object> referenceQueue = new
ReferenceQueue<>();
PhantomReference<Object> phantomReference = new
PhantomReference<>(object, referenceQueue);
56. Heap Dump and Analyze
56
Path To GC Roots | exclude weak references
63. Weak Reference
○ Look dude, I am creating this object as a weak reference. Even
though I need it, feel free to garbage collect it if you run out of
memory. I know this object can be GC'd any time and am
prepared to deal with it.
- http://www.programmr.com/blogs/what-every-java-developer-should-know-strong-and-weak-
references
○ Don’t care Weak References ???
63
64. java.util.WeakHashMap
○ Implementation note: The value objects in a WeakHashMap
are held by ordinary strong references. Thus care should be
taken to ensure that value objects do not strongly refer to
their own keys, either directly or indirectly, since that will
prevent the keys from being discarded.
- https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/util/WeakHashMap.html
64
69. JAXB memory leak?
○ -Dcom.sun.xml.bind.v2.bytecode.ClassTailor.noOptimize=true
○ https://stackoverflow.com/questions/44830943/metaspace-
memory-leak
○ https://stackoverflow.com/questions/33255578/old-jaxb-and-
jdk8-metaspace-outofmemory-issue/33431431Constant pool
Information
○ NO …
69
70. JAXB memory leak?
○ https://java.jiderhamn.se/2012/02/26/classloader-leaks-v-
common-mistakes-and-known-offenders/
● …JAXB Reference Implementation shipped with JDK 1.6+ …
(We are using Java 8, should be no problem …)
● …it seems that if for example you have a version of Xerces
inside your application, the factory method
(… check …)
70
96. CREDITS
Special thanks to all the people who made and released these awesome
resources for free:
○ Presentation template by SlidesCarnival
○ Photographs by Unsplash
96