In this session, the following topics have been discussed: code snippets that can generate memory leak, thread leak, CPU spike, unresponsiveness, BLOCKED threads, Deadlocks, Heavy I/O activity. If you can understand what triggers these problems, diagnosing and solving them might become easier.
3. How to simulate?
public class CPUSpikerThread extends Thread {
@Override
public void run() {
while (true) {
}
}
}
Open Source: https://github.com/ycrash/buggyapp
6. 2019-12-26 17:13:23
Full thread dump Java HotSpot(TM) 64-Bit Server VM (23.7-b01 mixed mode):
"Reconnection-1" prio=10 tid=0x00007f0442e10800 nid=0x112a waiting on condition [0x00007f042f719000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x007b3953a98> (a java.util.concurrent.locks.AbstractQueuedSynchr)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.lang.Thread.run(Thread.java:722)
:
:
Thread dump
"InvoiceThread-A996" prio=10 tid=0x00002b7cfc6fb000 nid=0x4479 runnable [0x00002b7d17ab8000]
java.lang.Thread.State: RUNNABLE
at com.buggycompany.rt.util.ItinerarySegmentProcessor.setConnectingFlight(ItinerarySegmentProcessor.java:380)
at com.buggycompany.rt.util.ItinerarySegmentProcessor.processTripType0(ItinerarySegmentProcessor.java:366)
at com.buggycompany.rt.util.ItinerarySegmentProcessor.processItineraryByTripType(ItinerarySegmentProcessor.java:254)
at com.buggycompany.rt.util.ItinerarySegmentProcessor.templateMethod(ItinerarySegmentProcessor.java:399)
at com.buggycompany.qc.gds.InvoiceGeneratedFacade.readTicketImage(InvoiceGeneratedFacade.java:252)
at com.buggycompany.qc.gds.InvoiceGeneratedFacade.doOrchestrate(InvoiceGeneratedFacade.java:151)
at com.buggycompany.framework.gdstask.BaseGDSFacade.orchestrate(BaseGDSFacade.java:32)
at com.buggycompany.framework.gdstask.BaseGDSFacade.doWork(BaseGDSFacade.java:22)
at com.buggycompany.framework.concurrent.BuggycompanyCallable.call(buggycompanyCallable.java:80)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
14. Will Java memory go beyond –Xmx?
main( )
a( )
b( )
c( )
x=1
y*
z=2.11
public class SimpleExample {
public static void main(String args[]) {
a();
}
public static void a() {
int x = 1;
b();
}
public static void b() {
Car y = new Car();
c();
}
public static void c() {
float z = 2.11f;
}
} Thread’s stack
Young old
metaspa
ce
car
-Xmx
-Xss
-XX:MaxMetaspaceSize
Others
Thread stacks
Garbage Collection
Code Generation
Socket Buffers
JNI
15. How to simulate?
public class StackOverflowDemo {
public void start() {
start();
}
}
Open Source: https://github.com/ycrash/buggyapp
19. OutOfMemorError: Unable to create new native thread
https://blog.ycrash.io/2021/04/09/chaos-engineering-thread-leak/
20. How to simulate?
public class ThreadLeakDemo {
public static void start() {
while (true) {
new ForeverThread().start();
}
}
}
public class ForeverThread extends Thread {
@Override
public void run() {
while (true) { // Put the thread to sleep forever, so they don't die.
try {
Thread.sleep(10 * 60 * 1000); // Sleeping for 10 minutes repeatedly
} catch (Exception e) {}
}
}
}
Open Source: https://github.com/ycrash/buggyapp
21. Troubleshooting Unable to create new native thread
http://fastthread.io/my-thread-
report.jsp?p=c2hhcmVkLzIwMTcvMDMvMTQvLS10cmFuc2l0aXZlLWJsb2Nrcy0xLnppcC0tMTEtNTUtMzA=&s=t
22. Java Heap + metaspace
Physical memory
Java Heap + metaspace
Physical memory
Process-
1
Process-
2
Key: Threads are created outside heap,
metspace
threads
Solution:
1. Fix thread leak
2. Increase the Thread Limits Set at
Operating System(ulimit –u)
3. Reduce Java Heap Size
4. Kills other processes
5. Increase physical memory size
6. Reduce thread stack size (-Xss).
Note: can cause StackOverflowError
OOM: Unable to create new native thread
