1. Axis2 client memory leak

2. Outline
• Issue & diagnosis
• Postmortem from technique point
– Axis2 1.4/1.4.1/1.5.6 client
– Implement & design issue

3. Issue
CServer exhaust 8G memory -> memory leak

4. diagnosis
jmap –histo pid
num #instances #bytes class name
----------------------------------------------
1: 1001744 246906600 [C //char array
2: 2855952 137085696 edu.emory.mathcs.backport.java.util.concurrent.ConcurrentHashMap$Segment
3: 282338 115692192 [I // int array
4: 677362 102508648 [Ljava.util.HashMap$Entry;
5: 192117 99164392 [B // byte array
6: 2856122 91732120 [Ledu.emory.mathcs.backport.java.util.concurrent.ConcurrentHashMap$HashEntry;
7: 2855952 91390464 edu.emory.mathcs.backport.java.util.concurrent.locks.ReentrantLock$NonfairSync
8: 1337498 53499920 java.lang.String
9: 433634 43130120 [Ljava.lang.Object;
10: 597908 38266112 java.util.HashMap
From the histogram, the memory leak is caused by Axis2.
1) The main change in this release is using web service client (axis2 1.4.1)
2) char[] / edu.emory.mathcs.backport.java.util.concurrent.ConcurrentHashMap
3) Google ‘axis2 1.4.1 concurrentHashMap memory leak‘

5. diagnosis
jmap –dump:file=cdump pid
• We got a 2.39G memory heap dump
– Very fast, less than 30 seconds (because 8G
memory in production)
• Problem : too big to open in dev box
– Need enough physical memory
– (May) need 64 bit OS
– Try Jhat/Jprofile/YJP/Jmat eclipse plugin
(windows/2G)
Open by Jmat standalone application

6. diagnosis
AxisConfiguration retain 540.9m heap

7. diagnosis
Hashtable allEndPoints holds more than 2k axis2 endpoint instances

8. diagnosis
All of the endpoint instances are about FavoriteService
1) Confirm in the code, only FavoriteService related did not call cleanup

9. Axis2 1.4 memory leak
• Reproduce
for( int i=0;i<count; i++) {
VersionStub stub = new VersionStub(configContext,null);
GetVersion request = new GetVersion();
stub.getVersion(request);
}

10. Axis2 1.4 memory leak
• Informal Model (OO design)
AxisConfiguration
Map allServices = new Hashtable();
Map allEndpoints = new Hashtable();

11. Axis2 1.4 memory leak
• Cause : Stub Initiation, add into AxisConfiguration
//Init, Add
allServices.put(serviceName, axisService);
..
allEndpoints.put(serviceName + "." + endpointName, axisService);

12. Axis2 1.4 memory leak
• Cause : Stub Finalize
//Stub
protected void finalize() throws Throwable {
super.finalize();
cleanup();
}
public void cleanup() throws AxisFault {
_service.getAxisConfiguration().removeService(_service.getName());
}
//AsixConiguration
public synchronized void removeService(String name) throws AxisFault {
AxisService service = (AxisService) allServices.remove(name);
if (service != null) {
AxisServiceGroup serviceGroup = service.getAxisServiceGroup();
serviceGroup.removeService(name);
log.debug(Messages.getMessage("serviceremoved", name));
}
}

13. Axis2 1.4 memory leak
• Cause : Client Finalize
//ServiceClient
protected void finalize() throws Throwable {
super.finalize();
cleanup();
}
public void cleanup() throws AxisFault {
…
axisConfiguration.removeServiceGroup(serviceGroupName);
…
}
//AsixConiguration
public AxisServiceGroup removeServiceGroup(String serviceGroupName) throws AxisFault {
…
Iterator services = axisServiceGroup.getServices();
while (services.hasNext()) {
AxisService axisService = (AxisService) services.next();
allServices.remove(axisService.getName());
…
}
…

14. Axis2 1.4 memory leak
• Cause
//Init, Add
allServices.put(serviceName, axisService);
..
allEndpoints.put(serviceName + "." + endpointName, axisService);
//Cleanup(Finalize), Remove
allServices.put(serviceName, axisService);
• Memory Leak : allEndpoints

15. Axis2 1.4.1 fix
• Fix the bug AXIS2-3870
//AsixConiguration.removeServiceGroup
//removes the endpoints to this service
String serviceName = axisService.getName();
String key = null;
for (Iterator iter = axisService.getEndpoints().keySet().iterator(); iter.hasNext();){
key = serviceName + "." + (String)iter.next();
this.allEndpoints.remove(key);
}

16. Axis2 1.4.1 memory leak
• Reproduce
for( int i=0;i<count; i++) {
VersionStub stub = new VersionStub(configContext,null);
GetVersion request = new GetVersion();
stub.getVersion(request);
stub.cleanup();
}
Programmer: 1.4 has memory leak issue, so call cleanup

17. Axis2 1.4.1 memory leak
• Cause : Stub Finalize (no change)
//Stub
protected void finalize() throws Throwable {
super.finalize();
cleanup();
}
public void cleanup() throws AxisFault {
_service.getAxisConfiguration().removeService(_service.getName());
}
//AsixConiguration
public synchronized void removeService(String name) throws AxisFault {
AxisService service = (AxisService) allServices.remove(name);
if (service != null) {
AxisServiceGroup serviceGroup = service.getAxisServiceGroup();
serviceGroup.removeService(name);
log.debug(Messages.getMessage("serviceremoved", name));
}
}

18. Axis2 1.4.1 memory leak
• Cause: Client Finalize (no change)
//ServiceClient
protected void finalize() throws Throwable {
super.finalize();
cleanup();
}
public void cleanup() throws AxisFault {
…
axisConfiguration.removeServiceGroup(serviceGroupName);
…
}

19. Axis2 1.4.1 memory leak
• Cause: Client Finalize (no change)
//AsixConiguration
public AxisServiceGroup removeServiceGroup(String serviceGroupName) throws AxisFault {
…
Iterator services = axisServiceGroup.getServices();
while (services.hasNext()) {
AxisService axisService = (AxisService) services.next();
allServices.remove(axisService.getName());
…
for (Iterator iter = axisService.getEndpoints().keySet().iterator(); iter.
key = serviceName + "." + (String)iter.next();
this.allEndpoints.remove(key);
}
…
}
Servce are already removed from ServiceGroup in Stub cleanup
…
}
The while loop would never enter

20. Axis2 1.4.1 memory leak
• Cause
– Stub cleanup and ServiceClient cleanup have
dependency
– Can not call in below order
Stub.cleanup
ServiceClient.cleanup
• Two are two memory leak bugs in 1.4, 1.4.1 only
fix 1 bug

21. Axis2 1.5(.6) fix
• Fix the bug AXIS2-4007 AXIS2-4163
//Stub
protected void finalize() throws Throwable {
super.finalize();
cleanup();
}
public void cleanup() throws AxisFault {
// _service.getAxisConfiguration().removeService(_service.getName());
_serviceClient.cleanup();
}

22. Implement issue
• Forget to cleanup
– Container object: add only, no remove
– Resource object: apply only, no return/close
• Cleanup dependency
– One object cleanup depend on other objects
– Two object cleanup/two method has order
dependency

23. Design Issue
• AxisConfiguration is a global shared object
– Usually only 1 instance even in client side.
• Purpose for put service/endpoint map in this
global object AxisConfiguration ?

24. Design Issue
• Message Dispatch in server side
// RequestURIBasedServiceDispatcher
public AxisService findService(MessageContext messageContext) throws AxisFault {
AxisConfiguration registry = configurationContext.getAxisConfiguration();
AxisService axisService = registry.getService(values[0]);
// If the axisService is not null we get the binding that the request came to add
// add it as a property to the messageContext
if (axisService != null) {
Map endpoints = axisService.getEndpoints();
if (endpoints != null) {
if (endpoints.size() == 1) {
messageContext.setProperty(WSDL2Constants.ENDPOINT_LOCAL_NAME,
endpoints.get(axisService.getEndpointName()));
} else {
String endpointName = values[0].substring(values[0].indexOf(".") + 1);
messageContext.setProperty(WSDL2Constants.ENDPOINT_LOCAL_NAME,
endpoints.get(endpointName));
}
}
}
return axisService;
}

25. Design Issue
• Service is ‘singleton’ in server side
// AxisConfiguration
public AxisService getService(String name) throws AxisFault {
AxisService axisService = (AxisService) allServices.get(name);
if (axisService != null) {
if (axisService.isActive()) {
return axisService;
} else {
throw new AxisFault(Messages
.getMessage("serviceinactive", name));
}
} else {
axisService = (AxisService) allEndpoints.get(name);
if (axisService != null) {
if (axisService.isActive()) {
return axisService;
} else {
throw new AxisFault(Messages
.getMessage("serviceinactive", name));
}
}
}
return null;
}

26. Design Issue
• Client side
– (Usually) New instance every method invocation
– No need for message routing
• Why still register in Axi2Configuration ?
axisConfig = configContext.getAxisConfiguration();
if (axisService == null) {
axisService = createAnonymousService();
}
this.axisService = axisService;
if (axisConfig.getService(axisService.getName()) == null) {
axisService.setClientSide(true);
axisConfig.addService(axisService);
} else {
…
}
…