OSGi World Congress 2005 - Developer Forum Day 1

1. Monitoring and
Managing Computer
Resource Usage on
OSGi Frameworks
Ikuo YAMASAKIIkuo YAMASAKI
Research EngineerResearch Engineer
NTT Cyber Solution LaboratoriesNTT Cyber Solution Laboratories

2. Background:Background: Service Aggregation PlatformService Aggregation Platform
•• OperationOperation CCenterenter andand Home Service Gateway (HSGW)Home Service Gateway (HSGW) are sharedare shared
by multipleby multiple Service ProvidersService Providers..
Home Service
Gateway
ホームサービ
ス
オペレーショ
ン
センタOperation
Center
(Portal site)
Focus on service
development
OSGi FW
One stop access by users
PCs
AV equipment
White goods
sensors
ServiceProviders
Platform Providers
Low initial and running costs for individual service providers.
• Cost of operation center and HSGWs
Multiple Service Providers’
bundles will be executed.

3. BackgroundBackground : resource starvation: resource starvation
OSGi FW
PCs
AV
equipment
White
goods
sensors
•• In OSGi,In OSGi, bundlebundles creates create threadsthreads
–– AfterAfter BundleActivator.start(bcBundleActivator.start(bc) returns, threads do their jobs.) returns, threads do their jobs.
•• ThreadsThreads can deplete computer resources (CPU usage,can deplete computer resources (CPU usage,
memory usage)memory usage)
–– Malicious codeMalicious code
–– Not intentionally, wrong implementation.Not intentionally, wrong implementation.
•• Especially, resource starvationEspecially, resource starvation by differentby different ServiceService
ProviderProviderss’’ bundlesbundles is a big problem.is a big problem.

4. Our General GoalOur General Goal
•• Develop (Remote)Develop (Remote) Monitoring and Management system ofMonitoring and Management system of
HSGWHSGW
–– ProtectProtect ‘‘goodgood’’ bundles by preventing malicious or misbehavingbundles by preventing malicious or misbehaving
bundles on the same FW from depleting resources.bundles on the same FW from depleting resources.
HSGW
ホームサービ
ス
オペレーショ
ン
センタOperation
Center
OSGi FW
PCs
AV equipment
White goods
sensors

5. DesignDesign ConceptConcept
•• LightweightLightweight
–– TThehe monitoring & managing systemmonitoring & managing system addsadds minimalminimal overhead.overhead.
–– useuse in service phase,in service phase, not test phasenot test phase..
•• Backwards CompatibilityBackwards Compatibility
–– AnyAny bundle thatbundle that has beenhas been writtenwritten must be able tomust be able to workwork on theon the
systemsystem..
–– Without rewriting source codesWithout rewriting source codes..
•• Independent of specific vendorsIndependent of specific vendors
–– Use only standardized measures.Use only standardized measures.
–– No specific vendorNo specific vendor’’s OSGis OSGi--FW or JVMFW or JVM..

6. Resource Usage to MonitorResource Usage to Monitor
•• ThreadsThreads created by a bundle consumecreated by a bundle consume
–– CPU resourcesCPU resources
•• OurOur Definition:Definition: Sum of CPU usage by allSum of CPU usage by all threadsthreads
created by the bundlecreated by the bundle..
–– Memory resourcesMemory resources
•• Definition ofDefinition of ““how much memory a thread and ahow much memory a thread and a
bundle is usingbundle is using”” is controversial.is controversial.
•• There is no way to monitor it, even in J2SE5.0.There is no way to monitor it, even in J2SE5.0.
Target of this work
Out of scope

7. To achieve the goalTo achieve the goal
•• Management EntityManagement Entity needs toneeds to
1.1. Monitor CPU usage consumed by each bundle.Monitor CPU usage consumed by each bundle.
a.a. Monitor CPU usage of each thread.Monitor CPU usage of each thread.
b.b. Manage the relations between bundles and threads, andManage the relations between bundles and threads, and
Calculate each bundleCalculate each bundle’’s CPU usage.s CPU usage.
2.2. According to theAccording to the management policymanagement policy adopted, someadopted, some
reaction needed.reaction needed.
•• If any bundle consumes too many resources,If any bundle consumes too many resources,
•• Take action against excessive bundle, to protect otherTake action against excessive bundle, to protect other
bundlesbundles’’ activitiesactivities..

8. Management Policies (Mgt. Policies)Management Policies (Mgt. Policies)
•• Adaptability to MultipleAdaptability to Multiple mgt. policiesmgt. policies is required.is required.
–– Triggers :Triggers : When and Which bundle should be suppressedWhen and Which bundle should be suppressed
because of its excessive consumption?because of its excessive consumption?
•• e.g.e.g.
–– If system CPU usage exceeds 97% for the latest 10 min, the bundlIf system CPU usage exceeds 97% for the latest 10 min, the bundlee
consuming the most should be suppressed.consuming the most should be suppressed.
–– A bundle whose CPU usage exceeds 90% for the last 10 min shouldA bundle whose CPU usage exceeds 90% for the last 10 min should bebe
suppressed.suppressed.
–– Reactions :Reactions : How to suppress the bundle?How to suppress the bundle?
a.a. Change the threadChange the thread’’s prioritys priority
b.b. Kill the threadKill the thread
c.c. Stop the bundle ( and kill all the threads created)Stop the bundle ( and kill all the threads created)
–– Activators :Activators : Which entityWhich entity triggerstriggers thethe reactionreaction??
1.1. Entity working on the same host.Entity working on the same host.
2.2. Remote monitoring and management entity.Remote monitoring and management entity.

9. Local Host
Other
Programs
or Bundles
on local
Our System ArchitectureOur System Architecture
1.1. Local entityLocal entity
–– Automatic reactionAutomatic reaction
•• Owns mgt. policy.Owns mgt. policy.
•• Even if the networkEven if the network
connection is dead, it works.connection is dead, it works.
OSGi FW
ThreadThread
ManagerManager
records
Remote Manager
2.2.Remote entityRemote entity
–– Automatic reactionAutomatic reaction
•• Owns mgt. policy.Owns mgt. policy.
–– Platform ProviderPlatform Provider’’ss
operatoroperator driven reactiondriven reaction
Bundles
Monitor & Manage
locally
JVM
Platform Provider provides
(as one of the Managed Agents)

10. How to monitorHow to monitor threadthread CPU usage ?CPU usage ?
•• Design ConceptDesign Concept
–– LightweightLightweight
–– Backward compatibleBackward compatible
–– Independent of specific vendorsIndependent of specific vendors
•• JMXJMX ofof J2SE5.0J2SE5.0 (used to be called(used to be called ““TigerTiger””) can) can
monitor every thread.monitor every thread.
–– Demerit: J2SE5.0 requires high spec machinesDemerit: J2SE5.0 requires high spec machines……
•• Difficult to deploy in embedded systems.Difficult to deploy in embedded systems.

11. 1.1. ProvidesProvides standardized measuresstandardized measures toto monitormonitor andand
managemanage Java applications.Java applications.
–– Developer can define ownDeveloper can define own MBeanMBean (Managed Bean(Managed Bean））
•• Getter / setter styleGetter / setter style
–– JMX Agent (JMX Agent (MBeanMBean ServerServer) enables management of target) enables management of target
object through registeredobject through registered MBeanMBean
JMX (Java ManagementJMX (Java Management eXtentioneXtention))
JVM (S2SE5.0)
Application
MBean Server （JMX Agent）
MBean
Managed object
MBean
Instrumentation
Level
Agent Level
Remote
Management Level

12. 2.2. Resources can be monitored and managed, eitherResources can be monitored and managed, either
remotelyremotely oror locallylocally..
–– Protocol is replaceable.Protocol is replaceable.
•• RMI connectorRMI connector uses Java RMI.uses Java RMI.
JMX (Java ManagementJMX (Java Management eXtentioneXtention))
JVM (S2SE5.0)
MBean Server
MBean MBean
Instrumentation
Level
Agent Level
Remote
Management Level
Protocol Adapter Protocol Adapter Connector
Web Browsers
HTTP
SNMP Mgt. App.
SNMP
JMX-compliant
Mgt, App.
JMX Remote Protocol

13. 3.3. PlatformPlatform MBeanMBean Server (Server (MXBeanMXBean Server)Server) is built inis built in JVMJVM
–– Several types ofSeveral types of MXBeanMXBeanss are defined in the spec.are defined in the spec.
•• Every thread can be monitored and managed throughEvery thread can be monitored and managed through
ThreadThreadMXBeanMXBean
–– includingincluding CPU usageCPU usage
–– MXBeansMXBeans are registered automatically.are registered automatically.
JMX (Java ManagementJMX (Java Management eXtentioneXtention))
Application
MXBean Server
ThreadMXBean
Connector
JMX-compliant
Mgt, App.
JMX Remote Protocol
Thread
JMX is lightweight

14. How to monitorHow to monitor bundlebundle CPU usage ?CPU usage ?
•• We defineWe define BundleBundle--ThreadThread--Tree (BTT)Tree (BTT)
–– in order forin order for Thread ManagerThread Manager to maintainto maintain thethe relationsrelations betweenbetween
every threadevery thread andand its creating bundleits creating bundle
•• At the creation of a thread,At the creation of a thread,
•• pair of thepair of the threadthread andand its creating bundleits creating bundle must be reported to themust be reported to the
Thread ManagerThread Manager..
•• If rewriting codes is acceptable, no problem.If rewriting codes is acceptable, no problem.
•• But ourBut our Design Concept: Backward compatibilityDesign Concept: Backward compatibility
–– No codes rewriting permitted.No codes rewriting permitted.
FW’s bundle
starting Thread
Thread D
Thread A
Thread C
Bundle 1
Thread E
Thread B
Bundle 2

15. Byte Code Weaving: AOP (Byte Code Weaving: AOP (AspectJAspectJ))
•• We adoptWe adopt Byte Code WeavingByte Code Weaving..
–– AOP (Aspect Oriented Programming)AOP (Aspect Oriented Programming):: AspectJAspectJ
•• ““WeaveWeave”” : AOP allows us to insert: AOP allows us to insert adviceadvice
before/after/aroundbefore/after/around pointcutpointcut in the byte code.in the byte code.
–– AdviceAdvice : byte code to be woven: byte code to be woven
–– PointcutPointcut : where to weave the byte code.: where to weave the byte code.
–– E.g.E.g. Insert the specified codeInsert the specified code at the beginning / end of theat the beginning / end of the
specified method of the specified class.specified method of the specified class.
•• Byte codeByte code can be altered evencan be altered even without source codewithout source code of theof the
program.program.

16. Offline procedures
Offline Bundle WeavingOffline Bundle Weaving
JAR of
Bundle A Weaving
(Aspect file:
ThreadAspect.aj )
1.1. Service ProviderService Provider ((SPSP) passes its bundle JAR to) passes its bundle JAR to
Platform ProviderPlatform Provider (without source code).(without source code).
2.2. Platform ProviderPlatform Provider weaves the aspects into the bundleweaves the aspects into the bundle
and posts it on the web server.and posts it on the web server.
3.3. OSGiOSGi--FWsFWs of HSGW will install the woven bundle fromof HSGW will install the woven bundle from
the web server.the web server.
Will be Installed
on Users’ FW
at BundleContext
.installBundle(uri)
ThreadAspect.class will be created.
Woven Jar of Bundle A
SP passes JAR

17. Aspect to Weave into bundles 1Aspect to Weave into bundles 1
• Pointcut 1: Before a bundle is started
• Advice Inserted: set its bundle object into the field
variable “bundle” in ThreadAspect class.
public class Activator implements BundleActivator {
public void start(BundleContext bc) throws Ex {
Bundle bundle = bc.getBundle();
ThreadAspect.getInstance().setBundle(bundle);
System.out.println(“OSGi: Bundle started”);
...
}
...
}
Pseudo woven code

18. Aspect to Weave into bundles 2Aspect to Weave into bundles 2
• Pointcut 2: Before a thread is started
• Advice Inserted:
– A) retrieve its bundle object from the field variable “bundle” in
ThreadAspect class, and
– B) notify the relation between the thread and bundle creating it
to ThreadManager.
public class ThreadCreate {
public void doTask() {
Thread t = new Thread(new Runnable() {
public void run() {...;}
});
Bundle bundle = ThreadAspect.getInstance().getBundle();
ThreadManager tm = ThreadManager.getInstance();
tm.addThread(t, bundle);
t.start();
}
} Pseudo woven code

19. OSGi FW
MXBean Server
Operation of the SystemOperation of the System
Thread Manager
Bundle
Bundle
1. Install woven bundle
2. BundleEvent is delivered to
Synchronous Bundle Listener impl.
3. construct Bundle-Thread-Tree
(BTT) BTT

20. OSGi FW
MXBean Server
Operation of the SystemOperation of the System
Thread Manager
Bundle
4. Bundle is started
ThreadAspect
bundle
5. Store Bundle object

21. OSGi FW
MXBean Server
Operation of the SystemOperation of the System
Thread Manager
Bundle
6. When a thread is created
ThreadAspect
bundle
7. Retrieve Bundle
Thread
8. Send them to
Thread Manager
BTT
9. Update Bundle-Thread-Tree
Thread A

22. OSGi FW
MXBean Server
Operation of the SystemOperation of the System
Thread Manager
Bundle
Threads
Thread Manager monitors
every thread’s CPU usage
periodically, and records it
locally.
Thread A
Thread B
BTT
ThreadMXBean
A
Sample
Thread
SleepB
records

23. OSGi FW
MXBean Server
ThreadManagerMBeanThreadManagerMBean
Thread Manager
Threads
Thread Manager registers its own
MBean “ThreadManagerMBean”,
which provides IF of
1.Monitoring recorded data.
2.Suppressing actions taken
against bundles and threads.
Thread A
Thread B
Bundle
ThreadMXBean
A B
records
ThreadManager-
MBean
Connector
JMX-compliant
Mgt, App.
JMX Remote Protocol

24. Local Host
Other
programs
or bundles
on local
No restrictions on Mgt. PoliciesNo restrictions on Mgt. Policies
OSGi FW
MXBean Server
Thread Manager
Thread A
Thread B
Bundle
ThreadMXBean
A B
records
JMX-compliant
Mgt, App.
ThreadManager-
MBean
Connector
•• DiverseDiverse mgt. policiesmgt. policies can be supportedcan be supported
–– TriggersTriggers
–– ReactionsReactions
–– ActivatorsActivators
Either is possible.

25. SummarySummary
•• Background:Background:
–– Threads created by bundle can deplete computer resources.Threads created by bundle can deplete computer resources.
–– Monitoring and management system is required, especially inMonitoring and management system is required, especially in
Service Aggregation Platform.Service Aggregation Platform.
•• Proposal:Proposal:
–– CPU usage management system on OSGi FW adoptingCPU usage management system on OSGi FW adopting
•• JMX in J2SE5.0JMX in J2SE5.0
–– LightweightLightweight
–– Independent of specific vendorsIndependent of specific vendors
•• AOP (AOP (AspectJAspectJ))
–– No rewrites of existing code neededNo rewrites of existing code needed
–– Proposed architecture can support diverse management policiesProposed architecture can support diverse management policies
•• Trigger decisionTrigger decision
•• Reactions to suppress the bundle and threads.Reactions to suppress the bundle and threads.
•• Activators (remotely and locally)Activators (remotely and locally)