Learn how to leverage the Solaris 11 DTrace tool in order to analyze and resolve performance problems

1. DTrace, for Solaris (zones
inside) and Java
 Amit Hurvitz, ISV Engineering, Oracle

2. Program Agenda
 Introductory Demo
 What's Dtrace?
 Enabling Dtrace from Inside the Zone
 Java Statically Defined Tracing
 TCP Client and Server Flow Tracing - Demo
 Future Thoughts

3. Int roduc t ory D o
em

4. What's DTrace? - cont.
 Zero performance impact when not in use
 Completely safe; no way to cause panics, crashes, data corruption or
pathological performance degradation
 Powerful data management primitives eliminate need for most postprocessing

5. DTrace – D Language
probe description
/predicate/
{
actions
}
Probes Which events we are interested in monitoring
 Predicates (optional) When do we want to monitor the events
 Actions (optional) What do we want to do when the above
happens

 One liner

# dtrace -n 'probe/predicate/{actions}'
{actions}

6. DTrace – Probes
 Programmable sensors (points of instrumentation) made available by
providers placed all over the Solaris system

provider:module:function:name
−
tcp:ip:tcp_send:entry
−
Syscall:::
 Providers: syscall,io,pid,profile, hotspot, tcp, udp, ip, iscsi,...
 Modules: nfs, zfs, cpc, …
 Names: entry,return
 Listing Probes

# dtrace -l [-P provider | -m module | -f function name | -n name]

7. DTrace – Predicates and Actions
 Predicates

/cpu == 0/

/execname == “date”/

/ppid != 0 && arg0 != 0/
 Actions

Commands separated by “;”

trace(execname)

printf(“%s %s %s”, execname, probefunc, copyinstr(arg0));
 Predefined Variables

execname, probefunc, pid, ppid, cpu, timestamp,
arg0, arg1, ...

8. DTrace – Aggregations
 Used to aggregate data and look for trends
 Has the general form:

@name[keys] = aggfunc(args)
 Aggregating functions

count(), sum(), avg(), min(), max(), quantize(), lquantize()
 Examples:

% dtrace -n 'syscall::read*:entry{@[execname]=count();}'

% dtrace -n 'syscall::read*:entry{@[execname,arg0]=count();}'

9. An Example – Off-CPU Tracing
#!/usr/sbin/dtrace -s
BEGIN
{
start_timestamp = timestamp;
}
sched:::off-cpu
/pid == $1/
{
self->ts = timestamp;
}
sched:::on-cpu
/self->ts/
{
@[stack(), jstack(), "ns"] = sum(timestamp - self->ts); self->ts = 0;
}
END
{
printf("elapsed time: %dn", timestamp - start_timestamp);
}

10. Enabling DTrace in a Zone

11. Java Statically Defined Tracing (JSDT)
 Make points of interest in your application easily monitored
 Insert your own DTrace probes in desired locations inside your
methods
– Use them in conjunction with any other probes
#!/usr/sbin/dtrace -s
MyProvider:::start
{
self->start_time = timestamp;
}
syscall:::entry
/self->start_time/
{
@[probefunc] = quantize();
}

12. JSDT – define a Provider
public interface MyProvider extends com.sun.tracing.Provider
{
Provider Interface
void methodEntry();
void methodReturn();
void start();
void dataAdded(int x, int y);
void myProbe(int intData, String stringData);
}

13. JSDT – insert probes to your Java code
import com.sun.tracing.*;
Provider Interface
MyProvider provider;
public static void main(String args) {
ProviderFactory factory = ProviderFactory.getDefaultFactory();
provider = factory.createProvider(MyProvider.class);
}
public void method() {
provider.methodEntry();
...
provider.myProbe(i, str);
...
provider.methodReturn();
}
Call probes actions
Create the
provider

14. Fl ow Trac e- D o
em

15. Next Thoughts
 A special ‘Java-DTrace’ utility to do implicit instrumentation

Probes look like native DTrace PID provider:
−
−


JDDT$target:class-name:method-name:entry
JDDT$target:class-name:method-name:return
# jdtrace java-dtrace-script.d -p <process-id>
jdtrace will take care of all required dynamic instrumentation
−

Clean instrumented code on script end
Any suggestions?

16. Thanks!
Amit.Hurvitz@oracle.com