It802 bruning

Using DTrace to InstrumentYour
System
Max Bruning,Training Director, Joyent
Wednesday, March 26, 14

Topics Covered
• DTrace
• What it does.
• Uses
• How Instruments uses it
• Examples

Tools Used in This Talk
• DTrace
• Allows runtime observability into entire software stack
• Instruments
• performance, analysis, and testing tool for dynamically tracing and
proﬁling OS X and iOS code
• Many of the instruments tools use DTrace (some use llvm)

What is DTrace?
• Tool that allows one to dynamically instrument code from application level and into the
kernel.
• Can be used safely on production systems.
• Uses:
• Performance Analysis
• Debugging
• Code coverage
• Find out what is happening in your software
• Available on OS X, SmartOS, illumos, *BSD, Solaris and derivatives.
• Useful with C, Objective-C, C++, as well as higher level languages (java, perl, node, python,
etc.).

Terminology
•Probe - An instrumentation point in the code
•Dynamic and Static probes are provided, and new ones can be added
•A probe is speciﬁed by a 4-tuple:
•provider:module:function:probename{action}
•Action - Executed when a probe ﬁres
•Predicate - Optional boolean to determine whether or not to
execute the action
•Example: syscall::read:entry/pid == 713/{trace();}

Some Simple Examples
•Show system calls made by a running process
# dtrace -n ‘syscall:::entry/pid==26442/{}’
dtrace: description 'syscall:::entry' matched 234 probes
CPU ID FUNCTION:NAME
1 10157 write:entry
1 10155 read:entry
4 10155 read:entry
...
# dtrace -n 'syscall:::entry/execname == "Safari"/{@[probefunc]=count();}'
dtrace: description 'syscall:::entry' matched 456 probes
(^C)
...
mmap 183
psynch_mutexdrop 470
psynch_mutexwait 470
kevent64 6149
workq_kernreturn 6326
• Count system calls made by a running safari process

An Example Measuring
System Call Latency•systime.d
#!/usr/sbin/dtrace -s
#pragma D option quiet
syscall:::entry
/execname == "node"/
{
self->ts = timestamp;
}
syscall:::return
/self->ts/
{
@[probefunc] = quantize(timestamp - self->ts);
self->ts = 0;
}
END
{
! printa("SYSCALL NSECS # OF OCCURANCESn%s%@lxn", @);
}

An Example Measuring
System Call Latency (Continued)# ./systime.d
...
SYSCALL NSECS # OF OCCURANCES
read
value ------------- Distribution ------------- count
1024 | 0
2048 |@@@@@@@@@@@@@ 3
4096 |@@@@@@@@@ 2
8192 |@@@@@@@@@ 2
16384 |@@@@ 1
32768 | 0
...
33554432 | 0
67108864 | 0
134217728 |@@@@ 1
268435456 | 0
...

DTrace Providers for OS X
# dtrace -l | grep -v PROVIDER | awk '{print $2}' | sort -u | sed -e 's/[0-9].*//' | uniq | paste
- - - - | awk '{ printf "%20-s %20-s %20-s %20-sn", $1, $2, $3, $4 }'
AppContainer AppSandbox Cocoa_Autorelease Cocoa_Layout
CoreData CoreImage CoreText HALC_IO
HALS_Engine HALS_IO HALS_System JavaScriptCore
MetadataFramework MetadataImporter MobileDevice NSApplication
NSTrackingArea NSXPCConnection NSXPCListener NSXPCProxy
PDEProbes PMTiming PackageKit PrintCore
QLSeamlessClosing QLSeamlessOpening QLThumbnail QuartzComposer
QuickTimeX RawCamera RemoteViewServices SoftwareUpdate
Suggestions afc cache codesign
dtrace fbt garbage_collection io
ip ldap_rb_stats lockstat mach_trap
magmalloc mds node objc_runtime
odprobes opencl_api opencl_cpu perl
pid plockstat proc profile
sched sdt security_debug security_exception
security_machserver securityd syscall syspolicy
tcp vminfo
• Providers not in grey are OS X speciﬁc (?)
• More providers may exist, and more can be added

Instruments
• Performance, analysis, and testing tool for dynamically tracing and
proﬁling OS X and iOS code
• Many of the instruments tools use DTrace (some use llvm)
• Instruments provides a GUI for implementing your own instruments
using DTrace

Instruments and DTrace
Instruments
using Dtrace Sudden termination, Carbon Events, Core Data Saves/
Fetches/Faults, Cocoa Events/Layout, File Locks/Attributes,
File Activity, Directory I/O, Garbage Collection, Reads/
Writes, Shared Memory, Process, Custom Instruments
Instruments that
Could be
implemented
using DTrace
Time Proﬁler, Network Activity Monitor, Sampler, Memory/
Disk/CPU/Activity Monitor,Thread States,VM Operations,
System Calls, Scheduling

Example Instrument -
File Activity

File Activity (Continued)
• Edit as needed and
save or cancel.

File Activity (Continued)

DTrace Script$ cat fileact.d
#!/bin/bash
#
# To run this script, please type the following command while in the same
# directory (or specify the full path rather than './'):
#
# sudo ./fileact.d -o dtrace_output.txt
#
# After the output file has been generated (here, it's dtrace_output.txt but you
# may call it whatever you like), call up the trace document where
# this script was exported from and choose the "DTrace Data Import..." option
# from the "File" pulldown menu to import that data.
#
TOOLCHAINS=default
/usr/sbin/dtrace $* -C -Z -s /dev/stdin << "End-Of-File"
#pragma D option switchrate=1msec
...

Running the D Script
$ sudo ./fileact.d -o fileact.out
•Run your workload while this is running
•When ready, control-c or kill
•Import fileact.out into Instruments
•Modifying the script outside of Instruments may cause output
to not be importable
•Importing of arbitrary D script output into Instruments does
not work

D Script Output
<dtrace_output_begin/>
1395363306364638000
3A85E3B6-1514-4ECC-8EC8-40FB699DCAF9
<e> -1 4 2 1246115 1395363306366033000
fstat64
dtrace
6
</e>
<s>
libsystem_kernel.dylib`fstat$INODE64+0xa
libsystem_c.dylib`__smakebuf+0x2a
libsystem_c.dylib`__swsetup+0x84
libsystem_c.dylib`__v2printf+0x55
libsystem_c.dylib`__xvprintf+0x279
libsystem_c.dylib`vfprintf+0x49
libdtrace.dylib`dt_printf+0xf9
libdtrace.dylib`dt_printf_format+0x19c
libdtrace.dylib`dtrace_fprintf+0x33
libdtrace.dylib`dt_consume_cpu+0xc47
libdtrace.dylib`dtrace_consume+0x3cb
libdtrace.dylib`dtrace_work+0x7a
dtrace`main+0x1c9c
libdyld.dylib`start+0x1
dtrace`0x7
</s>
<e> -1 6 6 1246115 1395363306367047000
stat64
dtrace
/Library/Symbols
</e>
<s>
libsystem_kernel.dylib`stat$INODE64+0xa
...

Example D Script
That “mimics” Time Proﬁler
#!/usr/sbin/dtrace -qs
BEGIN
{
printf("%-24s %-8s %-16s %-24s %-24sn",
"Timestamp", "Depth", "Process", "Hot Frame", "Caller");
}
profile-1ms
/arg0/
{
printf("%-24Y %-8d %-16s %-24p %-24pn",
walltimestamp, stackdepth, execname, arg0, caller);
stack();
}
profile-1ms
/arg1/
{
printf("%-24Y %-8d %-16s %-24p %-24pn",
walltimestamp, stackdepth, execname, arg1, caller);
ustack();
}

Output from Profiler D Script
$ sudo ./prof1.d > prof1.out
^C
bash-3.2$ ls -l prof1.out
-rw-r--r-- 1 max staff 33694762 Mar 21 11:20 prof1.out
bash-3.2$ less prof1.out
Timestamp Depth Process Hot Frame Caller
2014 Mar 21 11:20:04 6 dtrace ffffff80290d41b2 ffffff80291f3ae1
mach_kernel`0xffffff80290d4060+0x152
mach_kernel`VNOP_IOCTL+0x150
mach_kernel`0xffffff80291f39b0+0x131
mach_kernel`fo_ioctl+0x43
mach_kernelìoctl+0x466
mach_kernelùnix_syscall64+0x1f3
mach_kernel`hndl_unix_scall64+0x16
2014 Mar 21 11:20:04 1 kernel_task ffffff80290dd76e 0
mach_kernel`machine_idle+0x1de
mach_kernel`call_continuation+0x17
...

DTrace Scripts Available
on OS X$ man -k dtrace
bitesize.d(1m) - analyse disk I/O size by process. Uses DTrace
cpuwalk.d(1m) - Measure which CPUs a process runs on. Uses DTrace
creatbyproc.d(1m) - snoop creat()s by process name. Uses DTrace
dappprof(1m) - profile user and lib function usage. Uses DTrace
dapptrace(1m) - trace user and library function usage. Uses DTrace
diskhits(1m) - disk access by file offset. Uses DTrace
dispqlen.d(1m) - dispatcher queue length by CPU. Uses DTrace
dtrace(1) - generic front-end to the DTrace facility
dtruss(1m) - process syscall details. Uses DTrace
errinfo(1m) - print errno for syscall fails. Uses DTrace
execsnoop(1m) - snoop new process execution. Uses DTrace
fddist(1m) - file descriptor usage distributions. Uses DTrace
filebyproc.d(1m) - snoop opens by process name. Uses DTrace
hotspot.d(1m) - print disk event by location. Uses DTrace
...
•42 scripts available, and more can be found at http://www.dtracebook.com/index.php/Main_Page

Example - dtruss
• Trace system calls
$ sudo dtruss -n Mail
PID/THRD SYSCALL(args) = return
3113/0x47af: select(0x60, 0x61000020C600, 0x61000020C680, 0x0, 0x0) = 1 0
3113/0x47af: recvfrom(0x22, 0x1132ADAA0, 0x100) = 1 0
3113/0x1a3059: kevent64(0x3, 0x11623DE18, 0x1) = 1 0
3113/0x46b0: workq_kernreturn(0x20, 0x0, 0x1) = 0 0
3113/0x46b0: kevent64(0x3, 0x0, 0x0) = 1 0
3113/0x179187: fcntl(0x8, 0x8, 0x117BB8900) = 0 0
3113/0x179187: fcntl(0x8, 0x8, 0x117BB8920) = 0 0
3113/0x179187: fcntl(0x8, 0x8, 0x117BB8A60) = 0 0
3113/0x179187: pread(0x6, "r0", 0x1000, 0xC8C0000) = 4096 0
3113/0x179187: workq_kernreturn(0x20, 0x0, 0x1) = 0 0
3113/0x179187: pwrite(0x7, "0", 0x1018, 0x3DFC90) = 4120 0
3113/0x179187: pwrite(0x7, "0", 0x1018, 0x3E0CA8) = 4120 0
3113/0x1a28d7: open("/Users/max/Library/Mail/V2/IMAP-max@joyent.com@imap.gmail.com/
[Gmail].mbox/All Mail.mbox/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX/Data/8/7/1/Messages/
178177.emlx0", 0x8000, 0x0) = 46 0
...

Example - iosnoop
• Trace Disk I/O Events
$ sudo iosnoop -a
STRTIME DEVICE MAJ MIN UID PID D BLOCK SIZE PATHNAME ARGS
2014 Mar 21 15:33:00 ?? 1 2 502 3113 W 132761912 4096 ??/MailData/Envelope Index-wal Mail0
2014 Mar 21 15:33:00 ?? 1 2 502 3113 W 132763320 8192 ??/Messages/177915.emlx.tmp Mail0
2014 Mar 21 15:33:00 ?? 1 2 502 3113 W 132767000 8192 ??/Messages/177918.emlx Mail0
2014 Mar 21 15:33:00 ?? 1 2 0 128 R 24160560 4096 ??/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX/
reverseDirectoryStore mds_stores0
...

Example - opensnoop
• Trace open(2) calls on ﬁles
$ sudo opensnoop -a
TIME STRTIME UID PID FD ERR PATH ARGS
276685505483 2014 Mar 21 15:39:59 0 128 5 0 .
mds_stores0
276685505676 2014 Mar 21 15:39:59 0 128 5 0 .
mds_stores0
276688258585 2014 Mar 21 15:40:02 502 17327 10 0 /Users/max/Library/Saved
Application State/org.gnu.Aquamacs.savedState/data.data Aquamacs0
Application State/org.gnu.Aquamacs.savedState/windows.plist Aquamacs0
...

Example - hfsslower
• Trace slow reads/writes on hfs File System (from DTrace book)
$ sudo ./dtbook_scripts/Chap5/hfsslower.d -x dynvarsize=25m 1
TIME PROCESS D KB ms FILE
2014 Mar 22 14:09:48 mds W 0 3 journal.1
2014 Mar 22 14:09:50 mdworker R 0 4 Info.plist/..namedfork/rsrc
2014 Mar 22 14:10:42 bash R 0 4 ps/..namedfork/rsrc
2014 Mar 22 14:12:50 vmware-vmx W 15 4 Virtual Disk-s001.vmdk
2014 Mar 22 14:13:19 Mail W 1325 4 156241.emlx.tmp
2014 Mar 22 14:13:58 fontd R 0 4 GillSans.ttc/..namedfork/rsrc
2014 Mar 22 14:14:09 Mail W 1912 7 156197.emlx.tmp
2014 Mar 22 14:14:21 Mail R 1913 2 156197.emlx
2014 Mar 22 14:14:21 Keynote W 0 4 data.data
2014 Mar 22 14:14:26 AppleSpell R 0 4 findNames/..namedfork/rsrc
2014 Mar 22 14:14:34 mdworker R 0 4 Info.plist/..namedfork/rsrc
2014 Mar 22 14:14:49 mds W 0 3 journal.1
2014 Mar 22 14:14:56 Keynote R 256 1 using_dtrace.key

Example - soconnect_mac.d
• Trace connect(2) calls (from DTrace book)
$ sudo ./dtbook_scripts/Chap6/soconnect_mac.d
Password:
PID PROCESS FAM ADDRESS PORT LAT(us) RESULT
dtrace: error on enabled probe ID 2 (ID 1035: syscall::connectx:entry): invalid
address (0x0) in action #1 at DIF offset 32
166 acwebsecagent 2 XXX.XXX.XXX.XXX 80 41 In progress
...

(Continued)
• What is “dtrace: error on enabled probe ID 2 (ID 1035:
syscall::connectx:entry): invalid address (0x0) in action #1 at DIF offset 32” ?
• probe ID 2 is the second probe:
syscall::connect*:entry
{
/* assume this is sockaddr_in until we can examine family */
this->s = (struct sockaddr_in *)copyin(arg1, sizeof (struct sockaddr));
this->f = this->s->sin_family;
}

(Continued)
• What is arg1 to connectx()?
$ sudo dtrace -n 'syscall::connectx:entry{printf("arg0 = %x arg1 = %x arg2 = %x arg3 = %x arg4 = %xn", arg0,
arg1, arg2, arg3, arg4);}'
dtrace: description 'syscall::connectx:entry' matched 1 probe
CPU ID FUNCTION:NAME
2 1035 connectx:entry arg0 = 14 arg1 = 0 arg2 = 0 arg3 = 7fc42bd1fd10 arg4 = 10
4 1035 connectx:entry arg0 = 1a arg1 = 0 arg2 = 0 arg3 = 7fc42be471a0 arg4 = 10
0 1035 connectx:entry arg0 = 10 arg1 = 0 arg2 = 0 arg3 = 11b5380e0 arg4 = 10
7 1035 connectx:entry arg0 = 14 arg1 = 0 arg2 = 0 arg3 = 7fc429725bf0 arg4 = 10
...

(Continued)
• What is arg3 to connectx()?
• Assume it is a sockaddr_in.
#!/usr/sbin/dtrace -qs
inline int af_inet = 2;
dtrace:::BEGIN
{
printf("%-6s %-16s %-3s %-16s %-5s %8sn", "PID", "PROCESS", "FAM",
"ADDRESS", "PORT", "LAT(us)");
}
syscall::connectx:entry
{
this->s = (struct sockaddr_in *)copyin(arg3, sizeof (struct sockaddr));
this->f = this->s->sin_family;
self->inconnectx = 1;
}

(Continued)
syscall::connectx:entry
/this->f == af_inet/
{
self->family = this->f;
self->port = (this->s->sin_port & 0xFF00) >> 8;
self->port |= (this->s->sin_port & 0xFF) << 8;
this->a = (uint8_t *)&this->s->sin_addr;
this->addr1 = strjoin(lltostr(this->a[0] + 0ULL), strjoin(".",
strjoin(lltostr(this->a[1] + 0ULL), ".")));
strjoin(lltostr(this->a[1] + 0ULL), ".")));
lltostr(this->a[3] + 0ULL)));
self->address = strjoin(this->addr1, this->addr2);
self->start = timestamp;
}

(Continued)
syscall::connectx:return
/self->start/
{
this->delta = (timestamp - self->start) / 1000;
printf("%-6d %-16s %-3d %-16s %-5d %8dn", pid, execname,
self->family, self->address, self->port, this->delta);
self->family = 0;
self->address = 0;
self->port = 0;
self->start = 0;
}

(Continued)
$ sudo ./connectx.d
PID PROCESS FAM ADDRESS PORT LAT(us)
8801 com.apple.WebKi 2 XXX.XXX.XXX.XXX 80 41
21473 ocspd 2 XXX.XXX.XXX.XXX 80 97
...

Flame Graphs
• Visualization of proﬁled code paths
• Shows distribution of code paths over time
• https://github.com/brendangregg/FlameGraph/

An Example Flame Graph
•Colors have no meaning
•Left to right is not time
sequenced
•Moving cursor over item
shows function, number of
times that function occurs , and
percentage.
•Longer horizontal bands
implies longer time in that
function or a function called
from that function

Building Flame Graphs Example -
DTrace Script#!/usr/sbin/dtrace -s
#pragma D option quiet
profile:::profile-97hz/arg0 != 0/
{
@k[stack()] = count();
}
profile:::profile-97hz/arg1 != 0 && execname != "dtrace"/
{
@u[ustack()] = count();
}
tick-60s {exit(0);}

Building Flame Graphs Example
(Continued)
$ git clone https://github.com/brendangregg/FlameGraph/
$ sudo ./prof.d -o prof.out
$ ls -l prof3.out
-rw-r--r-- 1 root staff 2286203 Mar 24 11:35 prof.out
$ ./stackcollapse.pl prof.out | ./flamegraph.pl > prof.svg
$ open prof3.svg
$
•Alternatively
$ git clone https://github.com/davepacheco/node-stackvis
...
$ cd node-stackvis
$ make
...
$ stackvis dtrace flamegraph-svg < prof.out > prof1.svg

Node.js - Example
• Node.js (and other higher level languages) has probes built in.
• Some Node.js modules also have DTrace support
• Example is from https://github.com/mcavage/node-restify, (git
clone and then npm install)

Node.js - Example Continued
/* restifyserver.js */
var restify = require('restify');
var server = restify.createServer({
name: 'myapp',
version: '1.0.0'
});
server.use(restify.acceptParser(server.acceptable));
server.use(restify.queryParser());
server.use(restify.bodyParser());
server.get('/echo/:name', function (req, res, next) {
res.send(req.params);
return next();
});
server.listen(8080, function () {
console.log('%s listening at %s', server.name, server.url);
});

Node.js - Example Continued
/* restifyclient.js */
var assert = require('assert');
var restify = require('restify');
var client = restify.createJsonClient({
url: 'http://localhost:8080',
version: '~1.0'
});
client.get('/echo/mark', function (err, req, res, obj) {
assert.ifError(err);
console.log('Server returned: %j', obj);
});

Node.js Example - Continued
$ node restifyserver.js
myapp listening at http://0.0.0.0:8080
$ sudo dtrace -q -l -n 'restify*:::{}'
ID PROVIDER MODULE FUNCTION NAME
214546 restify28832 mod-0x100f00420 route-start route-start
214547 restify28832 mod-0x100f00420 handler-start handler-start
214548 restify28832 mod-0x100f00420 handler-done handler-done
214549 restify28832 mod-0x100f00420 route-done route-done
214550 restify28832 mod-0x100f00420 client-request client-
request
214551 restify28832 mod-0x100f00420 client-response client-
response
217839 restify28832 mod-0x100f00420 client-error client-error

$ sudo dtrace -l -n 'node*:::{}'
ID PROVIDER MODULE FUNCTION NAME
214536 node28832 node
_ZN4nodeL14dtrace_gc_doneEN2v86GCTypeENS0_15GCCallbackFlagsE
[node::dtrace_gc_done(v8::GCType, v8::GCCallbackFlags)] gc-done
214537 node28832 node
_ZN4nodeL15dtrace_gc_startEN2v86GCTypeENS0_15GCCallbackFlagsE
[node::dtrace_gc_start(v8::GCType, v8::GCCallbackFlags)] gc-start
214538 node28832 node
_ZN4node26DTRACE_HTTP_CLIENT_REQUESTERKN2v89ArgumentsE
[node::DTRACE_HTTP_CLIENT_REQUEST(v8::Arguments const&)] http-client-request
214539 node28832 node
_ZN4node27DTRACE_HTTP_CLIENT_RESPONSEERKN2v89ArgumentsE
[node::DTRACE_HTTP_CLIENT_RESPONSE(v8::Arguments const&)] http-client-response
214540 node28832 node
_ZN4node26DTRACE_HTTP_SERVER_REQUESTERKN2v89ArgumentsE
[node::DTRACE_HTTP_SERVER_REQUEST(v8::Arguments const&)] http-server-request
...

$ node ./restifyclient.js
Server returned: {"name":"mark"}$ sudo dtrace -qn 'restify*:::{printf("%s: %s:
%sn", probeprov, probefunc, probename);}'
restify28832: route-start: route-start
restify28832: handler-start: handler-start
restify28832: handler-done: handler-done
restify28832: route-done: route-done
restify28844: client-response: client-response
restify28844: client-request: client-request

What’s Missing?
• ustack helpers - Allows jstack() action (stack backtrace showing
function calls in higher level languages)
• CPU performance counters?
• Some source

References
• http://www.dtrace.org
• http://dtrace.org/blogs/dap/category/nodejs/
• http://dtrace.org/blogs/brendan/2011/10/10/top-10-dtrace-scripts-for-
mac-os-x/
@mrbruning, max@joyent.com

It802 bruning

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Semelhante a It802 bruning

Semelhante a It802 bruning (20)

It802 bruning