For a numbers of years, silicon vendors have been providing hardware tracing facilities to embedded developers. By using these, developers can resolve performance and latency issues more quickly, resulting in shorter time to market. In this talk, we will cover the hardware based tracing facilities offered by various manufacturers and see how they differ from their software counterparts with respect to their instrumentation capabilities, transport mechanisms, output formats, etc. We will also show how joint hardware and software tracing can be used by developers to gain deeper insights in their applications’ behaviour. Finally, we will outline the on-going work within the Linux Trace Toolkit next generation (LTTng) project to enhance hardware tracing support and tracing data visualization.
5. whoami
Christian Babeux, Software Developer, EfficiOS,
Background in embedded and ASIC tools,
Active contributor to the LTTng projects:
●
●
lttng-tools, lttng-ust, babeltrace,
CI infra, Website, Twitter.
AUR package maintainer for Arch Linux.
5
6. Content
What is hardware tracing?,
Why is it useful?,
ARM Coresight & ETM,
Freescale QorIQ & Nexus tracing,
LTTng & hardware tracing.
6
8. What is hardware tracing?
Hardware component(s) used to traces instructions
and data movement of a processing device
Real-time observation
Low intrusiveness
8
10. What is hardware tracing? Cont.
●
Pros
–
–
●
Can accommodate high data bandwidth
Minimal impact on system performance
–
Trace port not always available
–
Custom hardware needed
Cons
10
12. What is hardware tracing? Cont.
●
Pros
–
–
●
Self-contained, facilities can be used by host OS
No need for special hardware
–
Limited internal buffer space
–
Might impact system performance
Cons
12
13. Hardware tracing support
●
ARM
–
–
Program Trace Flow Macrocell (PTM),
–
●
Embedded Trace Macrocell (ETM),
System Trace Macrocell (STM)
PowerPC
–
–
●
Freescale QorIQ with Nexus tracing
Branch History Rolling Buffer (BHRB)
–
Intel Processor Tracing (PT)
–
Last Branch Record (LBR), Branch Trace Store (BTS)
Intel
13
14. Hardware tracing vs. software tracing
●
Software tracing:
–
–
Can be intrusive
–
Can be slow
–
●
Static instrumentation or dynamic code patching
Tracepoint level granularity
Hardware tracing:
–
Tracing done on hardware
–
Instrumentation not required
–
Instruction level granularity
14
19. ARM Coresight
●
Coresight
–
–
Trace and debug a complete SoC
–
●
Collection of hardware components
Open architecture
Trace source
–
Processing elements (CPU, DSP, etc.)
–
Buses
–
System trace (generated from software)
19
20. ARM ETM
●
ETM
–
Monitor the core internal bus
–
Instructions + data trace
–
Hardware filters and triggers
–
Trace stream compression
–
Traces can be saved in internal buffer (ETB) or shared
system memory
20
22. State of Coresight & ETM in Linux
●
ETM tracer implementation available in Linux
–
●
Coresight support status
–
●
Seems to work only on specific hardware
Framework patchset proposed by Pratik Pratel [3]
Trace decoder availability?
22
25. Freescale QorIQ
●
PowerPC based platform targeted for high-performance
communications usage
–
Multiple e500mc processors,
–
DPAA support (packet processing offloading),
–
Support the Nexus debugging & tracing standard.
Source: radiolocman 25
[4]
26. Nexus standard
●
●
●
ISO standard for debugging embedded systems (IEEEISTO-5001-2003),
Designed for low pin count, standard set of connectors
(JTAG or Debug port),
Multiple level of Nexus “compliance”.
26
27. Nexus standard level
●
Level 1:
–
–
●
Run time control only (run, stop, breakpoints, etc.)
Tracing not supported
Level 2:
–
●
Level 3:
–
●
Ownership and program trace
Data write trace & memory read/write on the fly
Level 4:
–
Memory substitution
–
Trace triggering via a watchpoint
27
28. Nexus standard format
●
Packet based output format,
–
Standard defines public messages, vendors can define
extensions (TCODES)
–
Fixed packet size per message, last packet can be of
variable length
–
Message can have an optional timestamp
28
30. State of Nexus in Linux
●
Nexus qoriq-debug kernel module,
–
Available in Freescale QorIQ SDK Yocto Layer
–
Implements a debugfs with memory mapped access to
Nexus control register
–
cat /sys/kernel/debug/npc/trace_buffer > trace
30
31. Nexus debugfs
root@p3041ds:~# ls -al /sys/kernel/debug/qoriq-dbg
[...]
drwxr-xr-x 2 root root 0 Aug 6 20:22 cpu0
drwxr-xr-x 2 root root 0 Aug 6 20:22 cpu1
drwxr-xr-x 2 root root 0 Aug 6 20:22 cpu2
drwxr-xr-x 2 root root 0 Aug 6 20:22 cpu3
drwxr-xr-x 2 root root 0 Aug 6 20:22 ddr1
drwxr-xr-x 2 root root 0 Aug 6 20:22 dpaa
[...]
drwxr-xr-x 2 root root 0 Aug 6 20:22 npc
drwxr-xr-x 2 root root 0 Aug 6 20:22 nxc
[...]
root@p3041ds:~# ls -al /sys/kernel/debug/qoriq-dbg/cpu0
[...]
-rw-rw-rw- 1 root root 0 Aug 6 20:22 dc1
-rw-rw-rw- 1 root root 0 Aug 6 20:22 dc2
-rw-rw-rw- 1 root root 0 Aug 6 20:22 dc4
--w--w--w- 1 root root 0 Aug 6 20:22 ddam
[...]
31
32. State of Nexus in Linux
●
Nexus decoder availability
–
●
Released as part of the babeltrace project
Open-source software to reconstruct program flow not
yet developed
–
Integrate such functionality in an IDE ? perf?
32
35. Tracers
●
●
lttng-modules: kernel tracer module, compatible
with kernels from 2.6.38* to 3.11,
lttng-ust: user-space tracer, in-process library.
* Kernel tracing is now possible on 2.6.32 to 2.6.37 by backport of 3
Linux Kernel patches [1].
35
38. Hardware tracing support
●
Initial attempt to support hardware tracing:
–
–
●
Babeltrace Nexus to CTF converter [5]
Goal: Leverage existing traces visualizer
Encountered issues:
–
Traces are not self-contained, need sideband
information
–
Internal trace buffer size limitation
–
Synchronization with other traces can be tricky
38
40. Future work
●
●
●
Decoder/Converter for ARM ETM?
Control hardware tracing facilities with the lttngtools command-line?
Custom views for hardware traces in Eclipse plugin
40
41. Conclusion
●
Availability and usefulness of hardware tracing
●
Initial support for self-hosted hardware tracing
●
Common abstraction for hardware tracing in the
Linux kernel?
41