1. Accelerating the Development of Embedded
Linux Devices with JTAG On-Chip Debugging
The two most common scenarios where the JTAG connection
Table of Contents can be used are when an Ethernet connection is not available
for agent-based debugging and when the developer needs
Introduction......................................................................... 1 to resolve issues that occur in the Linux kernel or user space,
Overview of the Linux Operating System........................... 1 and system-mode debugging is required. With the next-
Complexities of Debugging Embedded generation JTAG tools from Wind River, developers can
Linux Devices...................................................................... 2
. debug the hardware, bootloader, Linux kernel, and user
The Bootloader................................................................... 2
. space, spotting system crash events and other problems that
occur between the kernel, user space, and target. Moreover,
Bootloader Debug Issues................................................ 2
for developers using the Wind River Linux operating system
Simplifying Bootloader Debug........................................ 3 (OS), developers can move seamlessly between agent and
Linux Kernel and Kernel Modules....................................... 3
. JTAG debugging. These innovations in on-chip debugging
Using JTAG to Debug..................................................... 3 provide powerful alternatives for projects where agent-based
Linux Application Debug..................................................... 4 debugging is either unavailable or is too expensive.
Wind River Solutions for JTAG-Based Overview of the Linux Operating System
Linux Debug........................................................................ 5
The first step in troubleshooting Linux is to understand the
Advanced Wind River Solutions for Linux OS and the areas in which a debugger needs to analyze
Linux Debug........................................................................ 5 interactions. Linux is typically comprised of multiple
Wind River On-Chip Debugging for the interacting components and employs complex memory
Entire Linux Development Process................................. 6 mapping and management schemes. Debugging a Linux-
Conclusion........................................................................... 6 based embedded system requires powerful tools that expose
Notes ................................................................................. 6 the operation of the OS and its interactions with the hardware.
The main components of the Linux OS are the Linux kernel,
kernel modules, and application software. The Linux kernel is
Introduction the core of the OS and has ultimate authority. Linux kernel
Debugging embedded Linux products has traditionally modules are elements of the OS that are dynamically loaded
required a mixture of hardware and software tools—JTAG and unloaded and are often used for device drivers. Once a
tools for hardware bring-up and agent-based solutions for
software development. These JTAG- and agent-based tools
often solved point problems but were not designed for
integrated Linux development. User Mode Applications
Wind River has changed the way developers can debug Linux
by merging traditional JTAG hardware debugging with Linux
kernel configuration; patch management; and user-space Kernel Modules
application development, debug, and analysis in an Eclipse-
Kernel Mode
based integrated development environment (IDE) known as
Wind River Workbench. This capability enables developers to Linux Kernel
use the JTAG connection when traditional agent-based
solutions are not technically or economically feasible. Figure 1: Components of the Linux OS
2. Virtual Physical kernel modules, care must be taken to account for the static
Memory Memory and dynamic address translation capabilities of the Linux
(RAM)
kernel as well as its ability to dynamically allocate processes
in memory.
0x10000788
Complexities of Debugging Embedded Linux Devices
0x1EA78788 Linux is continuing to rapidly grow in embedded devices.
Mapped According to technology market research firm Venture
Development Corporation (VDC), 23 percent of new projects
Figure 2: Linux Memory Management
will use Linux1. Debugging Linux embedded projects can be
extremely complex as development spans bootloader, Linux
Linux kernel module is loaded, it has the same level of kernel, kernel module, and applications. Issues that Linux
authority as the Linux kernel. The Linux kernel partitions all developers must contend with include establishing a target
software to run in either user mode or Linux kernel mode. configuration file for the bootloader, traversing Linux virtual
Applications that run in user mode on a Linux system have addressing from user mode to kernel mode and back,
reduced privileges. For example, they cannot access the mapping kernel symbol information, and troubleshooting
Linux kernel’s memory or hardware directly. This is designed difficult defects that span across user and kernel space. None
to prevent damage to the underlying system by an of these issues is easily solved with agent-based debugging
application. Therefore, the Linux kernel is responsible for solutions such as the kernel GNU debugger (KGDB) or the
providing all system-level access. It allows the application GNU debugger (GDB).
software controlled access to the system, including
peripherals and memory. Debug Challenges in Linux
In a Linux system, applications typically run in user mode • Hardware initialization
while device drivers run in kernel mode. This scheme aids in
• Debugging the bootloader
protecting the core (Linux kernel) from potential system
crashes or reduced system availability but has the trade-off • Access to instruction and data cache
of reduced performance, as Linux kernel modules typically • Access to initialization code in the Linux kernel
have better performance than user-space applications. • Debugging interactions between applications and
Linux kernel
Unlike simpler “flat” or “statically” allocated 8-, 16-, and
32-bit operating systems utilized in many embedded devices,
Linux implements an advanced technique of virtual
addressing. Virtual addressing allows Linux kernel modules The Bootloader
and applications loaded and running in the system to appear Linux relies on a bootloader to start the OS. The bootloader
to be in a continuous memory address space, while the actual is code that resides in flash or other nonvolatile memory and
location of the software in physical memory may be executes after the system is powered on or reset. Low-level
fragmented across multiple memory regions. The Linux initialization of the CPU, memory, and critical I/O and
kernel handles memory mapping and memory allocation for peripherals is performed by the bootloader. Once bootloader
Linux kernel modules and applications. Statically mapped execution starts, it copies itself to RAM and then it starts the
memory management, used for the Linux kernel and Linux Linux OS.
kernel modules, involves mapping from virtual to physical
Bootloader Debug Issues
addressing with a defined constant (e.g., offset). Dynamic
mapping from virtual to physical memory, used for user- Debugging the bootloader can be quite complex because it
mode applications, may use complex methods to map data in is hardware-centric and developers must contend with the
physical memory. bootloader relocating from flash to RAM after startup. With
modern system-on-chip (SoC) processors there can be
The Linux kernel also has the ability to allocate processes
hundreds of configuration registers that need to be
into memory to run; processes may include Linux kernel
initialized. This requires perusal of thousands of pages of
modules or user-mode applications. These processes can be
documentation on specific settings. Setting the wrong value
allocated into memory to complete a specific task and then
on registers could cause issues downstream when starting
be removed once they are no longer needed, providing
the Linux kernel or debugging applications. Manual editing
better system memory management. Therefore, when
to set registers is a very cumbersome process.
traversing memory from application to the Linux kernel and
2 | Accelerating the Development of Embedded Linux Devices with JTAG On-Chip Debugging
3. Another common challenge with bootloader development is kernel module debug will also require transparency of
debugging issues that occur as the bootloader loads Linux initialization code while dealing with dynamically allocated
into RAM and launches the OS. As the bootloader loads the memory in RAM.
Linux kernel into physical memory, the kernel creates its own
KGDB requires a stable Linux kernel and custom hardware
virtual memory map, which often makes it difficult to
interfaces such as device drivers to be ready for the agent to
determine which addresses the kernel is using. Therefore, it
work. Agent-based debugging does not have the visibility
can be quite complex to determine how to relocate the
into the hardware and does not provide the full diagnostic
symbol table for the Linux OS once the kernel begins
capabilities required to understand the interaction between
executing out of RAM. Unfortunately, agent-based
the hardware and the Linux kernel. Agents require
debugging solutions do not support bootloader debug
instrumentation of the Linux kernel and this may cause side
because the OS is not functional during this process.
effects in embedded devices.
Therefore, developers rely on JTAG tools for debugging this
critical process. Other considerations when using agents to debug the Linux
kernel and kernel modules involve stopping or freezing the
system when a breakpoint is hit. For example, KGDB will not
Spending a significant amount of time developing and
stop the CPU (especially in a multicore or multiprocessing
debugging the bootloader may prevent developers from
environment) to permit the developer to examine the current
focusing on stabilizing and debugging device software
state of the system. KGDB will not help debug a crashed
and applications.
system because it requires an operational system for the
agent to work. Additionally, KGDB requires a communication
Simplifying Bootloader Debug port such as Ethernet to communicate from a host system to
Agent-based debug solutions such as KGDB and GDB will not the target. Therefore, using an agent for Linux kernel-mode
work for bootloader debug. However, JTAG debug solutions debug requires that a communication channel be established
provide powerful capabilities to help developers quickly and with a working IP stack, a stable Linux kernel, and working
efficiently troubleshoot and test the bootloader. With JTAG device drivers. This may be a challenge during OS
debug solutions, developers are easily able to set register stabilization where such communication facilities may not be
values and peruse register contents easily. The ability to set available or may themselves be required to be debugged.
hardware breakpoints enables code stepping in flash
Using JTAG to Debug
memory to quickly identify the source of errors. IDEs that
support disassembly and mixed modes for viewing of source Comprehensive debugging solutions that operate with the
code along with assembly code simplify the debug process. Linux kernel and kernel modules are needed to validate and
Powerful symbol management capabilities can help with debug stabilize the Linux kernel for a target system. JTAG-based
by accounting for the relocation of code from flash to RAM. debugging solutions offer significant value in this workflow.
Features of these JTAG-based debugging tools include the
An added benefit of some JTAG debug solutions is the ability
ability to view local/global symbols and registers as well as
to load the Linux kernel without a bootloader. This capability
view data and instruction cache contents. Some commercial-
is accomplished with the use of boot line parameters and is
based JTAG solutions provide seamless address mapping
particularly useful for those project managers intending to
from virtual to physical memory, allowing the developer to
start development of their systems even before the
bootloader is ready and available. JTAG-based solutions that
Software
provide a boot line capability enable parallel bootloader
development and OS stabilization and help shorten
development schedules. App1 App2
Linux Kernel and Kernel Modules Applications
Linux kernel and kernel modules are the core components of
the Linux OS. The Linux kernel is the first to start after the
system has been initialized by the bootloader. Linux kernel
Linux Kernel
modules are then loaded as needed. Kernel Modules
(Device Drivers)
During the period of OS bring-up, developers focus on the
optimization or customization of the Linux OS and the
development of kernel modules. Interaction between the I/O Interrupts
hardware and software is monitored very closely during this
phase. For Linux kernel debugging, developers require Hardware
visibility to registers, caches, and other low-level data. Linux
Figure 3: Software and Hardware Interaction in Linux
3 | Accelerating the Development of Embedded Linux Devices with JTAG On-Chip Debugging
4. view the correct memory address and content. These Keeping track of memory addresses when stepping into and
solutions permit debug of the initialization of Linux kernel out of the two modes is challenging because of the virtual
modules and the ability to load and unload these multiple memory management structure of Linux. Relying on the same
times without having to disconnect and reconnect to the physical address will not work. Correctly keeping track of
target system. memory mapping and memory allocation during this process
is critical. As a result, agent solutions require the use of both
The ability to completely halt the system and examine the
GDB and KGDB to trace system calls into the Linux kernel and
state of the OS and applications is a powerful feature of JTAG
kernel modules. The use of multiple agent-based debug
debug solutions. Known as system mode debug, this
tools may complicate the debug process.
capability is extremely useful for Linux kernel and Linux
kernel module debug. With system mode debug, the Linux application developers may also encounter embedded
developer has the ability to stop the entire system, including systems that do not have an Ethernet or serial communication
the processor, OS, and all threads, and interrupt processing. port on the device. In these cases, agent-based debugging
By stopping the system in this manner, it is possible to obtain cannot be used. Even if the device has a communication port,
detailed views of the hardware and software. It is also possible agents require that the interface be ready. If the
to restart the system and to step through code after restart. communication port is not ready or requires debug, or the
memory is not available for the IP stack or agent software,
A JTAG solution will work when KGDB cannot, especially agent-based debugging will not work.
in Linux kernel panic and target crash conditions. These JTAG-based debugging provides deep visibility into
capabilities are particularly useful for stabilizing device applications that are operating in Linux user mode. For
drivers and the OS. applications that make system calls, the dual-mode JTAG
debugging solution permits visibility into both the Linux
kernel and user mode. All of this occurs without any Linux
Linux Application Debug
kernel instrumentation. The dual-mode solution enables
Linux applications are user programs that run under the
viewing of all application threads, their contexts, and which
control of the Linux kernel. Linux applications gain access to
threads entered the Linux kernel, as well as the parameters
system resources by making system calls. The Linux kernel
used and the thread variables. For devices that lack a
handles the system calls and it decides how to provide access
communication port (e.g., mobile devices, medical devices,
to hardware and memory.
automotive systems, etc.), JTAG-based solutions provide an
For user-mode application debugging, developers require excellent alternative to agent-based debug.
direct access to application threads with the ability to start
and stop a thread and view
variables and the stack. Since
an application may consist of
many processes or threads, it
may be necessary to stop all
threads related to the
application thread being
debugged, including threads
that may influence the
application. It may also be
necessary to watch peripheral
registers across different
processes and CPUs. GDB
only operates at the thread
level and is only able to stop a
single thread. GDB is not able
to stop an entire system or
multiple threads
simultaneously.
Developers debugging in
user mode will require
stepping into system calls
from user mode to Linux
kernel mode and back. Figure 4: An Eclipse-Based IDE for Linux Application Debug
4 | Accelerating the Development of Embedded Linux Devices with JTAG On-Chip Debugging
5. Debugging multithreaded applications is simplified with the
use of system-mode debug, which is the ability to halt the
processor and examine the state of the OS and all threads. As Wind River
On-Chip Debugging
mentioned previously, many problems occur due to the
interaction of multiple threads. Since agent-based debugging
can’t stop all threads simultaneously, it can be harder to
pinpoint problems, which means it can take significantly
longer to debug a project. System-mode debug permits a
detailed view of the current state of the system (e.g., each
Ethernet
thread’s state, variables, etc.) and is a better alternative
compared to agent-based debug because it allows full view of
Linux Target System
the system at a given point in time.
The JTAG-based debug solution is a nonintrusive connection
JTAG
to the target hardware. It can connect to systems already
Connection
running that have hit errors. It can connect without changing
the state of processor registers and synchronize contexts for
the Linux kernel and applications to debug. An example is a
Linux-based system stuck in a deadlock condition. With a Figure 5: Wind River JTAG-Based On-Chip Debugging Solution
JTAG solution, the developer may connect to the target
system without altering its state. The developer then is able to Advanced Wind River Solutions for Linux Debug
view Linux kernel objects, application contexts to determine Running on the industry-standard Eclipse-based platform,
the threads, system calls, and parameters used for system Wind River Workbench, On-Chip Debugging Edition offers
calls that caused the fault condition. It is a single end-to-end advanced JTAG-based debugging features for embedded
solution especially useful where an agent-based development Linux systems. Workbench On-Chip Debugging supports
solution cannot be used. JTAG debugging reduces Wind River Linux. For roll-your-own Linux or semiconductor
debugging time through development tool simplicity. Linux distributions, Wind River has extended the on-chip
debugging capabilities to enable JTAG-based kernel and
Wind River Solutions for JTAG-Based Linux Debug
user-mode debugging of kernel.org Linux. Wind River
JTAG-based on-chip debugging solutions from Wind River debugging solutions support a variety of target hardware,
complement rather than replace popular agent-based including the most advanced multicore processors.
solutions. Solutions that aren’t JTAG-based such as the
agent-based KGDB and GDB may not be suitable for early-
stage development, such as bootloader debug, OS
stabilization, and device driver debug. This can impact the
development of products that rely only on agent-based
debugging.
Typically, most devices include JTAG ports to enable
debugging and testing. However, many consumer products—
particularly those that are highly price-sensitive, space
constrained, or memory constrained—do not include a
communication port that is necessary to perform agent-based
debugging. In other cases, agent-based debugging is not
available because of stringent physical and memory space
requirements, security concerns, or the lack of an IP stack.
JTAG-based solutions are ideal for these environments.
Figure 6: Wind River Workbench, On-Chip Debugging Edition and Wind
JTAG also provides an important view into the overall system River ICE for JTAG-Based Debug
that is not available when using an agent for debug. Given the
dual-user and Linux kernel-mode structure of Linux, agent-
based debugging using GDB and KGDB may cause difficulty
for developers who need to obtain information regarding the
application, the OS, and the underlying hardware all at the
same time. JTAG solutions can give important visibility into
these areas simultaneously, making debugging easier and
faster.
5 | Accelerating the Development of Embedded Linux Devices with JTAG On-Chip Debugging
