If you want to trace or extract specific data from a Java application with OpenTelemetry, you usually have to modify the application’s code. However, this is often not possible, especially with bought-in software. We would like to show, how the open source inspectIT Ocelot Java agent can be used to dynamically inject OpenTelemetry code at runtime for extracting specific application and business data – and all this without having to adapt the application itself.

1. inspectIT Ocelot: Dynamic OpenTelemetry
Instrumentation at Runtime
Open Source Monitoring Conference 2021

2. The Scenario
2
Is the application
working?
How many users
are accessing it?
How long did my
task take?
How many errors
occur?
What is its
memory footprint?
What other applications
are called?

3. The Scenario
3
Java Class
public Result getQuote(Subject subject) {
long start = System.currentTimeMillis();
validate(params);
Result result = quoteManager.getQuote(subject);
long duration = System.currentTimeMillis() - start;
Attributes attributes = create("quote-subject", subject);
Observer.record("getQuote", duration, attributes);
return result;
}
Capturing the start time
Application logic
Calculate the execution time
Record the execution time
with context information
How long did my application take to
provide a quote for a car?
…
getQuote {‘quote-subject’:‘car’} 252ms
getQuote {‘quote-subject’:‘bike’} 78ms
…
It takes 252 milliseconds.

4. Our Goal
4
Java Class
public Result getQuote(Subject subject) {
long start = System.currentTimeMillis();
validate(params);
Result result = quoteManager.getQuote(subject);
long duration = System.currentTimeMillis() - start;
Attributes attributes = create("quote-subject", subject);
Observer.record("getQuote", duration, attributes);
return result;
}
Do we have to do this for each
application?
But we need something consistent
across all applications!
How do I get this data into my
analysis tool?

5. OpenTelemetry
5

6. § Provides uniform standard for telemetry data.
§ Data can be correlated across systems.
§ Data can be exported to a variety of analysis tools.
What is OpenTelemetry?
6
“OpenTelemetry is a collection of tools, APIs, and SDKs. Use it to instrument,
generate, collect, and export telemetry data (metrics, logs, and traces) to help you
analyze your software’s performance and behavior.”
Source: https://opentelemetry.io/
DATA

7. How is it working?
7
OpenTelemetry API
Used to generate
telemetry data.
OpenTelemetry SDK
Implementation of the
API
OpenTelemetry
Extentions / Exporters
Additional API extensions, which
are not part of the core API.
Additional Components
E.g.: OTel-Collector, Jaeger, …

8. Manual Instrumentation via OTel API
8
Java Class
public Result getQuote(Subject subject) {
Span span = tracer.spanBuilder("getQuote").startSpan();
span.setAttribute("quote-subject", subject);
try (Scope scope = span.makeCurrent()) {
validate(params);
return quoteManager.getQuote(subject);
} finally {
span.end();
}
} Collected data is exported using
configured OpenTelemetry exporter
Application logic
Start span and assign data
End and record the span
Context Propagation
Changes need to be made to the source code.

9. § OpenTelemetry Java agent can be used to dynamically
injects bytecode to capture telemetry of popular libraries
and frameworks.
§ But: no ability to extract
specific data from
certain classes.
Auto-Instrumentation
9
Java Class
public Result getQuote(Subject subject) {
validate(params);
return quoteManager.getQuote(subject);
}
Java Class
public Result getQuote(Subject subject) {
Span span = tracer.spanBuilder("getQuote").startSpan();
span.setAttribute("quote-subject", subject);
try (Scope scope = span.makeCurrent()) {
validate(params);
return quoteManager.getQuote(subject);
} finally {
span.end();
}
}
Java agent automatically adds
OpenTelemetry instrumentation
to classes using Bytecode
Instrumentation (BCI)
Modified (instrumented) Class
Original Class
Java Agent
Class Transformation

10. inspectIT Ocelot
10

11. § A highly configurable Java agent for collecting telemetry and any
(business) data of Java applications.
§ Instrumentation (injected OpenTelemetry bytecode) can be
changed during application runtime.
§ Pre-configured to capture telemetry data of popular libraries and
frameworks.
§ Utility components that support the use of the agent (e.g.
configuration-server).
What is inspectIT Ocelot
11
Ocelot
Java Agent
Configuration
Server
Managing agents and
agent configurations
Fetch Configuration
http://inspectit.rocks/

12. § The configuration-server provides a simple UI for creating configurations
Basic Tracing
12
Easy selection of the
methods to be traced with
the ”class browser”.

13. Advanced YAML-based Configuration
13
§ Configuration determines what the agent does and consists of
three basic components:
Actions
What should the agent do
Scopes
Where should the agent do
something
Rules
Uses scopes and rules to instruct
the agent what to do
E.g.: Capture the first
method argument
E.g.: method “execute”
of class “Worker”
E.g.: trace the method “execute” of
class “worker”, capture the first
method argument of it and attach it
to the current span

14. Advanced YAML-based Configuration - Tracing
Predefined instruction set –
trace all methods in the scopes
Which scopes should be used
Which actions should be
executed when a method starts
Which data should be recorded
Scope
Action
Rule
Result (e.g.: Jaeger):

15. Advanced YAML-based Configuration - Metrics
Which scopes should be used
Which actions should be
executed when a method starts
Use the recorded data as a
metric’s value and export it
Scope
Action
Rule
Result (e.g. Prometheus):

16. Let's see it in action!
16

17. § Check out the talk “Open Source Application Performance Monitoring
in the Enterprise” by Sascha Brechmann of VHV Versicherung
Real-World Usage of inspectIT Ocelot
17

18. Novatec Consulting GmbH
Bertha-Benz-Platz 1
D-70771 Leinfelden-Echterdingen
T. +49 711 22040-700
info@novatec-gmbh.de
www.novatec-gmbh.de
18
Senior Consultant
Marius Oehler
E-Mail: marius.oehler@novatec-gmbh.de