Root cause analysis with e bpf & python

•Transferir como PPTX, PDF•

2 gostaram•711 visualizações

Pavel Rogovoy

This slide is an introduction to eBPF. This is a slide of a talk I gave on Pycon 19 Israel

Tecnologia

Root Cause Analysis with
eBPF & Python
Pavel Rogovoy
PyCon IL 2019

Pavel Rogovoy
Performance & Cloud Engineer @

$from bcc import BPF sensor=''int kprobe__sys_clone(void *ctx) { bpf_trace_printk("Hello, World!n"); return 0; }' BPF(text=sensor).trace_print()$

Applications
System Libraries
System Call Interface
Sockets
TCP/UDP
IP
Ethernet
VFS
File Systems
Volume Manager
Block Device
Interface
Device Drivers
Scheduler
Virtual
Memory
syscount

summarize aggregated latencies of syscalls
./syscount.py -L -p `pgrep -nx my_process`
Tracing syscalls, printing top 10... Ctrl+C to quit.
[09:41:32]
SYSCALL COUNT TIME (us)
nanosleep 9 7020894.226
futex 6 220.009
write 4 90.292
clone 2 70.935
Detaching...

top 3 failed syscalls
/usr/share/bcc/tools/syscount -x -T 3
Tracing failed syscalls, printing top 3... Ctrl+C to quit.
[13:27:46]
SYSCALL COUNT
recvmsg 6152
futex 346
read 32
^C
Detaching...

finish_task_switch
Previous task Next task
Stack
Trace TSTS

@
CPYTHON 3.6+
USDT: User Statically-Defined Tracing

System Libraries
System Call Interface
Sockets
TCP/UDP
IP
Ethernet
VFS
File Systems
Volume Manager
Block Device
Interface
Device Drivers
Scheduler
Virtual
Memory
Trace
PYTHON
CPYTHON 3.6+

Listing tracepoints of CPython interpreter
tplist.py -l ./python
./python python:line
./python python:function__entry
./python python:function__return
./python python:import__find__load__start
./python python:import__find__load__done
./python python:gc__start
./python python:gc__done

import time
def my_func():
print('Hello')
while True:
time.sleep(3)
my_func()

Things You Must Be Aware
Of
Not everything is perfect...

● Maximum of 4096 ebpf assembly
instructions
● No loops allowed
● Poor ecosystem

Brendan Gregg
Performance Engineer
@ Netflix
Big contributor to
BCC, eBPF &
FlameGraphs projects

THANKS
Email: pavelr@final.co.il
LinkedIn: progovoy
Twitter: @p_rogovoy

Mais conteúdo relacionado

Mais procurados

Scriptfauzasmg

Tomas Hlavacek - IP fragmentation attack on DNSDefconRussia

Ceph Day Shanghai - Ceph Performance Tools Ceph Community

Gameboy emulator in rust and web assemblyYodalee

Ntp cheat sheetcsystemltd

Computer Performance Microscopy with SHIMhiyangxi

Scaling FastAGI Applications with GoDigium

Berkeley Packet FiltersKernel TLV

Introduction to nand2 tetrisYodalee

Using zone.jsStanda Opichal

Staring into the eBPF AbyssSasha Goldshtein

multi-line record grepRyoichi KATO

Device-specific Clang Tooling for Embedded SystemsemBO_Conference

Defcon 2011 network forensics 解题记录insight-labs

Sup introchase pettet

Defeating the entropy downgrade attackSeth Wahle

serverstatsBen De Koster

Performance testing of microservices in ActionAlexander Kachur

Building a DSL with GraalVM (VoxxedDays Luxembourg)Maarten Mulders

异步io框架的实现rfyiamcool

Mais procurados (20)

Script

Tomas Hlavacek - IP fragmentation attack on DNS

Ceph Day Shanghai - Ceph Performance Tools

Gameboy emulator in rust and web assembly

Ntp cheat sheet

Computer Performance Microscopy with SHIM

Scaling FastAGI Applications with Go

Berkeley Packet Filters

Introduction to nand2 tetris

Using zone.js

Staring into the eBPF Abyss

multi-line record grep

Device-specific Clang Tooling for Embedded Systems

Defcon 2011 network forensics 解题记录

Sup intro

Defeating the entropy downgrade attack

serverstats

Performance testing of microservices in Action

Building a DSL with GraalVM (VoxxedDays Luxembourg)

异步io框架的实现

Semelhante a Root cause analysis with e bpf & python

UM2019 Extended BPF: A New Type of SoftwareBrendan Gregg

Security Monitoring with eBPFAlex Maestretti

ebpf and IO Visor: The What, how, and what next!Affan Syed

DCSF 19 eBPF SuperpowersDocker, Inc.

CorePy High-Productivity CellB.E. ProgrammingSlide_N

Network DriversAnil Kumar Pugalia

emips_overview_apr08Neil Pittman

Efficient System Monitoring in Cloud Native EnvironmentsGergely Szabó

Please help with the below 3 questions, the python script is at the.pdfsupport58

05 module managing your network enviornmentAsif

Velocity 2017 Performance analysis superpowers with Linux eBPFBrendan Gregg

#Include os - From bootloader to REST API with the new C++IncludeOS

eBPF Tooling and Debugging InfrastructureNetronome

USENIX ATC 2017 Performance Superpowers with Enhanced BPFBrendan Gregg

UCX-Python - A Flexible Communication Library for Python ApplicationsMatthew Rocklin

Track c-High speed transaction-based hw-sw coverification -evechiportal

Performance Analysis Tools for Linux Kernellcplcp1

Global Interpreter Lock: Episode I - Break the SealTzung-Bi Shih

Buffer overflow – Smashing The StackTomer Zait

20081114 Friday Food iLabt Bart Jorisimec.archive

Semelhante a Root cause analysis with e bpf & python (20)

UM2019 Extended BPF: A New Type of Software

Security Monitoring with eBPF

ebpf and IO Visor: The What, how, and what next!

DCSF 19 eBPF Superpowers

CorePy High-Productivity CellB.E. Programming

Network Drivers

emips_overview_apr08

Efficient System Monitoring in Cloud Native Environments

Please help with the below 3 questions, the python script is at the.pdf

05 module managing your network enviornment

Velocity 2017 Performance analysis superpowers with Linux eBPF

#Include os - From bootloader to REST API with the new C++

eBPF Tooling and Debugging Infrastructure

USENIX ATC 2017 Performance Superpowers with Enhanced BPF

UCX-Python - A Flexible Communication Library for Python Applications

Track c-High speed transaction-based hw-sw coverification -eve

Performance Analysis Tools for Linux Kernel

Global Interpreter Lock: Episode I - Break the Seal

Buffer overflow – Smashing The Stack

20081114 Friday Food iLabt Bart Joris

Último

Vector Search -An Introduction in Oracle Database 23ai.pptxRemote DBA Services

Why Teams call analytics are critical to your entire businesspanagenda

Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobeapidays

[BuildWithAI] Introduction to Gemini.pdfSandro Moreira

Artificial Intelligence Chap.5 : UncertaintyKhushali Kathiriya

Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024Victor Rentea

Six Myths about Ontologies: The Basics of Formal Ontologyjohnbeverley2021

Elevate Developer Efficiency & build GenAI Application with Amazon QBhuvaneswari Subramani

Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...apidays

Platformless Horizons for Digital AdaptabilityWSO2

DEV meet-up UiPath Document Understanding May 7 2024 AmsterdamUiPathCommunity

presentation ICT roal in 21st century educationjfdjdjcjdnsjd

Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...Jeffrey Haguewood

Finding Java's Hidden Performance Traps @ DevoxxUK 2024Victor Rentea

Mcleodganj Call Girls 🥰 8617370543 Service Offer VIP Hot ModelDeepika Singh

"I see eyes in my soup": How Delivery Hero implemented the safety system for ...Zilliz

EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWERMadyBayot

Architecting Cloud Native ApplicationsWSO2

Strategies for Landing an Oracle DBA Job as a FresherRemote DBA Services

DBX First Quarter 2024 Investor PresentationDropbox

Root cause analysis with e bpf & python

1. Root Cause Analysis with eBPF & Python Pavel Rogovoy PyCon IL 2019

4. eBPF

5. Waiting on a lock 20 seconds sleep

6. Pavel Rogovoy Performance & Cloud Engineer @

7. LogicInput Output

9. Our App

10.

11. Our App extended Berkley Packet Filter

12. Our App

13. A real life example

14.

15. from bcc import BPF sensor=''int kprobe__sys_clone(void *ctx) { bpf_trace_printk("Hello, World!n"); return 0; }' BPF(text=sensor).trace_print()

16.

17.

18. Applications System Libraries System Call Interface Sockets TCP/UDP IP Ethernet VFS File Systems Volume Manager Block Device Interface Device Drivers Scheduler Virtual Memory syscount

19. summarize aggregated latencies of syscalls ./syscount.py -L -p `pgrep -nx my_process` Tracing syscalls, printing top 10... Ctrl+C to quit. [09:41:32] SYSCALL COUNT TIME (us) nanosleep 9 7020894.226 futex 6 220.009 write 4 90.292 clone 2 70.935 Detaching...

20. top 3 failed syscalls /usr/share/bcc/tools/syscount -x -T 3 Tracing failed syscalls, printing top 3... Ctrl+C to quit. [13:27:46] SYSCALL COUNT recvmsg 6152 futex 346 read 32 ^C Detaching...

21. Applications System Libraries System Call Interface Sockets TCP/UDP IP Ethernet VFS File Systems Volume Manager Block Device Interface Device Drivers Scheduler Virtual Memory offcputime

22. What is off-cpu time?

23.

24. offcputime

25. finish_task_switch Previous task Next task Stack Trace TSTS

26. @ CPYTHON 3.6+ USDT: User Statically-Defined Tracing

27. System Libraries System Call Interface Sockets TCP/UDP IP Ethernet VFS File Systems Volume Manager Block Device Interface Device Drivers Scheduler Virtual Memory Trace PYTHON CPYTHON 3.6+

28. Listing tracepoints of CPython interpreter tplist.py -l ./python ./python python:line ./python python:function__entry ./python python:function__return ./python python:import__find__load__start ./python python:import__find__load__done ./python python:gc__start ./python python:gc__done

29. import time def my_func(): print('Hello') while True: time.sleep(3) my_func()

30.

31. Things You Must Be Aware Of Not everything is perfect...

32. ● Maximum of 4096 ebpf assembly instructions ● No loops allowed ● Poor ecosystem

33. Brendan Gregg Performance Engineer @ Netflix Big contributor to BCC, eBPF & FlameGraphs projects

34. QUESTIONS?

35. https://www.final.co.il

36. THANKS Email: pavelr@final.co.il LinkedIn: progovoy Twitter: @p_rogovoy

Notas do Editor

these are the ones that might be worth investigating with follow-up tools like opensnoop, execsnoop, or trace
It works by taking samples of stack traces at timed intervals, and frequency counting them in kernel context for efficiency
For further read. Refer to his blog

Root cause analysis with e bpf & python

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Semelhante a Root cause analysis with e bpf & python

Semelhante a Root cause analysis with e bpf & python (20)

Último

Último (20)

Root cause analysis with e bpf & python

Notas do Editor

Root cause analysis with e bpf &amp; python

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Semelhante a Root cause analysis with e bpf &amp; python

Semelhante a Root cause analysis with e bpf &amp; python (20)

Último

Último (20)

Root cause analysis with e bpf &amp; python

Notas do Editor

Root cause analysis with e bpf & python

Semelhante a Root cause analysis with e bpf & python

Semelhante a Root cause analysis with e bpf & python (20)

Root cause analysis with e bpf & python