Compared to load unaware packet distribution mechanisms often used in the run to completion model, an event scheduler improves core utilization and better handles dynamic traffic mixes by scheduling packets to cores according to their load. It simultaneously provides both atomicity and packet ordering. Hardware-based event schedulers can also provide low-latency inter-core communication. The libeventdev library from Data Plane Development Kit (DPDK) helps developers leverage the event scheduler model.
About the presenter: Sundar Vedantham, Intel, is a Senior Technical Manager working in the Data Center Group in Allentown, PA. His research interests include network traffic and congestion management, high-speed networking, and theoretical computer models, areas in which he holds patents and has published papers, book chapter & articles. He received his Ph.D. in Computer Science in 1997 from Louisiana State University. He enjoys writing articles in English and Tamil to help improve public understanding of technical details behind the fields he has worked on and to attract young students to get into STEM fields.
3. 3
Multicore Applications
• Networking applications use one of
two models:
• Run-to-completion
• Packet ordering for pinned flows
• Flow atomicity is supported
• Cache locality advantage
• Load imbalance could be a con.
4. 4
Multicore Applications
• Pipeline processing
• Ensures packet ordering always
• Flow atomicity is supported
• Cache locality advantage
• Core under utilization is a con
• May not be flexible to handle different
traffic types
5. 5
Why Eventdev?
• Event Scheduler addresses the shortcomings of both models.
• Improves core utilization and better handles dynamic traffic mixes
by scheduling packets to cores according to their load.
• Simultaneously provides both atomicity and packet ordering
• Hardware-based event schedulers can provide low-latency inter-
core communication.
7. 7
Current Status
• It is a DPDK Library for scheduling events
• Initially proposed by Jerin Jacob (Cavium)
• Worked with Intel + NXP to evolve API to current state
• Easy setup of pipeline processing
• Eventdev SQM PMDs does scheduling work
• Development ongoing since Nov 2016
• http://dpdk.org/browse/next/dpdk-next-eventdev/
• Community feedback / patches included in the DPDK 17.05 release
9. 9
Eventdev Theory - Pipelines
• Pipeline made up of “stages”
• Each “stage” is a stepping-stone to the next
• One action performed at each stage
• Eg: Flow Classification, Firewall, ACL
• Load balancing
• Worker logical cores (lcore) perform all stages of work
• Ensures all lcores have equal load
• No bottleneck due to single worker-lcore
10. 10
Eventdev Theory - Pipelines
RX
W1
TXAtomic #1 W..
WN
W1
W..
WN
Atomic #2 Single Link
Wi - Worker core i
11. 11
Eventdev Theory - Components
• Device
• Instance of an eventdev PMD
• Can be SW or HW
• SW PMD is available
• Ports and Queues are part of the device
13. 13
Eventdev Theory – Components – Event Port
• Port
• Event ports are logical ports associated with cores
• Where CPU core and eventdev interact
• Enqueue / Dequeue of events
• 1 port used by 1 lcore
• Events dequeued from port must be returned to it again
• Eventdev tracks inflight events per port
15. 15
Eventdev Theory – Components - Queue
• Queue
• Each queue is a stage of a pipeline
• Destination for events
• Has a Scheduling type
• Atomic, Ordered, Parallel
• Holds events until space available in linked port
17. 17
Eventdev Theory - Concepts
• Scheduling Modes
• Atomic - Flow ID used to ensure flow is not on > 1 lcore
• Ordered - Events reordered into ingress order
• Parallel - Events allowed to continue out-of-order
18. 18
Eventdev Theory - Concepts
• Linking queues and ports
• Ports need to be mapped to queues
• Linking port to queue results in events from queue arriving at the port
• Ports can be linked to multiple queues
• Load balancing use-case
20. 20
Eventdev Theory - Concepts
• Workers
• Dequeue burst of events from port
• For ( events )
• Read QUEUE_ID to get last stage
• Process()
• Write QUEUE_ID to set next stage
• Enqueue burst back to port
21. 21
Eventdev API – rte_event
• The rte_event struct
• The struct that is used in eventdev
• DPDK packets are struct rte_mbuf *
• Eventdev events are struct rte_event (note no *)
• Size is 16 bytes
• 8 bytes metadata
• 8 bytes payload
Event Metadata Event Pointer
22. 22
Eventdev API – rte_event
• The rte_event struct Metadata
• Enough info for PMD to schedule event to Queue ID
• Uses Flow ID, Operation type, Priority
Flow ID
Sub event
type
Even
t type
Sched
type
Queue ID Priority
OP
type
23. 23
Eventdev API - Configuration
• API calling order to setup a Device
• Device Configuration
• Queue Setup
• Port Setup
• Port Link
• Device Start
25. 25
Eventdev SW PMD - Overview
• Software PMD for doing scheduling work
• Implements the eventdev API
• Merged into eventdev tree
• Included in the DPDK 17.05 release
27. 27
Eventdev SW PMD - Performance
• Best Known Configuration (BKC) depends on
• Number of stages
• Cycles needed for work per stage
• Number of workers
• Adding cores is NOT always going to go faster!
Performance comes from balance!
28. 28
Eventdev SW PMD – Performance - Prototyping
• Use the eventdev_pipeline sample app
• Configure to requirements
• Number of stages
• Cycles needed for work per stage
• Number of Flows as expected
• Choose realistic lcore count
• Leave defaults for remaining values
• Measure baseline
• Use Monitoring tools
• Cycles/pkt per port
• Port burst distribution
• “What % of bursts were 1-4 pkts?”
29. 29
Eventdev API – Resources
• DPDK Eventdev Docs + Sample App
• http://dpdk.org/doc/api/
• Download DPDK-master
• Run make doc-guides-html
• View Eventdev docs at
• build/doc/html/guides/prog_guide/eventdev.html
• Eventdev header file: lib/librte_eventdev/rte_eventdev.h
• http://dpdk.org/doc/api/rte__eventdev_8h.html
30. 30
Eventdev Pipeline Documentation
• Two docs patches
• Sample App User Guide
wget -O docs_sample_app.patch http://dpdk.org/dev/patchwork/patch/23800/raw/
• Eventdev Programmer Guide
wget -O docs_eventdev_prog_guide.patch http://dpdk.org/dev/patchwork/patch/23801/raw/
• Build documentation
• make doc-guides-html
• Output files in .html format <dpdk>/build/doc/html/guides/
• http://dpdk.org/doc/guides/contributing/documentation.html#building-the-documentation
