Downloaded 32 times
Improving Network Application Performance using Load Aware Libeventdev
- 1. Intel Confidential –Internal Use Only Improving Network Application performance using Load Aware Libeventdev Sundar Vedantham - June 13, 2017
- 2. 2 Agenda • Multicore Applications • Why do we need Eventdev? • Eventdev Theory • Eventdev API & Application • Code Availability – Take Away
- 3. 3 Multicore Applications • Networkingapplications use one of two models: • Run-to-completion • Packet ordering for pinned flows • Flow atomicity is supported • Cache locality advantage • Load imbalance could be a con.
- 4. 4 Multicore Applications • Pipelineprocessing • Ensures packet ordering always • Flow atomicity is supported • Cache locality advantage • Core under utilization is a con • May not be flexible to handle different traffic types
- 5. 5 Why Eventdev? • EventScheduler addresses the shortcomings of both models. • Improves core utilization and better handles dynamic traffic mixes by scheduling packets to cores according to their load. • Simultaneously provides both atomicity and packet ordering • Hardware-based event schedulers can provide low-latency inter- core communication.
- 6. Why Eventdev? Eventdev Scheduler Rx Queue Tx Queue Corei Packet Rx Core m Packet Tx Core j Core k Firewall/Flow Classifier/Metering/ Routing Firewall/Flow Classifier/Metering/ Routing
- 7. 7 Current Status • Itis a DPDK Library for scheduling events • Initially proposed by Jerin Jacob (Cavium) • Worked with Intel + NXP to evolve API to current state • Easy setup of pipeline processing • Eventdev SQM PMDs does scheduling work • Development ongoing since Nov 2016 • http://dpdk.org/browse/next/dpdk-next-eventdev/ • Community feedback / patches included in the DPDK 17.05 release
- 8. 8 Eventdev Theory • Pipelines • Components • Device • Ports • Queues • Concepts • Scheduling types • Linking queues and ports
- 9. 9 Eventdev Theory -Pipelines • Pipeline made up of “stages” • Each “stage” is a stepping-stone to the next • One action performed at each stage • Eg: Flow Classification, Firewall, ACL • Load balancing • Worker logical cores (lcore) perform all stages of work • Ensures all lcores have equal load • No bottleneck due to single worker-lcore
- 10. 10 Eventdev Theory -Pipelines RX W1 TXAtomic #1 W.. WN W1 W.. WN Atomic #2 Single Link Wi - Worker core i
- 11. 11 Eventdev Theory -Components • Device • Instance of an eventdev PMD • Can be SW or HW • SW PMD is available • Ports and Queues are part of the device
- 12. 12 Eventdev Theory -Components NIC
- 13. 13 Eventdev Theory –Components – Event Port • Port • Event ports are logical ports associated with cores • Where CPU core and eventdev interact • Enqueue / Dequeue of events • 1 port used by 1 lcore • Events dequeued from port must be returned to it again • Eventdev tracks inflight events per port
- 14. 14 Eventdev Theory –Components – Event Port NIC
- 15. 15 Eventdev Theory –Components - Queue • Queue • Each queue is a stage of a pipeline • Destination for events • Has a Scheduling type • Atomic, Ordered, Parallel • Holds events until space available in linked port
- 16. 16 Eventdev Theory –Components - Queue NIC
- 17. 17 Eventdev Theory -Concepts • Scheduling Modes • Atomic - Flow ID used to ensure flow is not on > 1 lcore • Ordered - Events reordered into ingress order • Parallel - Events allowed to continue out-of-order
- 18. 18 Eventdev Theory -Concepts • Linking queues and ports • Ports need to be mapped to queues • Linking port to queue results in events from queue arriving at the port • Ports can be linked to multiple queues • Load balancing use-case
- 19. 19 Eventdev Theory –Concepts - Linking Link: - From Port #2 - To Queue #1 NIC
- 20. 20 Eventdev Theory -Concepts • Workers • Dequeue burst of events from port • For ( events ) • Read QUEUE_ID to get last stage • Process() • Write QUEUE_ID to set next stage • Enqueue burst back to port
- 21. 21 Eventdev API –rte_event • The rte_event struct • The struct that is used in eventdev • DPDK packets are struct rte_mbuf * • Eventdev events are struct rte_event (note no *) • Size is 16 bytes • 8 bytes metadata • 8 bytes payload Event Metadata Event Pointer
- 22. 22 Eventdev API –rte_event • The rte_event struct Metadata • Enough info for PMD to schedule event to Queue ID • Uses Flow ID, Operation type, Priority Flow ID Sub event type Even t type Sched type Queue ID Priority OP type
- 23. 23 Eventdev API -Configuration • API calling order to setup a Device • Device Configuration • Queue Setup • Port Setup • Port Link • Device Start
- 24. 24 Eventdev API –Runtime usage • RX lcore • NEW event ingress • Worker lcores • FORWARD existing event • TX lcore • RELEASE event on egress • Eventdev Pipeline Sample App Demonstrates usage
- 25. 25 Eventdev SW PMD- Overview • Software PMD for doing scheduling work • Implements the eventdev API • Merged into eventdev tree • Included in the DPDK 17.05 release
- 26.
- 27. 27 Eventdev SW PMD- Performance • Best Known Configuration (BKC) depends on • Number of stages • Cycles needed for work per stage • Number of workers • Adding cores is NOT always going to go faster! Performance comes from balance!
- 28. 28 Eventdev SW PMD– Performance - Prototyping • Use the eventdev_pipeline sample app • Configure to requirements • Number of stages • Cycles needed for work per stage • Number of Flows as expected • Choose realistic lcore count • Leave defaults for remaining values • Measure baseline • Use Monitoring tools • Cycles/pkt per port • Port burst distribution • “What % of bursts were 1-4 pkts?”
- 29. 29 Eventdev API –Resources • DPDK Eventdev Docs + Sample App • http://dpdk.org/doc/api/ • Download DPDK-master • Run make doc-guides-html • View Eventdev docs at • build/doc/html/guides/prog_guide/eventdev.html • Eventdev header file: lib/librte_eventdev/rte_eventdev.h • http://dpdk.org/doc/api/rte__eventdev_8h.html
- 30. 30 Eventdev Pipeline Documentation • Two docs patches • Sample App User Guide wget -O docs_sample_app.patch http://dpdk.org/dev/patchwork/patch/23800/raw/ • Eventdev Programmer Guide wget -O docs_eventdev_prog_guide.patch http://dpdk.org/dev/patchwork/patch/23801/raw/ • Build documentation • make doc-guides-html • Output files in .html format <dpdk>/build/doc/html/guides/ • http://dpdk.org/doc/guides/contributing/documentation.html#building-the-documentation