Running DPDK Forwarding Applications With Pktgen-DPDK

This article is co-authored by Mishal Shah

As part of the evaluation stage of our bachelor thesis, we set up a testbed for running forwarding applications in DPDK and with Pktgen-DPDK as the traffic generator. In this blog, we aim to cover

• how to run Pktgen-DPDK using Lua scripts in some complex scenarios
• how to run L2fwd, L3fwd (with exact match and longest prefix match) and L3fwd power DPDK applications.

We did not find a lot of resources for setting up these tests and with this blog, we aim to bridge the gap.

DPDK and Pktgen-DPDK

The Data Plane Development Kit is an open-source software project managed by the Linux Foundation. It provides a set of data plane libraries and network interface controller polling-mode drivers running in userspace. This way, the NIC is directly accessible by the DPDK application. The advantage of using DPDK is that you can utilize the link speed completely in the case of high-speed links whereas, in the standard packet processing, the link may be underutilized. Thus, high performance can be achieved with DPDK.

Pktgen-DPDK is a software-based traffic generator powered by DPDK. Traffic generators are often used to simulate various situations and test the performance of the application.

Test Setup

All the tests were done with two systems, one as the System Under Test which is a bare metal server, and the other one as a generator which is a VM. The SUT has two NUMA sockets where each socket has two 14-core Intel Xeon Gold 5120 2.20 GHz processors, which has four Intel I350 10 Gigabit NICs and four Ethernet Controller X710/X557-AT 10GBASE-T NICs. The generator has two 12-core Intel Xeon Gold 5120 2.20 GHz processors, 94GB memory, and one Intel 82540EM Gigabit NIC and Ethernet Controller X710/X557-AT 10GBASE-T NICs. Each system runs Ubuntu 20.04 LTS.

Installation of both DPDK and Pktgen-DPDK was done by compiling from source. igb_uio kernel driver was used and 2048 huge pages were allotted. Here is a useful tutorial on installing and setting up DPDK and Pktgen-DPDK on your system. Tapped interfaces can be used in case your system does not have enough NICs. Here’s a diagrammatic representation of the setup.

Testbed

Lua Scripts with Pktgen-DPDK

We used Lua scripts to automate the process of setting parameters and configuring flows in Pktgen-DPDK. Pktgen-DPDK can be run on the generator using the command shown below. It is run in promiscuous mode with the first four cores (0,1,2,3), and 4 memory channels. The Lua script for each application is mentioned using the -f option. The port mapping to cores is done as:

• core 1 handles port 0, 1 RX/TX
• core 2 handles port 2, 3 RX/TX.

Note: The core 0 is used for keyboard, timer, and screen output and hence shouldn’t be used for port mapping. pktgen-dpdk with Lua script should be run from the pktgen directory only, otherwise, it gives an error.

sudo ./app/x86_64-native-linux-gcc/pktgen -c 0xf -n 4 — -P -m 1.[0–1] -m 2.[2–3] -f /path/to/script.lua

Situation 1: Flow for a Fixed Time

We were required to run tests where the packets were sent for a fixed time period. Since pktgen by default doesn’t support timers, we used os functions for the same.

where send_for_secs is the time for which we want to send.

Note: sleep(send_for_secs) was not used because it would block Pktgen’s output refreshing. So continuous check after a second and checking until the difference reaches the send_for_secs is done here.

Situation 2: Different Packet Size and Rate

We also wanted to test with different packet sizes and rates (rate in pktgen-dpdk is to be given as a percentage of the line rate). Since changing the packet size or rate in the Lua script after every run is tedious, we ask for user input for the packet size and rate at runtime using the script below.

Forwarding Applications in DPDK

We mention the steps to run the forwarding application and a corresponding Lua script to set relevant parameters. The command to run Pktgen-DPDK with the Lua script has been given in the previous section.

l2fwd

Layer 2 forwarding (switching) does not need IP addresses to work. Thus, we set only the MAC address in the packets sent from pktgen-DPDK.

• Start the l2fwd application on the SUT: The l2fwd example can be run with the following command on the SUT. This command starts l2fwd on four cores and with four ports. The SUT has two NUMA sockets but we use only one. The even-numbered cores are on socket 0 and we use the first four cores on it, i.e, cores 0,2,4,6. Thus the core mask is given as 0x55. The port mask is 0xf as we use four ports.

sudo ./${RTE_SDK}/examples/l2fwd/${RTE_TARGET}/app/l2fwd -c 0x55 -n 4 -- -p 0xf

• Start pktgen-dpdk on the generator with Lua script: Get the MAC address of the ports by starting the l2fwd application on the SUT. Set them as the destination MAC address for the ports 0 and 2 in pktgen through the Lua script shown below.

l3fwd

A layer 3 forwarder matches the destination IP of an incoming packet to the entries stored in its forwarding table and updates the layer 2 header before sending the packet to the next-hop.

• Longest Prefix Match: For the LPM method, an LPM object is used to emulate the forwarding stage for IPv4 packets. The LPM object is used as the routing table to identify the next hop for each input packet at runtime. The example can be run with the following command on the SUT.

sudo ./${RTE_SDK}/examples/l3fwd/${RTE_TARGET}/app/l3fwd -c 0x55 -n 4 — -P -p 0xf — config=(0,0,0),(1,0,2),(2,0,4),(3,0,6)

The IP addresses obtained from the source code of l3fwd are set as shown below.

• Exact Match: In the exact match lookup method, the forwarding is based on a hash for which a hash object is used to emulate the flow classification stage. The example can be run with the following command on the SUT. The -E parameter selects the exact match method.

sudo ./${RTE_SDK}/examples/l3fwd/${RTE_TARGET}/app/l3fwd -c 0x55 -n 4 -- -P -E --parse-ptype -p 0xf --config=(0,0,0),(1,0,2),(2,0,4),(3,0,6)

From the code for the l3fwd application, the exact match works only with TCP packets and thus we set the protocol to TCP in the packets generated from pktgen. The exact match is based on the combination of IP addresses and port numbers, as seen in the routing table in the code. We set the IP addresses and destination and source ports as shown below obtained from the source code.

l3fwd-power

The L3 Forwarding with Power Management application is an example of power-aware packet processing using the DPDK. The example can be run with the following command on the SUT.

sudo ./${RTE_SDK}/examples/l3fwd-power/${RTE_TARGET}/app/l3fwd-power -c 0x55 -n 4 -- -P -p 0xf --config=(0,0,0),(1,0,2),(2,0,4),(3,0,6)

The flow configured on pktgen with IP addresses can be done as below. The IP addresses are obtained from the source code of l3fwd-power. It runs the longest prefix match by default.

Conclusions

In this article, we looked at running various forwarding applications of DPDK with Pktgen-DPDKas the traffic generator with four ports. The main challenge we faced while trying to run these applications was a lack of proper documentation in setting up various cases with pktgen and making sure the traffic forwards on the application and reaches back to the other port. If you do not configure the IP/MAC address properly, it is possible that the packets do not reach the other port but come back on the same port.

Both DPDK and Pktgen-DPDK require a bit of work to set up and understand how the applications work, and undoubtedly there is a good learning curve. The DPDK documentation provides enough information about running the example applications and so we have not covered that in detail.

The complete Lua scripts can be found here.

Acknowledgement

We thank our project guide, Dr Mohit P. Tahiliani, and mentor Leslie Monis for their continuous guidance throughout the course of the project and our project teammate Pavan Vachhani for his valuable contribution to the final project.

References

• DPDK Sample Application Guide: l2fwd, l3fwd, l3fwd-power
• Pktgen-DPDK
• DPDK Source Code: https://github.com/DPDK/dpdk