[Intel-wired-lan] zero-copy between interfaces

[Magnus Karlsson]

On Fri, Jan 17, 2020 at 6:16 PM Ryan Goodfellow <rgoodfel at isi.edu> wrote:
>
> On Fri, Jan 17, 2020 at 01:32:07PM +0100, Björn Töpel wrote:
> > On Mon, 13 Jan 2020 at 01:28, Ryan Goodfellow <rgoodfel at isi.edu> wrote:
> > >
> > [...]
> > >
> > > I could not get zero-copy to work with the i40e driver as it would crash. I've
> > > attached the corresponding traces from dmesg.
> >
> > Thanks Ryan! I had a look at the crash, and it's in the XDP setup:
> >
> > i40e_xdp_setup:
> > ...
> >  for (i = 0; i < vsi->num_queue_pairs; i++)
> >      WRITE_ONCE(vsi->rx_rings[i]->xdp_prog, vsi->xdp_prog);
> >
> > and the vsi->rx_ring[0] is NULL. This is clearly broken.
> >
> > It would help with more lines from your dmesg: the cut i40e log hints
> > that something is really broken:
> >
> > [  328.579154] i40e 0000:b7:00.2: failed to get tracking for 256
> > queues for VSI 0 err -12
> > [  328.579280] i40e 0000:b7:00.2: setup of MAIN VSI failed
> > [  328.579367] i40e 0000:b7:00.2: can't remove VEB 162 with 0 VSIs left
> >
> > Is it possible to dig out the complete log?
>
> Hi Björn,
>
> I've linked a full dmesg log from an XDP setup crash. Note that there are
> two i40e cards on this machine. The X710 (0000:65:00.0, 0000:65:00.1) works
> fine, the X722 (0000:b7:00.0, 0000:b7:00.1, 0000:b7:00.2, 0000:b7:00.3) is the
> one that is crashing on XDP setup.

Ryan,

I was wondering if you could run two small experiments since I cannot
reproduce this?

1: Run your program using the two ports on your X710 card. Does it
work? This is my setup and works for me.
2: On your Mellanox setup, insert a kick_tx() call for each of your
two sockets before the poll() call in your forward() function. Just to
see if it works when we explicitly wake up the driver.

Thanks: Magnus

> https://gitlab.com/mergetb/tech/network-emulation/kernel/snippets/1931080
>
> Some info that may be useful:
>
> ry at turbine:~$ sudo ethtool -i eno7
> driver: i40e
> version: 2.8.20-k
> firmware-version: 3.33 0x80001006 1.1747.0
> expansion-rom-version:
> bus-info: 0000:b7:00.2
> supports-statistics: yes
> supports-test: yes
> supports-eeprom-access: yes
> supports-register-dump: yes
> supports-priv-flags: yes
>
> The firmware version 3.33 was the latest I could find as of a few weeks ago.
>
> ry at turbine:~$ sudo lspci -vvv | grep 722
> b7:00.0 Ethernet controller: Intel Corporation Ethernet Connection X722 for 10GBASE-T (rev 04)
>         DeviceName: Intel LAN X722 #1
>         Subsystem: Super Micro Computer Inc Ethernet Connection X722 for 10GBASE-T
> b7:00.1 Ethernet controller: Intel Corporation Ethernet Connection X722 for 10GBASE-T (rev 04)
>         DeviceName: Intel LAN X722 #2
>         Subsystem: Super Micro Computer Inc Ethernet Connection X722 for 10GBASE-T
> b7:00.2 Ethernet controller: Intel Corporation Ethernet Connection X722 for 10GbE SFP+ (rev 04)
>         DeviceName: Intel LAN X722 #3
>         Subsystem: Super Micro Computer Inc Ethernet Connection X722 for 10GbE SFP+
> b7:00.3 Ethernet controller: Intel Corporation Ethernet Connection X722 for 10GbE SFP+ (rev 04)
>         DeviceName: Intel LAN X722 #4
>         Subsystem: Super Micro Computer Inc Ethernet Connection X722 for 10GbE SFP+
>
> ry at ryzen:~$ uname -a
> Linux ryzen 4.19.0-6-amd64 #1 SMP Debian 4.19.67-2+deb10u2 (2019-11-11) x86_64 GNU/Linux
>
> ry at turbine:~/kmoa/bpf-next$ git log -2
> commit 60d71397d27e7859fdaaaaab6594e4d977ae46e2 (HEAD -> master)
> Author: Ryan Goodfellow <rgoodfel at isi.edu>
> Date:   Wed Jan 15 16:54:39 2020 -0500
>
>     add xdpsock_multidev sample program
>
>     This is a simple program that uses AF_XDP sockets to forward packets
>     between two interfaces using a common memory region and no copying of
>     packets.
>
>     Signed-off-by: Ryan Goodfellow <rgoodfel at isi.edu>
>
> commit 9173cac3b64e6785dd604f5075e6035b045a0026 (origin/master, origin/HEAD)
> Author: Andrii Nakryiko <andriin at fb.com>
> Date:   Wed Jan 15 11:08:56 2020 -0800
>
>     libbpf: Support .text sub-calls relocations
>
>     The LLVM patch https://reviews.llvm.org/D72197 makes LLVM emit function call
>     relocations within the same section. This includes a default .text section,
>     which contains any BPF sub-programs. This wasn't the case before and so libbpf
>     was able to get a way with slightly simpler handling of subprogram call
>     relocations.
>
>     This patch adds support for .text section relocations. It needs to ensure
>     correct order of relocations, so does two passes:
>     - first, relocate .text instructions, if there are any relocations in it;
>     - then process all the other programs and copy over patched .text instructions
>     for all sub-program calls.
>
>     v1->v2:
>     - break early once .text program is processed.
>
>     Signed-off-by: Andrii Nakryiko <andriin at fb.com>
>     Signed-off-by: Alexei Starovoitov <ast at kernel.org>
>     Acked-by: Yonghong Song <yhs at fb.com>
>     Cc: Alexei Starovoitov <ast at kernel.org>
>     Link: https://lore.kernel.org/bpf/20200115190856.2391325-1-andriin@fb.com
>
> --
> ~ ry