[Intel-wired-lan] i40e/iavf bandwidth spikes to 500Gbps & returning IAVF_ERR_PARAM

[Kubalewski, Arkadiusz]

Good day JD!

The message you have provided: 
iavf 0000:00:05.0: PF returned error -5 (IAVF_ERR_PARAM) to our request 15
indicates that there was a failure while communicating with parent PF port.
In fact, failed command was trying to get statistics of a VF port from parent PF.
Command have failed, so the stats returned shall be equal to 0.
Probably Prometheus considered them valid and that is why it shows "impossible" stats.

About the command failure...
It is related to old issue, where PF and VF are out of sync on their communication channel.

The issue was already fixed in 2.12.6, please use the latest driver to get rid of the issue 

Hope this helps 😊

Best Regards,

Arkadiusz Kubalewski
Software Engineer
CG EPG SW ITP Linux base driver

-----Original Message-----
From: Intel-wired-lan <intel-wired-lan-bounces at osuosl.org> On Behalf Of JD
Sent: piątek, 30 października 2020 18:33
To: intel-wired-lan <intel-wired-lan at lists.osuosl.org>
Subject: [Intel-wired-lan] i40e/iavf bandwidth spikes to 500Gbps & returning IAVF_ERR_PARAM

Hello,

Over the past month I've observed some KVM servers exhibiting extremely high bandwidth activity (500gbit, which is impossible).
Please see the attached graphs from Prometheus. It starts small then progressively gets worse over time.

During these spikes, I see the following appear on the KVM guest about once every minute.
iavf 0000:00:05.0: PF returned error -5 (IAVF_ERR_PARAM) to our request 15

And on the KVM parent, I see this about once every minute:
i40e 0000:81:00.0: VF 20 failed opcode 15, retval: -5

This doesn't seem to happen with any obvious cause, the only other thing I see in dmesg that may be related is the following (which is repeated a few times over several hours, but not at the same rate)

vfio-pci 0000:81:0c.3: Event logged [IO_PAGE_FAULT domain=0x0000
address=0xfffffffdf8040000 flags=0x0008]


The KVM guest is running the iavf driver:
driver: iavf
version: 3.9.3

The KVM parent is running the i40e driver:
driver: i40e
version: 2.11.21

I'm running 2 of the following NIC's on the KVM parent in a bonded setup (mode 4, hash policy 3+4, IEEE 802.3ad dynamic link aggregation)
:
Intel Corporation Ethernet Controller XXV710 for 25GbE SFP28 (rev 02)

Both the parent/guest are running the same kernel version of 4.19.107

I'm not sure whether this is a NIC issue, driver issue, or something else. I'm happy to provide any more information about the system on request if it is relevant (qemu versions, mobo/cpu/ram).  I've observed this issue on 3 different KVM parents/guests in different regions.

If this has already been fixed or is a known issue, then I apologize, but I could not find anything by searching the mailing list w/ my codes from dmesg.

If anyone can provide any information about this or any pointers on this or how to narrow the issue down, I'd greatly appreciate it.

Thank you.
---------------------------------------------------------------------
Intel Technology Poland sp. z o.o.
ul. Sowackiego 173 | 80-298 Gdask | Sd Rejonowy Gdask Pnoc | VII Wydzia Gospodarczy Krajowego Rejestru Sdowego - KRS 101882 | NIP 957-07-52-316 | Kapita zakadowy 200.000 PLN.
Ta wiadomo wraz z zacznikami jest przeznaczona dla okrelonego adresata i moe zawiera informacje poufne. W razie przypadkowego otrzymania tej wiadomoci, prosimy o powiadomienie nadawcy oraz trwae jej usunicie; jakiekolwiek przegldanie lub rozpowszechnianie jest zabronione.
This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). If you are not the intended recipient, please contact the sender and delete all copies; any review or distribution by others is strictly prohibited.