[Intel-wired-lan] [PATCH net v2] ice: avoid bonding causing auxiliary plug/unplug under RTNL lock

[Linus Heckemann]

"Ertman, David M" <david.m.ertman at intel.com> writes:

>> -----Original Message-----
>> From: Linus Heckemann <git at sphalerite.org>
>> Sent: Thursday, February 16, 2023 9:24 AM
>> To: Ertman, David M <david.m.ertman at intel.com>; intel-wired-
>> lan at lists.osuosl.org
>> Cc: Jaroslav Pulchart <jaroslav.pulchart at gooddata.com>
>> Subject: Re: [Intel-wired-lan] [PATCH net v2] ice: avoid bonding causing
>> auxiliary plug/unplug under RTNL lock
>> 
>> Dave Ertman <david.m.ertman at intel.com> writes:
>> > RDMA is not supported in ice on a PF that has been added to a bonded
>> > interface. To enforce this, when an interface enters a bond, we unplug
>> > the auxiliary device that supports RDMA functionality.  This unplug
>> > currently happens in the context of handling the netdev bonding event.
>> > This event is sent to the ice driver under RTNL context.  This is causing
>> > a deadlock where the RDMA driver is waiting for the RTNL lock to complete
>> > the removal.
>> >
>> > Defer the unplugging/re-plugging of the auxiliary device to the service
>> > task so that it is not performed under the RTNL lock context.
>> >
>> > Reported-by: Jaroslav Pulchart <jaroslav.pulchart at gooddata.com>
>> > Link: https://lore.kernel.org/linux-rdma/68b14b11-d0c7-65c9-4eeb-
>> 0487c95e395d at leemhuis.info/
>> > Fixes: 5cb1ebdbc434 ("ice: Fix race condition during interface enslave")
>> > Fixes: 425c9bd06b7a ("RDMA/irdma: Report the correct link speed")
>> > Signed-off-by: Dave Ertman <david.m.ertman at intel.com>
>> > ---
>> > Changes since v1:
>> > Reversed order of bit processing in ice_service_task for PLUG/UNPLUG
>> 
>> Hi Dave,
>> 
>> Thanks for your continued work on this! We've tested this on a system
>> affected by the original issue (with 8086:1593 cards) and, unlike v1 of
>> the patch, it appears not to resolve it:
>
> Hi Linus,
>
> This error confuses me.  The only difference between v1 and v2 of this patch
> is the order in which we process state bits in the service task thread.  They are
> still being processed outside of RTNL context.
>
> Can you provide the steps you used to reproduce this issue? 
>
> Thanks,
> DaveE

Hi Dave,

It confuses me as well!

Like before, this was reproduced by booting a system configured to bond
the interfaces provided by two of the cards (using systemd-networkd,
relevant config below). The failure occurred less frequently than prior
to applying the patch, but still enough to be quite an annoyance!

According to the provider, the machine's card was on an older firmware
(3.00 0x8000893f 20.5.13), and upgrading to the latest available version
resolved this issue for our purposes. Nevertheless, I think the kernel
shouldn't be deadlock on the RTNL lock regardless of which firmware
version is running. If there's any more information that would be
helpful for debugging, let us know -- though we can't get at machines
running the old firmware trivially, so it's hard for us to reproduce at
this point.

As mentioned, upgrading the firmware has resolved the problem for us,
though it certainly feels unsatisfying to leave the bug there. I have no
strong opinion on whether the patch should be included as is
anyway. Maybe the firmware version info is enough to help you reproduce
the problem?

Linus


________
/etc/systemd/network/10-bond0.netdev:
[NetDev]
Kind=bond
Name=bond0

[Bond]
DownDelaySec=0.200000
LACPTransmitRate=fast
MIIMonitorSec=0.100000
Mode=802.3ad
TransmitHashPolicy=layer3+4
UpDelaySec=0.200000

________
/etc/systemd/network/40-bond0.network:
[Match]
Name=bond0

[Link]
#MACAddress=<omitted>
RequiredForOnline=carrier

[Network]
LinkLocalAddressing=no

# some Address and Route sections omitted
________
/etc/systemd/network/30-eno12419.network:
[Match]
Name=eno12419

[Network]
Bond=bond0

________
/etc/systemd/network/30-eno12399.network:
[Match]
Name=eno12399

[Network]
Bond=bond0