[Intel-wired-lan] [PATCH net v2] ice: avoid bonding causing auxiliary plug/unplug under RTNL lock

[Tony Nguyen]

On 3/2/2023 4:38 AM, Linus Heckemann wrote:
> "Ertman, David M" <david.m.ertman at intel.com> writes:
> 
>>> -----Original Message-----
>>> From: Linus Heckemann <git at sphalerite.org>
>>> Sent: Thursday, February 16, 2023 9:24 AM
>>> To: Ertman, David M <david.m.ertman at intel.com>; intel-wired-
>>> lan at lists.osuosl.org
>>> Cc: Jaroslav Pulchart <jaroslav.pulchart at gooddata.com>
>>> Subject: Re: [Intel-wired-lan] [PATCH net v2] ice: avoid bonding causing
>>> auxiliary plug/unplug under RTNL lock
>>>
>>> Dave Ertman <david.m.ertman at intel.com> writes:
>>>> RDMA is not supported in ice on a PF that has been added to a bonded
>>>> interface. To enforce this, when an interface enters a bond, we unplug
>>>> the auxiliary device that supports RDMA functionality.  This unplug
>>>> currently happens in the context of handling the netdev bonding event.
>>>> This event is sent to the ice driver under RTNL context.  This is causing
>>>> a deadlock where the RDMA driver is waiting for the RTNL lock to complete
>>>> the removal.
>>>>
>>>> Defer the unplugging/re-plugging of the auxiliary device to the service
>>>> task so that it is not performed under the RTNL lock context.
>>>>
>>>> Reported-by: Jaroslav Pulchart <jaroslav.pulchart at gooddata.com>
>>>> Link: https://lore.kernel.org/linux-rdma/68b14b11-d0c7-65c9-4eeb-
>>> 0487c95e395d at leemhuis.info/
>>>> Fixes: 5cb1ebdbc434 ("ice: Fix race condition during interface enslave")
>>>> Fixes: 425c9bd06b7a ("RDMA/irdma: Report the correct link speed")
>>>> Signed-off-by: Dave Ertman <david.m.ertman at intel.com>
>>>> ---
>>>> Changes since v1:
>>>> Reversed order of bit processing in ice_service_task for PLUG/UNPLUG
>>>
>>> Hi Dave,
>>>
>>> Thanks for your continued work on this! We've tested this on a system
>>> affected by the original issue (with 8086:1593 cards) and, unlike v1 of
>>> the patch, it appears not to resolve it:
>>
>> Hi Linus,
>>
>> This error confuses me.  The only difference between v1 and v2 of this patch
>> is the order in which we process state bits in the service task thread.  They are
>> still being processed outside of RTNL context.
>>
>> Can you provide the steps you used to reproduce this issue?
>>
>> Thanks,
>> DaveE
> 
> Hi Dave,
> 
> It confuses me as well!
> 
> Like before, this was reproduced by booting a system configured to bond
> the interfaces provided by two of the cards (using systemd-networkd,
> relevant config below). The failure occurred less frequently than prior
> to applying the patch, but still enough to be quite an annoyance!
> 
> According to the provider, the machine's card was on an older firmware
> (3.00 0x8000893f 20.5.13), and upgrading to the latest available version
> resolved this issue for our purposes. Nevertheless, I think the kernel
> shouldn't be deadlock on the RTNL lock regardless of which firmware
> version is running. If there's any more information that would be
> helpful for debugging, let us know -- though we can't get at machines
> running the old firmware trivially, so it's hard for us to reproduce at
> this point.
> 
> As mentioned, upgrading the firmware has resolved the problem for us,
> though it certainly feels unsatisfying to leave the bug there. I have no
> strong opinion on whether the patch should be included as is
> anyway. Maybe the firmware version info is enough to help you reproduce
> the problem?

Thanks for the additional information Linus. Unfortunately, our 
validation was unable to reproduce the issue with the specified firmware 
and setup. We were able to see failure without patch, but not with the 
patch applied. If you are able to run across the issue again or have any 
other setup info that may help with a reproduction, please let us know.

Thanks,
Tony

> Linus
> 
> 
> ________
> /etc/systemd/network/10-bond0.netdev:
> [NetDev]
> Kind=bond
> Name=bond0
> 
> [Bond]
> DownDelaySec=0.200000
> LACPTransmitRate=fast
> MIIMonitorSec=0.100000
> Mode=802.3ad
> TransmitHashPolicy=layer3+4
> UpDelaySec=0.200000
> 
> ________
> /etc/systemd/network/40-bond0.network:
> [Match]
> Name=bond0
> 
> [Link]
> #MACAddress=<omitted>
> RequiredForOnline=carrier
> 
> [Network]
> LinkLocalAddressing=no
> 
> # some Address and Route sections omitted
> ________
> /etc/systemd/network/30-eno12419.network:
> [Match]
> Name=eno12419
> 
> [Network]
> Bond=bond0
> 
> ________
> /etc/systemd/network/30-eno12399.network:
> [Match]
> Name=eno12399
> 
> [Network]
> Bond=bond0
> 
> _______________________________________________
> Intel-wired-lan mailing list
> Intel-wired-lan at osuosl.org
> https://lists.osuosl.org/mailman/listinfo/intel-wired-lan