[Intel-wired-lan] [ixgbe] Problem with Linux 4.8.4, link doesn’t receive packets

Tantilov, Emil S emil.s.tantilov at intel.com
Thu Oct 27 19:05:18 UTC 2016


>-----Original Message-----
>From: Intel-wired-lan [mailto:intel-wired-lan-bounces at lists.osuosl.org] On
>Behalf Of Paul Menzel
>Sent: Thursday, October 27, 2016 11:37 AM
>To: intel-wired-lan at lists.osuosl.org; Kirsher, Jeffrey T
><jeffrey.t.kirsher at intel.com>
>Subject: [Intel-wired-lan] [ixgbe] Problem with Linux 4.8.4, link doesn’t
>receive packets
>
>Dear Linux developers,
>
>
>Booting Linux 4.8.4, we have the following behavior on a TYAN S8812 with
>the devices below.
>
>[I mark it up as citation, as otherwise Mozilla Thunderbird autowraps it.]
>
>> 04:00.0 Ethernet controller [0200]: Intel Corporation 82574L Gigabit
>Network Connection [8086:10d3]
>> 05:00.0 Ethernet controller [0200]: Intel Corporation 82576 Gigabit
>Network Connection [8086:10c9] (rev 01)
>> 05:00.1 Ethernet controller [0200]: Intel Corporation 82576 Gigabit
>Network Connection [8086:10c9] (rev 01)
>> 06:00.0 RAID bus controller [0104]: 3ware Inc 9750 SAS2/SATA-II RAID PCIe
>[13c1:1010] (rev 05)
>> 07:00.0 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit
>SFI/SFP+ Network Connection [8086:10fb] (rev 01)
>> 07:00.1 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit
>SFI/SFP+ Network Connection [8086:10fb] (rev 01)
>
>The problem is, the link is up, but doesn’t receive any packets. Both
>LEDs are on, the switch doesn’t show a connection though.
>
>### Linux 4.4.14 ###
>
>Here is the working boot with Linux 4.4.14.
>
>> [   14.407431] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver -
>version 4.2.1-k
>> [   14.407878] ixgbe: Copyright (c) 1999-2015 Intel Corporation.
>> [   15.509390] ixgbe 0000:07:00.0: Multiqueue Enabled: Rx Queue count =
>63, Tx Queue count = 63
>> [   15.509960] ixgbe 0000:07:00.0: PCI Express bandwidth of 32GT/s
>available
>> [   15.510194] ixgbe 0000:07:00.0: (Speed:5.0GT/s, Width: x8, Encoding
>Loss:20%)
>> [   15.510516] ixgbe 0000:07:00.0: MAC: 2, PHY: 1, PBA No: E68793-004
>> [   15.510794] ixgbe 0000:07:00.0: 00:1b:21:d6:ca:bc
>> [   15.513170] ixgbe 0000:07:00.0: Intel(R) 10 Gigabit Network Connection
>> [   15.682545] ixgbe 0000:07:00.1: Multiqueue Enabled: Rx Queue count =
>63, Tx Queue count = 63
>> [   15.683115] ixgbe 0000:07:00.1: PCI Express bandwidth of 32GT/s
>available
>> [   15.683352] ixgbe 0000:07:00.1: (Speed:5.0GT/s, Width: x8, Encoding
>Loss:20%)
>> [   15.683672] ixgbe 0000:07:00.1: MAC: 2, PHY: 18, SFP+: 6, PBA No:
>E68793-004
>> [   15.683949] ixgbe 0000:07:00.1: 00:1b:21:d6:ca:bd
>> [   15.686311] ixgbe 0000:07:00.1: Intel(R) 10 Gigabit Network Connection
>> [   18.493786] ixgbe 0000:07:00.0 net03: renamed from eth3
>> [   18.507056] ixgbe 0000:07:00.1 net04: renamed from eth4
>> [   18.839122] ixgbe 0000:07:00.1: registered PHC device on net04
>> [   19.010366] ixgbe 0000:07:00.1 net04: detected SFP+: 6
>> [   19.253458] ixgbe 0000:07:00.1 net04: NIC Link is Up 10 Gbps, Flow
>Control: RX/TX
>
>### Linux 4.8.4 ###
>
>> [   22.136906] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver -
>version 4.4.0-k
>> [   22.137360] ixgbe: Copyright (c) 1999-2016 Intel Corporation.
>> [   23.272410] ixgbe 0000:07:00.0: Multiqueue Enabled: Rx Queue count =
>63, Tx Queue count = 63
>> [   23.272931] ixgbe 0000:07:00.0: PCI Express bandwidth of 32GT/s
>available
>> [   23.273217] ixgbe 0000:07:00.0: (Speed:5.0GT/s, Width: x8, Encoding
>Loss:20%)
>> [   23.273532] ixgbe 0000:07:00.0: MAC: 2, PHY: 1, PBA No: E68793-004
>> [   23.273765] ixgbe 0000:07:00.0: 00:1b:21:d6:ca:bc
>> [   23.276469] ixgbe 0000:07:00.0: Intel(R) 10 Gigabit Network Connection
>> [   23.438461] ixgbe 0000:07:00.1: Multiqueue Enabled: Rx Queue count =
>63, Tx Queue count = 63
>> [   23.438981] ixgbe 0000:07:00.1: PCI Express bandwidth of 32GT/s
>available
>> [   23.439266] ixgbe 0000:07:00.1: (Speed:5.0GT/s, Width: x8, Encoding
>Loss:20%)
>> [   23.439580] ixgbe 0000:07:00.1: MAC: 2, PHY: 18, SFP+: 6, PBA No:
>E68793-004
>> [   23.439817] ixgbe 0000:07:00.1: 00:1b:21:d6:ca:bd
>> [   23.442483] ixgbe 0000:07:00.1: Intel(R) 10 Gigabit Network Connection
>> [   26.911415] ixgbe 0000:07:00.0 net03: renamed from eth3
>> [   26.927234] ixgbe 0000:07:00.1 net04: renamed from eth4
>> [   27.277634] ixgbe 0000:07:00.1: registered PHC device on net04
>> [   27.457916] ixgbe 0000:07:00.1 net04: detected SFP+: 6
>> [   27.713989] ixgbe 0000:07:00.1 net04: NIC Link is Up 10 Gbps, Flow
>Control: RX/TX
>
>No packets are received on the interface. After unplugging the network
>cable, and plugging it back in, it works.
>
>> [ 2009.455327] ixgbe 0000:07:00.1 net04: NIC Link is Down
>> [ 2012.354168] ixgbe 0000:07:00.1 net04: NIC Link is Up 10 Gbps, Flow
>Control: RX/TX
>
>
>After a reboot, there is the *same* result.
>
>> [   21.427370] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver -
>version 4.4.0-k
>> [   21.431447] ixgbe: Copyright (c) 1999-2016 Intel Corporation.
>> [   22.568344] ixgbe 0000:07:00.0: Multiqueue Enabled: Rx Queue count =
>63, Tx Queue count = 63
>> [   22.568506] ixgbe 0000:07:00.0: PCI Express bandwidth of 32GT/s
>available
>> [   22.568508] ixgbe 0000:07:00.0: (Speed:5.0GT/s, Width: x8, Encoding
>Loss:20%)
>> [   22.568588] ixgbe 0000:07:00.0: MAC: 2, PHY: 1, PBA No: E68793-004
>> [   22.568589] ixgbe 0000:07:00.0: 00:1b:21:d6:ca:bc
>> [   22.571967] ixgbe 0000:07:00.0: Intel(R) 10 Gigabit Network Connection
>> [   22.734311] ixgbe 0000:07:00.1: Multiqueue Enabled: Rx Queue count =
>63, Tx Queue count = 63
>> [   22.734833] ixgbe 0000:07:00.1: PCI Express bandwidth of 32GT/s
>available
>> [   22.735126] ixgbe 0000:07:00.1: (Speed:5.0GT/s, Width: x8, Encoding
>Loss:20%)
>> [   22.735441] ixgbe 0000:07:00.1: MAC: 2, PHY: 18, SFP+: 6, PBA No:
>E68793-004
>> [   22.735675] ixgbe 0000:07:00.1: 00:1b:21:d6:ca:bd
>> [   22.738115] ixgbe 0000:07:00.1: Intel(R) 10 Gigabit Network Connection
>> [   25.980145] ixgbe 0000:07:00.0 net03: renamed from eth3
>> [   26.005989] ixgbe 0000:07:00.1 net04: renamed from eth4
>> [   26.379269] ixgbe 0000:07:00.1: registered PHC device on net04
>> [   26.551580] ixgbe 0000:07:00.1 net04: detected SFP+: 6
>> [   26.802655] ixgbe 0000:07:00.1 net04: NIC Link is Up 10 Gbps, Flow
>Control: RX/TX
>
>
>After doing `ip link set dev net04 down`, and `ip link set dev net04
>up`, the link works as expected.
>
>> [  151.437972] ixgbe 0000:07:00.1: removed PHC on net04
>> [  162.174209] ixgbe 0000:07:00.1: registered PHC device on net04
>> [  162.350749] ixgbe 0000:07:00.1 net04: detected SFP+: 6
>> [  162.599821] ixgbe 0000:07:00.1 net04: NIC Link is Up 10 Gbps, Flow
>Control: RX/TX
>
>So I have some questions.
>
>1. Have you heard of such problems in Linux kernels after 4.4?
 
We have now. 

From what you are describing it appears that the switch is not detecting
a link when the interface is brought up. Resetting the interface will
also force a re-negotiation of the link which apparently helps.
 
>2. Can you explain the behavior, that there is no message *NIC Link is
>Down* from Linux after `ip link set dev net04 down`?

The "Link Up/Down" messages are interrupt driven and when you bring the
interface down there are no interrupts and hence no way to detect the 
link change. BTW just because you brought the interface down it does not
mean that the link is down.
 
>3. If it is a bug, what can I do to get it fixed? Is this mailing list
>the right forum, or should I create a ticket at the Linux Kernel
>Bugtracker [1]? What other information than above do you need?

What is the SFP module you are using and the model of the switch?
Are you using a supported SFP+ module (not loading the driver with
allow_unsupported_sfp)?

If it worked in previous versions of the kernel/driver then it could
be a bug introduced with a more recent patch. If you can bisect it you
should be able to pinpoint the patch that introduced this issue.

Also you may want to test with the current development tree from:
git://git.kernel.org/pub/scm/linux/kernel/git/jkirsher/next-queue.git

In case this issue has already been addressed in the development branch.

Thanks,
Emil




More information about the Intel-wired-lan mailing list