[Intel-wired-lan] Counter spikes in /proc/net/dev for E810-CQDA2 interfaces (ice driver) on kernel >=6.2

Fri Nov 17 12:05:17 UTC 2023

Hey Przemek,

thanks for picking up on this so quickly!

On 17.11.23 11:13, Przemek Kitszel wrote:
>> If you require any more information to narrow down the issue, please 
>> don't hesitate to contact me.
>
> Was there anything logged in dmesg or other system logs at that time?

Nothing I would correlate with this issue:

--- cut ---
[...]
[Thu Nov 16 09:46:05 2023] brqe861cc6a-7a: port 4(tap12f22cf7-9f) 
entered blocking state
[Thu Nov 16 09:46:05 2023] brqe861cc6a-7a: port 4(tap12f22cf7-9f) 
entered disabled state
[Thu Nov 16 09:46:05 2023] device tap12f22cf7-9f entered promiscuous mode
[Thu Nov 16 09:46:05 2023] brqe861cc6a-7a: port 4(tap12f22cf7-9f) 
entered blocking state
[Thu Nov 16 09:46:05 2023] brqe861cc6a-7a: port 4(tap12f22cf7-9f) 
entered forwarding state
[Thu Nov 16 09:46:41 2023] vxlan-710: e2:d6:9b:e7:3e:1f migrated from 
10.101.11.101 to 10.101.11.99
[Thu Nov 16 09:46:49 2023] vxlan-112: fa:16:3e:0a:e3:78 migrated from 
10.101.11.101 to 10.101.11.97
[Thu Nov 16 09:46:57 2023] vxlan-449: fa:16:3e:3b:a2:96 migrated from 
10.101.11.101 to 10.101.11.98
[Thu Nov 16 09:47:13 2023] vxlan-710: b2:61:c3:32:2b:a3 migrated from 
10.101.11.101 to 10.101.11.98
[Thu Nov 16 09:47:14 2023] vxlan-167: ce:48:98:bb:37:b5 migrated from 
10.101.11.101 to 10.101.11.98
[Thu Nov 16 09:47:16 2023] vxlan-449: fa:16:3e:0c:e2:dd migrated from 
10.101.11.101 to 10.101.11.99
[Thu Nov 16 09:47:21 2023] vxlan-449: fa:16:3e:5e:f8:65 migrated from 
10.101.11.101 to 10.101.11.99
[Thu Nov 16 10:09:10 2023] clocksource: timekeeping watchdog on CPU44: 
hpet wd-wd read-back delay of 245282ns
[Thu Nov 16 10:09:10 2023] clocksource: wd-tsc-wd read-back delay of 
237530ns, clock-skew test skipped!
[Thu Nov 16 11:09:00 2023] brq993a79c9-f7: port 2(tap6b35cfee-6c) 
entered disabled state
[Thu Nov 16 11:09:00 2023] device tap6b35cfee-6c left promiscuous mode
[Thu Nov 16 11:09:00 2023] brq993a79c9-f7: port 2(tap6b35cfee-6c) 
entered disabled state
[Thu Nov 16 11:09:00 2023] brq8daed714-05: port 10(tapcd40cc09-db) 
entered disabled state
[Thu Nov 16 11:09:00 2023] device tapcd40cc09-db left promiscuous mode
[Thu Nov 16 11:09:00 2023] brq8daed714-05: port 10(tapcd40cc09-db) 
entered disabled state
[Thu Nov 16 11:09:03 2023] device vxlan-393 left promiscuous mode
[Thu Nov 16 11:09:03 2023] brqbe0ca29c-93: port 1(vxlan-393) entered 
disabled state
[Thu Nov 16 11:09:03 2023] device tap39935f01-25 left promiscuous mode
[Thu Nov 16 11:09:03 2023] brqbe0ca29c-93: port 2(tap39935f01-25) 
entered disabled state
[Thu Nov 16 11:09:05 2023] device vxlan-715 left promiscuous mode
[Thu Nov 16 11:09:05 2023] brq993a79c9-f7: port 1(vxlan-715) entered 
disabled state
[Thu Nov 16 11:13:03 2023] clocksource: timekeeping watchdog on CPU71: 
hpet wd-wd read-back delay of 245352ns
[Thu Nov 16 11:13:03 2023] clocksource: wd-tsc-wd read-back delay of 
244653ns, clock-skew test skipped!
[Thu Nov 16 12:11:25 2023] brqf6366546-d3: port 1(tapa284b204-c0) 
entered blocking state
[Thu Nov 16 12:11:25 2023] brqf6366546-d3: port 1(tapa284b204-c0) 
entered disabled state
[Thu Nov 16 12:11:25 2023] device tapa284b204-c0 entered promiscuous mode
[Thu Nov 16 12:11:25 2023] brqf6366546-d3: port 1(tapa284b204-c0) 
entered blocking state
[Thu Nov 16 12:11:25 2023] brqf6366546-d3: port 1(tapa284b204-c0) 
entered forwarding state
[Thu Nov 16 12:22:21 2023] clocksource: timekeeping watchdog on CPU107: 
hpet wd-wd read-back delay of 242419ns
[Thu Nov 16 12:22:21 2023] clocksource: wd-tsc-wd read-back delay of 
234387ns, clock-skew test skipped!
[Thu Nov 16 12:50:37 2023] clocksource: timekeeping watchdog on CPU83: 
hpet wd-wd read-back delay of 245073ns
[Thu Nov 16 12:50:37 2023] clocksource: wd-tsc-wd read-back delay of 
243257ns, clock-skew test skipped!
[Thu Nov 16 13:32:22 2023] clocksource: timekeeping watchdog on CPU13: 
hpet wd-wd read-back delay of 248006ns
[Thu Nov 16 13:32:22 2023] clocksource: wd-tsc-wd read-back delay of 
245352ns, clock-skew test skipped!
[Thu Nov 16 13:52:43 2023] clocksource: timekeeping watchdog on CPU63: 
hpet wd-wd read-back delay of 244863ns
[Thu Nov 16 13:52:43 2023] clocksource: wd-tsc-wd read-back delay of 
245561ns, clock-skew test skipped!
[Thu Nov 16 14:28:24 2023] clocksource: timekeeping watchdog on CPU33: 
hpet wd-wd read-back delay of 245282ns
[Thu Nov 16 14:28:24 2023] clocksource: wd-tsc-wd read-back delay of 
245352ns, clock-skew test skipped!
[Thu Nov 16 15:22:12 2023] clocksource: timekeeping watchdog on CPU1: 
hpet wd-wd read-back delay of 244863ns
[Thu Nov 16 15:22:12 2023] clocksource: wd-tsc-wd read-back delay of 
245631ns, clock-skew test skipped!
[Thu Nov 16 15:29:20 2023] clocksource: timekeeping watchdog on CPU1: 
hpet wd-wd read-back delay of 245282ns
[Thu Nov 16 15:29:20 2023] clocksource: wd-tsc-wd read-back delay of 
245282ns, clock-skew test skipped!
[Thu Nov 16 15:43:34 2023] clocksource: timekeeping watchdog on CPU85: 
hpet wd-wd read-back delay of 245142ns
[Thu Nov 16 15:43:34 2023] clocksource: wd-tsc-wd read-back delay of 
242907ns, clock-skew test skipped!
[Thu Nov 16 16:13:06 2023] clocksource: timekeeping watchdog on CPU85: 
hpet wd-wd read-back delay of 245212ns
[Thu Nov 16 16:13:06 2023] clocksource: wd-tsc-wd read-back delay of 
245561ns, clock-skew test skipped!
[Thu Nov 16 16:27:22 2023] clocksource: timekeeping watchdog on CPU45: 
hpet wd-wd read-back delay of 246260ns
[Thu Nov 16 16:27:22 2023] clocksource: wd-tsc-wd read-back delay of 
237600ns, clock-skew test skipped!
[Thu Nov 16 16:41:32 2023] clocksource: timekeeping watchdog on CPU122: 
hpet wd-wd read-back delay of 245631ns
[Thu Nov 16 16:41:32 2023] clocksource: wd-tsc-wd read-back delay of 
245282ns, clock-skew test skipped!
[Thu Nov 16 16:52:43 2023] clocksource: timekeeping watchdog on CPU95: 
hpet wd-wd read-back delay of 245282ns
[Thu Nov 16 16:52:43 2023] clocksource: wd-tsc-wd read-back delay of 
245282ns, clock-skew test skipped!
[Thu Nov 16 17:15:04 2023] clocksource: timekeeping watchdog on CPU2: 
hpet wd-wd read-back delay of 246190ns
[Thu Nov 16 17:15:04 2023] clocksource: wd-tsc-wd read-back delay of 
234666ns, clock-skew test skipped!
[Thu Nov 16 17:57:42 2023] clocksource: timekeeping watchdog on CPU38: 
hpet wd-wd read-back delay of 244863ns
[Thu Nov 16 17:57:42 2023] clocksource: wd-tsc-wd read-back delay of 
245561ns, clock-skew test skipped!
[Thu Nov 16 18:44:33 2023] clocksource: timekeeping watchdog on CPU67: 
hpet wd-wd read-back delay of 245073ns
[Thu Nov 16 18:44:33 2023] clocksource: wd-tsc-wd read-back delay of 
235155ns, clock-skew test skipped!
[Thu Nov 16 19:02:50 2023] clocksource: timekeeping watchdog on CPU125: 
hpet wd-wd read-back delay of 245352ns
[Thu Nov 16 19:02:50 2023] clocksource: wd-tsc-wd read-back delay of 
244793ns, clock-skew test skipped!
[Thu Nov 16 19:19:00 2023] clocksource: timekeeping watchdog on CPU57: 
hpet wd-wd read-back delay of 245212ns
[Thu Nov 16 19:19:00 2023] clocksource: wd-tsc-wd read-back delay of 
245142ns, clock-skew test skipped!
[Thu Nov 16 19:27:09 2023] clocksource: timekeeping watchdog on CPU51: 
hpet wd-wd read-back delay of 245561ns
[Thu Nov 16 19:27:09 2023] clocksource: wd-tsc-wd read-back delay of 
245352ns, clock-skew test skipped!
[Thu Nov 16 19:50:34 2023] clocksource: timekeeping watchdog on CPU85: 
hpet wd-wd read-back delay of 245282ns
[Thu Nov 16 19:50:34 2023] clocksource: wd-tsc-wd read-back delay of 
245142ns, clock-skew test skipped!
[Thu Nov 16 20:11:49 2023] clocksource: timekeeping watchdog on CPU115: 
hpet wd-wd read-back delay of 245701ns
[Thu Nov 16 20:11:49 2023] clocksource: wd-tsc-wd read-back delay of 
245142ns, clock-skew test skipped!
[Thu Nov 16 20:31:06 2023] clocksource: timekeeping watchdog on CPU38: 
hpet wd-wd read-back delay of 241860ns
[Thu Nov 16 20:31:06 2023] clocksource: wd-tsc-wd read-back delay of 
248984ns, clock-skew test skipped!
[Thu Nov 16 20:37:08 2023] clocksource: timekeeping watchdog on CPU34: 
hpet wd-wd read-back delay of 245561ns
[Thu Nov 16 20:37:08 2023] clocksource: wd-tsc-wd read-back delay of 
244933ns, clock-skew test skipped!
[Thu Nov 16 20:45:18 2023] clocksource: timekeeping watchdog on CPU30: 
hpet wd-wd read-back delay of 245282ns
[Thu Nov 16 20:45:18 2023] clocksource: wd-tsc-wd read-back delay of 
245073ns, clock-skew test skipped!
[Thu Nov 16 21:02:35 2023] clocksource: timekeeping watchdog on CPU96: 
hpet wd-wd read-back delay of 245352ns
[Thu Nov 16 21:02:35 2023] clocksource: wd-tsc-wd read-back delay of 
245212ns, clock-skew test skipped!
[Thu Nov 16 21:02:51 2023] vxlan-117: fa:16:3e:9f:4b:81 migrated from 
10.101.11.17 to 10.101.11.101
[Thu Nov 16 21:10:32 2023] vxlan-117: fa:16:3e:a2:d8:b1 migrated from 
10.101.11.16 to 10.101.11.101
[Thu Nov 16 21:11:46 2023] clocksource: timekeeping watchdog on CPU85: 
hpet wd-wd read-back delay of 242279ns
[Thu Nov 16 21:11:46 2023] clocksource: wd-tsc-wd read-back delay of 
248425ns, clock-skew test skipped!
[Thu Nov 16 21:14:41 2023] vxlan-117: fa:16:3e:03:ec:41 migrated from 
10.101.11.17 to 10.101.11.98
[Thu Nov 16 21:19:30 2023] vxlan-117: fa:16:3e:3e:6e:72 migrated from 
10.101.11.17 to 10.101.11.97
[Thu Nov 16 21:23:02 2023] vxlan-117: fa:16:3e:a3:7d:e1 migrated from 
10.101.11.17 to 10.101.11.97
[Thu Nov 16 21:37:10 2023] clocksource: timekeeping watchdog on CPU103: 
hpet wd-wd read-back delay of 245422ns
[Thu Nov 16 21:37:10 2023] clocksource: wd-tsc-wd read-back delay of 
245212ns, clock-skew test skipped!
[Thu Nov 16 22:09:45 2023] clocksource: timekeeping watchdog on CPU85: 
hpet wd-wd read-back delay of 245422ns
[Thu Nov 16 22:09:45 2023] clocksource: wd-tsc-wd read-back delay of 
244863ns, clock-skew test skipped!
[...]
--- cut ---

BTW, this machine runs OpenStack Nova + Neutron, that's the bridge, tap 
and vxlan messages you see there.
We observe the issue on multiple machines. But some of them either don't 
have the issue or very very sendomly, others have this every hour or so.
All of them are the exact same make and model including all of the 
peripherals and firmware versions.

> Thank you for the report, I will take a look.
>
> We have already received similar report from Nebojsa Stevanovic, CCed.
>
> Sorry that the issue is not resolved yet. I will review what we have
> changed in the drivers between 6.1 and 6.2, where bug was introduced. 

Thank again for looking into this.

Regards

Christian