Instable connection when moving from PCI Express to PCI Express FD

[felixwidmaier]

I used `candump -t d` with a filter to only show a specific frames that is sent from the motor board with 1 kHz.
Using the classic PCAN cards (i.e. without FD) the output looked like this:

Code: Select all

     (000.001032)  can5  040   [8]  00 00 E6 F0 00 00 00 00
     (000.001019)  can5  040   [8]  00 00 D7 D1 00 00 00 00
     (000.000999)  can5  040   [8]  00 00 60 D7 00 00 00 00
     (000.001031)  can5  040   [8]  00 00 03 53 00 00 00 00
     (000.001001)  can5  040   [8]  FF FE 1F 0F 00 00 00 00
     (000.000986)  can5  040   [8]  FF FD B4 E7 00 00 00 00
     (000.000993)  can5  040   [8]  FF FE 54 34 00 00 00 00

So here the timing is pretty stable.

When switching to the FD cards, the same test gives an output like this:

Code: Select all

     (000.001098)  can0  040   [8]  00 19 45 EA FF FF C3 D2
     (000.001104)  can0  040   [8]  00 07 89 A3 00 03 06 0A
     (000.000017)  can0  040   [8]  FF F2 27 E3 00 04 6C DF
     (000.001098)  can0  040   [8]  FF DD 0F 1B 00 05 5B AB
     (000.001499)  can0  040   [8]  FF D1 F1 62 00 04 74 B1
     (000.001098)  can0  040   [8]  FF C8 42 01 00 03 0B E7
     (000.001099)  can0  040   [8]  FF D4 BD C4 00 00 87 4F
     (000.000017)  can0  040   [8]  FF E2 04 D5 FF FE D1 4E
     (000.001202)  can0  040   [8]  FF F2 EE C4 FF FC 4C 3D
     (000.001100)  can0  040   [8]  00 08 4D 86 FF FB 72 AA
     (000.001499)  can0  040   [8]  00 1E 01 5B FF FA A2 A7

Here the timing is rather unreliable. In practice this becomes apparent as the motor control is much less stable, resulting in the motor to vibrate.

We are sending frames from a C++ application using the socket API. The corresponding code is running in a real-time thread (using Linux with the PREEMPT_RT kernel patch). I'm looking more closely into this code now, to see if I find anything fishy. However, the delays in the received messages can also be observed while the sending application is not running.

I finally checked with the logic analyzer. At least on first glance, I could not find anything obviously wrong. Actually, I cannot even observe the delays that candump shows on the bus, the rate of the frames seems to be pretty constant. Could it be that the delay is somehow caused in the driver (i.e. after receiving)? We are using the default drivers that are shipped with the kernel, I don't know how real-time-capable they are.

---------------------------

Here are longer dumps (using "candump -t z -xe") including all frames.
With the old card:

Code: Select all

 (000.000000)  can5  RX - -  020   [8]  FF DD 6E 48 FF FE F7 40
 (000.000120)  can5  RX - -  030   [8]  00 03 38 E7 FF FF B6 34
 (000.000227)  can5  TX - -  005   [8]  FF DB 40 49 00 00 00 00
 (000.000366)  can5  RX - -  040   [8]  00 00 02 2C FF FF FF EF
 (000.000486)  can5  RX - -  050   [8]  00 4E B0 00 00 8C 50 00
 (000.000991)  can5  RX - -  020   [8]  FF D6 A8 C0 FF FA 21 90
 (000.001111)  can5  RX - -  030   [8]  00 03 38 E7 FF FF B6 34
 (000.001218)  can5  TX - -  005   [8]  FF DC F9 D6 00 00 00 00
 (000.001357)  can5  RX - -  040   [8]  00 00 00 0D FF FF FF EF
 (000.001478)  can5  RX - -  050   [8]  00 4E 70 00 00 8C 20 00
 (000.001985)  can5  RX - -  020   [8]  FF DD FD 10 00 03 30 30
 (000.002104)  can5  RX - -  030   [8]  00 03 35 A2 FF FF B6 34
 (000.002209)  can5  TX - -  005   [8]  FF DD 08 EB 00 00 00 00
 (000.002344)  can5  RX - -  040   [8]  FF FF 50 A8 FF FF FF EF
 (000.002464)  can5  RX - -  050   [8]  00 4E A0 00 00 8C 30 00
 (000.002976)  can5  RX - -  020   [8]  FF DB F7 60 FF F9 2F 98
 (000.003094)  can5  RX - -  030   [8]  00 03 32 5C FF FF B6 34
 (000.003218)  can5  RX - -  040   [8]  FF FF A0 76 FF FF FF EF
 (000.003326)  can5  TX - -  005   [8]  FF E2 0E 1F 00 00 00 00
 (000.003462)  can5  RX - -  050   [8]  00 4E B0 00 00 8C 30 00
 (000.003967)  can5  RX - -  020   [8]  FF D5 9A 48 00 02 DA E0
 (000.004086)  can5  RX - -  030   [8]  00 03 2F 16 FF FF B6 34
 (000.004208)  can5  RX - -  040   [8]  FF FF BC C2 FF FF FF EF
 (000.004316)  can5  TX - -  005   [8]  FF DF FC C0 00 00 00 00
 (000.004452)  can5  RX - -  050   [8]  00 4E 70 00 00 8C 00 00
 (000.004959)  can5  RX - -  020   [8]  FF DA 14 A8 00 01 13 B8
 (000.005078)  can5  RX - -  030   [8]  00 03 2B D0 FF FF B6 34
 (000.005199)  can5  RX - -  040   [8]  FF FF BD C2 FF FF FF EF
 (000.005307)  can5  TX - -  005   [8]  FF DF 7F B8 00 00 00 00
 (000.005443)  can5  RX - -  050   [8]  00 4E 60 00 00 8B F0 00
 (000.005951)  can5  RX - -  020   [8]  FF E0 65 C0 FF FF 21 E8
 (000.006069)  can5  RX - -  030   [8]  00 03 2B D0 FF FF B6 34
 (000.006192)  can5  RX - -  040   [8]  FF FF FD A5 FF FF FF EF
 (000.006299)  can5  TX - -  005   [8]  FF DF 78 9C 00 00 00 00
 (000.006434)  can5  RX - -  050   [8]  00 4E 80 00 00 8C 30 00
 (000.006944)  can5  RX - -  020   [8]  FF DF 0A 50 FF FC 13 48
 (000.007063)  can5  RX - -  030   [8]  00 03 2B D0 FF FF B6 34
 (000.007190)  can5  RX - -  040   [8]  FF FF FF DF FF FF FF EF
 (000.007308)  can5  RX - -  050   [8]  00 4E 90 00 00 8C 20 00
 (000.007413)  can5  TX - -  005   [8]  FF DD B2 56 00 00 00 00
 (000.007936)  can5  RX - -  020   [8]  FF D9 E2 80 FF FE 2F D8
 (000.008054)  can5  RX - -  030   [8]  00 03 2B D0 FF FF B6 34
 (000.008176)  can5  RX - -  040   [8]  FF FF FF EC FF FF FF EF
 (000.008298)  can5  RX - -  050   [8]  00 4E 70 00 00 8C 10 00
 (000.008405)  can5  TX - -  005   [8]  FF DD A2 81 00 00 00 00
 (000.008929)  can5  RX - -  020   [8]  FF DF 51 90 FF FE 5A 80
 (000.009046)  can5  RX - -  030   [8]  00 03 2F 15 FF FF B6 34
 (000.009168)  can5  RX - -  040   [8]  00 00 59 65 FF FF FF EF
 (000.009291)  can5  RX - -  050   [8]  00 4E 70 00 00 8C 10 00
 (000.009399)  can5  TX - -  005   [8]  FF DA FF E5 00 00 00 00
 (000.009918)  can5  RX - -  020   [8]  FF D5 D2 70 00 07 B0 90
 (000.010036)  can5  RX - -  030   [8]  00 03 32 5B FF FF B6 34
 (000.010158)  can5  RX - -  040   [8]  00 00 42 EE FF FF FF EF
 (000.010280)  can5  RX - -  050   [8]  00 4E 90 00 00 8C 40 00
 (000.010386)  can5  TX - -  005   [8]  FF DB 79 8C 00 00 00 00
 (000.010910)  can5  RX - -  020   [8]  FF D7 9E 90 00 03 05 88
 (000.011029)  can5  RX - -  030   [8]  00 03 35 A1 FF FF B6 34
 (000.011152)  can5  RX - -  040   [8]  00 00 2F E5 FF FF FF EF
 (000.011276)  can5  RX - -  050   [8]  00 4E 80 00 00 8C 50 00
 (000.011402)  can5  TX - -  005   [8]  FF DB DA CF 00 00 00 00
 (000.011902)  can5  RX - -  020   [8]  FF DC FD 38 FF FB E8 A0
 (000.012022)  can5  RX - -  030   [8]  00 03 38 E7 FF FF B6 34
 (000.012145)  can5  RX - -  040   [8]  00 00 2F 41 FF FF FF EF
 (000.012267)  can5  RX - -  050   [8]  00 4E C0 00 00 8C 50 00
 (000.012411)  can5  TX - -  005   [8]  FF DB B9 47 00 00 00 00
 (000.012893)  can5  RX - -  020   [8]  FF DC FD 38 FF FF 21 D0
 (000.013013)  can5  RX - -  030   [8]  00 03 38 E7 FF FF B6 34
 (000.013137)  can5  RX - -  040   [8]  00 00 01 96 FF FF FF EF
 (000.013257)  can5  RX - -  050   [8]  00 4E B0 00 00 8C 50 00
 (000.013433)  can5  TX - -  005   [8]  FF DC FE 01 00 00 00 00
 (000.013883)  can5  RX - -  020   [8]  FF E7 B9 38 FF F5 A1 18
 (000.014003)  can5  RX - -  030   [8]  00 03 38 E7 FF FF B6 34
 (000.014126)  can5  RX - -  040   [8]  00 00 00 06 FF FF FF EF
 (000.014247)  can5  RX - -  050   [8]  00 4E 90 00 00 8C 20 00
 (000.014454)  can5  TX - -  005   [8]  FF DD 09 1D 00 00 00 00
 (000.014878)  can5  RX - -  020   [8]  FF DE 95 E0 FF EE 59 88
 (000.014998)  can5  RX - -  030   [8]  00 03 38 E7 FF FF B6 34
 (000.015124)  can5  RX - -  040   [8]  00 00 00 00 FF FF FF EF
 (000.015246)  can5  RX - -  050   [8]  00 4E 90 00 00 8C 30 00
 (000.015477)  can5  TX - -  005   [8]  FF DD 09 48 00 00 00 00
 (000.015867)  can5  RX - -  020   [8]  FF C8 58 B8 00 04 4C E8
 (000.015985)  can5  RX - -  030   [8]  00 03 35 A2 FF FF B6 34
 (000.016105)  can5  RX - -  040   [8]  FF FE E7 66 FF FF FF EF
 (000.016225)  can5  RX - -  050   [8]  00 4E A0 00 00 8C 50 00
 (000.016496)  can5  TX - -  005   [8]  FF E4 FA 92 00 00 00 00
 (000.016861)  can5  RX - -  020   [8]  FF D5 D3 C0 00 04 CC C8
 (000.016979)  can5  RX - -  030   [8]  00 03 35 A2 FF FF B6 34
 (000.017100)  can5  RX - -  040   [8]  FF FF E8 AA FF FF FF EF
 (000.017220)  can5  RX - -  050   [8]  00 4E 80 00 00 8C 30 00
 (000.017518)  can5  TX - -  005   [8]  FF DD D5 41 00 00 00 00
 (000.017852)  can5  RX - -  020   [8]  FF E8 1E 18 00 02 30 58
 (000.017970)  can5  RX - -  030   [8]  00 03 32 5C FF FF B6 34
 (000.018091)  can5  RX - -  040   [8]  FF FF D1 89 FF FF FF EF
 (000.018214)  can5  RX - -  050   [8]  00 4E D0 00 00 8C 90 00
 (000.018540)  can5  TX - -  005   [8]  FF DE 9F CE 00 00 00 00
 (000.018847)  can5  RX - -  020   [8]  FF DF 01 20 00 00 76 F8
 (000.018966)  can5  RX - -  030   [8]  00 03 32 5C FF FF B6 34
 (000.019087)  can5  RX - -  040   [8]  FF FF FE 56 FF FF FF EF
 (000.019207)  can5  RX - -  050   [8]  00 4E A0 00 00 8C 50 00
 (000.019560)  can5  TX - -  005   [8]  FF DD 61 3E 00 00 00 00

with the new (FD) card:

Code: Select all

 (000.000000)  can0  RX - -  040   [8]  FF F2 B3 4E 00 00 0A AD
 (000.000004)  can0  RX - -  050   [8]  00 4E 40 00 00 8C 00 00
 (000.000005)  can0  TX - -  005   [8]  00 A8 58 57 00 00 00 00
 (000.000009)  can0  RX - -  020   [8]  00 D0 5A 08 00 0B 1F 00
 (000.000013)  can0  RX - -  030   [8]  FF FE 49 11 00 00 06 8B
 (000.000017)  can0  RX - -  040   [8]  00 02 50 22 00 00 00 79
 (000.001102)  can0  RX - -  050   [8]  00 4E 50 00 00 8C 20 00
 (000.001103)  can0  TX - -  005   [8]  00 A8 58 57 00 00 00 00
 (000.001107)  can0  RX - -  020   [8]  00 87 C4 E0 00 04 5C 90
 (000.001111)  can0  RX - -  030   [8]  FF FE 8A 89 00 00 06 8B
 (000.001115)  can0  RX - -  040   [8]  00 11 5A AA 00 00 00 00
 (000.001119)  can0  RX - -  050   [8]  00 4E 40 00 00 8C 20 00
 (000.002202)  can0  TX - -  005   [8]  00 06 8B E2 00 00 00 00
 (000.002206)  can0  RX - -  020   [8]  00 90 5E 38 00 04 58 B8
 (000.002210)  can0  RX - -  030   [8]  FF FE F6 8F 00 00 06 8B
 (000.002214)  can0  RX - -  040   [8]  00 19 C7 D7 00 00 00 00
 (000.002218)  can0  RX - -  050   [8]  00 4D F0 00 00 8C 00 00
 (000.002219)  can0  TX - -  005   [8]  FF 98 9E 88 00 00 00 00
 (000.003709)  can0  RX - -  020   [8]  00 36 08 A0 FF FB D2 50
 (000.003713)  can0  RX - -  030   [8]  FF FF C3 4B 00 00 03 46
 (000.003717)  can0  RX - -  040   [8]  00 46 07 8D FF FF 62 38
 (000.003721)  can0  RX - -  050   [8]  00 4E 00 00 00 8C 10 00
 (000.003722)  can0  TX - -  005   [8]  FF 57 CA CE 00 00 00 00
 (000.003726)  can0  RX - -  020   [8]  00 05 70 78 FF FA 2A 48
 (000.004811)  can0  RX - -  030   [8]  00 00 53 53 00 00 00 00
 (000.004815)  can0  RX - -  040   [8]  00 21 13 00 FF FE 47 71
 (000.004819)  can0  RX - -  050   [8]  00 4E 30 00 00 8C 00 00
 (000.004820)  can0  TX - -  005   [8]  FE 13 D9 C8 00 00 00 00
 (000.004824)  can0  RX - -  020   [8]  FF AD E2 10 00 04 5B A0
 (000.004828)  can0  RX - -  030   [8]  00 00 C9 2B 00 00 00 00
 (000.005914)  can0  RX - -  040   [8]  00 19 4E 3B FF FF EA A5
 (000.005918)  can0  RX - -  050   [8]  00 4E 50 00 00 8C 10 00
 (000.005919)  can0  TX - -  005   [8]  FF 0E B8 EE 00 00 00 00
 (000.005923)  can0  RX - -  020   [8]  FE F0 7B C8 FF FB DA 48
 (000.005927)  can0  RX - -  030   [8]  00 01 14 75 00 00 00 00
 (000.005931)  can0  RX - -  040   [8]  00 0F A3 82 FF FF FE ED
 (000.007016)  can0  RX - -  050   [8]  00 4E 90 00 00 8C 20 00
 (000.007017)  can0  TX - -  005   [8]  FF 87 47 03 00 00 00 00
 (000.007021)  can0  RX - -  020   [8]  FF 17 1A 78 FF FB DA 48
 (000.007025)  can0  RX - -  030   [8]  00 01 11 30 00 00 00 00
 (000.007029)  can0  RX - -  040   [8]  FF FB 7D 55 FF FF FF EA
 (000.007033)  can0  RX - -  050   [8]  00 4E B0 00 00 8C 40 00
 (000.008316)  can0  TX - -  005   [8]  00 16 B2 C8 00 00 00 00
 (000.008320)  can0  RX - -  020   [8]  FF 90 0D 48 00 04 5B A0
 (000.008324)  can0  RX - -  030   [8]  00 00 BC 14 00 00 00 00
 (000.008332)  can0  RX - -  040   [8]  FF EB 2B A9 FF FF FF EF
 (000.008336)  can0  RX - -  050   [8]  00 4E 90 00 00 8C 10 00
 (000.008337)  can0  TX - -  005   [8]  00 16 B2 C8 00 00 00 00
 (000.009722)  can0  RX - -  020   [8]  FF E1 88 30 00 04 5C 18
 (000.009726)  can0  RX - -  030   [8]  00 00 32 98 00 00 03 45
 (000.009730)  can0  RX - -  040   [8]  FF E0 3D 05 00 00 DC D1
 (000.009734)  can0  RX - -  050   [8]  00 4E D0 00 00 8C 10 00
 (000.009735)  can0  TX - -  005   [8]  00 8E 9A D5 00 00 00 00
 (000.009739)  can0  RX - -  020   [8]  FF F6 B5 D8 FF FF 38 C8
 (000.010825)  can0  RX - -  030   [8]  FF FF 58 C4 00 00 06 8B
 (000.010829)  can0  RX - -  040   [8]  FF B6 AB 85 00 00 E7 84
 (000.010833)  can0  RX - -  050   [8]  00 4E C0 00 00 8C 10 00
 (000.010834)  can0  TX - -  005   [8]  00 E2 96 8D 00 00 00 00
 (000.010838)  can0  RX - -  020   [8]  00 39 05 28 FF FD 7E 48
 (000.010842)  can0  RX - -  030   [8]  FF FE C5 76 00 00 09 D1
 (000.011927)  can0  RX - -  040   [8]  FF DE 20 6A 00 00 A8 EE
 (000.011931)  can0  RX - -  050   [8]  00 4E B0 00 00 8C 30 00
 (000.011932)  can0  TX - -  005   [8]  02 00 00 00 00 00 00 00
 (000.011936)  can0  RX - -  020   [8]  00 74 0A D8 00 02 B5 60
 (000.011940)  can0  RX - -  030   [8]  FF FE 4C 58 00 00 0D 17
 (000.011944)  can0  RX - -  040   [8]  FF E5 C4 13 00 00 E5 06
 (000.013030)  can0  RX - -  050   [8]  00 4E 70 00 00 8C 20 00
 (000.013031)  can0  TX - -  005   [8]  00 D1 62 70 00 00 00 00
 (000.013035)  can0  RX - -  020   [8]  01 55 AD F8 00 0B 49 60
 (000.013039)  can0  RX - -  030   [8]  FF FE 04 54 00 00 0D 17
 (000.013043)  can0  RX - -  040   [8]  FF F1 2F 93 00 00 0B 10
 (000.013047)  can0  RX - -  050   [8]  00 4E 70 00 00 8C 00 00
 (000.014430)  can0  TX - -  005   [8]  00 83 74 89 00 00 00 00
 (000.014434)  can0  RX - -  020   [8]  00 EA 21 30 FF FD 78 F0
 (000.014438)  can0  RX - -  030   [8]  FF FE 07 99 00 00 0D 17
 (000.014442)  can0  RX - -  040   [8]  00 04 62 BC 00 00 00 7E
 (000.014446)  can0  RX - -  050   [8]  00 4E 20 00 00 8C 00 00
 (000.014447)  can0  TX - -  005   [8]  00 83 74 89 00 00 00 00
 (000.015636)  can0  RX - -  020   [8]  00 8A 70 68 00 07 BE 28
 (000.015640)  can0  RX - -  030   [8]  FF FE 4F 9D 00 00 09 D2
 (000.015644)  can0  RX - -  040   [8]  00 12 2E 77 FF FF AF 7F
 (000.015648)  can0  RX - -  050   [8]  00 4E 10 00 00 8C 30 00
 (000.015649)  can0  TX - -  005   [8]  FF FA C8 C1 00 00 00 00
 (000.015653)  can0  RX - -  020   [8]  00 67 F6 50 00 04 5B 88
 (000.016738)  can0  RX - -  030   [8]  FF FE C2 2F 00 00 03 46
 (000.016742)  can0  RX - -  040   [8]  00 1B A5 18 FF FE 11 AA
 (000.016746)  can0  RX - -  050   [8]  00 4D F0 00 00 8C 10 00
 (000.016747)  can0  TX - -  005   [8]  FF 95 6A 17 00 00 00 00
 (000.016752)  can0  RX - -  020   [8]  00 36 FE 10 FF F5 0D 10
 (000.016756)  can0  RX - -  030   [8]  FF FF 55 7D 00 00 00 00
 (000.017941)  can0  RX - -  040   [8]  00 22 01 9D FF FE 37 22
 (000.017945)  can0  RX - -  050   [8]  00 4E 10 00 00 8C 10 00
 (000.017946)  can0  TX - -  005   [8]  FF 46 38 B0 00 00 00 00
 (000.017950)  can0  RX - -  020   [8]  FF F9 36 58 FF FF 41 38
 (000.017954)  can0  RX - -  030   [8]  00 00 2F 51 00 00 00 00
 (000.017958)  can0  RX - -  040   [8]  00 26 5E 7D FF FF E9 D9
 (000.019044)  can0  RX - -  050   [8]  00 4E 30 00 00 8C 20 00
 (000.019045)  can0  TX - -  005   [8]  FE F0 10 BC 00 00 00 00
 (000.019049)  can0  RX - -  020   [8]  FF B2 98 A0 FF FF 41 38
 (000.019053)  can0  RX - -  030   [8]  00 00 B2 41 00 00 00 00

[M.Heidemann]

Hello,

Thank you for providing the information.

i contacted our Linux-Developers regarding this behaviour.
This seems like a really rare case in which the way the reception is handled on the
PCAN-PCI Express FD will cause the seen reception fluctuations.
As a way to control both reactivity and the load (as in number of interrupts) the
PCAN-PCIe FD card will have two parameteres, one defines how many messages ought to
be in the reception queue before an interrupt is sent (CANFD_CTL_IRQ_CL_DEF) and the other defines how long
the PCAN-PCIe FD core will wait before sending a interrupt if at least one message is in the queue(CANFD_CTL_IRQ_TL_DEF).

Are you comfortable with building your own kernel or do you prefer a solution using our proprietary driver?

Please give us feedback regarding this

Best Regards

Marvin

[felixwidmaier]

So changing the values of these parameters might help?

I am comfortable with building the kernel myself, I anyway do this already for the preempt_rt patch.

[M.Heidemann]

Exactly,

First we should see if the behaviour is corrected and then make adjustments, if we need/want to improve perfomance further.

I would like to refer you to:

https://git.kernel.org/pub/scm/linux/ke ... ain.c#n114

for reference.

The defines previously mentioned are:

Code: Select all

/* Rx IRQ Count and Time Limits */
#define CANFD_CTL_IRQ_CL_DEF	16	/* Rx msg max nb per IRQ in Rx DMA */
#define CANFD_CTL_IRQ_TL_DEF	10	/* Time before IRQ if < CL (x100 µs) */

in line 115 and 116 respectively.

We would recommend to change CANFD_CTL_IRQ_TL_DEF to 5 as of now (For confirmation), rebuild the kernel
and check the behaviour.

Please give us feedback regarding your findings.

Best Regards

Marvin

[felixwidmaier]

It is working!

I just did a quick test so far but it seems that with CANFD_CTL_IRQ_TL_DEF = 5 the delays are mostly gone. There are still some irregularities from time to time but most of the time it is perfect:

Code: Select all

 (000.001002)  can0  RX - -  040   [8]  00 02 EF 89 00 00 00 00
 (000.001003)  can0  RX - -  040   [8]  00 02 1C 52 00 00 00 00
 (000.001002)  can0  RX - -  040   [8]  00 00 6C 13 00 00 00 00
 (000.001002)  can0  RX - -  040   [8]  00 00 03 B5 00 00 00 00
 (000.001002)  can0  RX - -  040   [8]  FF FF 0A AE 00 00 00 00
 (000.001002)  can0  RX - -  040   [8]  FF FC ED 2E 00 00 00 00
 (000.001003)  can0  RX - -  040   [8]  FF FD E2 58 00 00 00 00
 (000.001002)  can0  RX - -  040   [8]  FF FD EA D7 00 00 00 00
 (000.001002)  can0  RX - -  040   [8]  FF FE CA 76 00 00 00 00
 (000.001003)  can0  RX - -  040   [8]  FF FF F5 41 00 00 00 00
 (000.001002)  can0  RX - -  040   [8]  00 02 0D 8F 00 00 00 00
 (000.001002)  can0  RX - -  040   [8]  00 02 EF DE 00 00 00 00
 (000.000902)  can0  RX - -  040   [8]  00 03 34 EC 00 00 00 00
 (000.001203)  can0  RX - -  040   [8]  00 01 CA 2A 00 00 00 00
 (000.000803)  can0  RX - -  040   [8]  00 00 AD 99 00 00 00 00
 (000.001301)  can0  RX - -  040   [8]  00 00 05 F6 00 00 00 00
 (000.000703)  can0  RX - -  040   [8]  FF FE A8 95 FF FF 62 38
 (000.001202)  can0  RX - -  040   [8]  FF FC A3 AC FF FF F8 54
 (000.000802)  can0  RX - -  040   [8]  FF FD 5A 33 FF FF FF 97
 (000.000702)  can0  RX - -  040   [8]  FF FD 94 34 FF FF FF EF
 (000.001303)  can0  RX - -  040   [8]  FF FE C7 7E FF FF FF EF
 (000.000702)  can0  RX - -  040   [8]  FF FF F5 26 FF FF FF EF
 (000.001303)  can0  RX - -  040   [8]  00 02 0D 8F FF FF FF EF
 (000.000702)  can0  RX - -  040   [8]  00 02 EF DE FF FF FF EF
 (000.001302)  can0  RX - -  040   [8]  00 02 4E 6B FF FF FF EF
 (000.000802)  can0  RX - -  040   [8]  00 02 09 B9 FF FF FF EF
 (000.001003)  can0  RX - -  040   [8]  00 00 6B 75 FF FF FF EF
 (000.001002)  can0  RX - -  040   [8]  00 00 03 AF FF FF FF EF
 (000.001002)  can0  RX - -  040   [8]  FF FE CA CB FF FF FF EF
 (000.001103)  can0  RX - -  040   [8]  FF FE 23 AA FF FF FF EF
 (000.001003)  can0  RX - -  040   [8]  FF FF 12 90 FF FF FF EF
 (000.001001)  can0  RX - -  040   [8]  FF FF 27 D9 FF FF FF EF
 (000.001003)  can0  RX - -  040   [8]  FF FF F8 7A FF FF FF EF
 (000.001002)  can0  RX - -  040   [8]  FF FF FF B1 FF FF FF EF
 (000.001003)  can0  RX - -  040   [8]  00 02 3A 63 00 00 70 A4
 (000.001002)  can0  RX - -  040   [8]  00 02 4E 16 00 00 05 6A
 (000.001002)  can0  RX - -  040   [8]  00 01 49 9F 00 00 00 37
 (000.001002)  can0  RX - -  040   [8]  00 00 A9 2C 00 00 00 00
 (000.001003)  can0  RX - -  040   [8]  00 00 26 67 00 00 00 00
 (000.001003)  can0  RX - -  040   [8]  00 00 01 47 00 00 00 00
 (000.001002)  can0  RX - -  040   [8]  FF FE DC F5 00 00 00 00
 (000.001002)  can0  RX - -  040   [8]  FF FD 77 16 00 00 00 00
 (000.001002)  can0  RX - -  040   [8]  FF FE 77 83 00 00 00 00
 (000.001002)  can0  RX - -  040   [8]  FF FF 35 89 00 00 00 00
 (000.000902)  can0  RX - -  040   [8]  FF FF B9 13 00 00 00 00
 (000.001002)  can0  RX - -  040   [8]  FF FF FD 7E 00 00 00 00
 (000.001003)  can0  RX - -  040   [8]  00 00 CF C1 00 00 00 00
 (000.001002)  can0  RX - -  040   [8]  00 01 FC 85 00 00 00 00
 (000.001002)  can0  RX - -  040   [8]  00 02 06 E1 00 00 00 00
 (000.001003)  can0  RX - -  040   [8]  00 01 34 EC 00 00 00 00

Likewise, the motor control is working smoothly now.

So I assume the next step is to fine-tune these parameters?

[M.Heidemann]

Hello,

Thank you for the fast repsonse.

Yes, next step is fine tuning.

You can lower the values stepwise, remember however that this was implemented
to prevent heavy loads due to interrupts.
Our recommendation would be to change CANFD_CTL_IRQ_CL_DEF next.
For CANFD_CTL_IRQ_CL_DEF you could as go low as 1, which means that every message received will
cause the core to send an interrupt (As real-time as possible), but this will be very heavy load and most likely
render the rest of your system quite unresponsive.

We'd recommend to make small changes, one parameter at a time, compile and see if the kinks are ironed out.

Please report back to us with your findings.

Best Regards

Marvin

[felixwidmaier]

Okay, I will play around with this and let you know which values I end up with.

Thank you so much for your enduring help (and also everyone who might have been involved in the background)! This is something I would pretty sure never have figured out on my own.

Just some clarification question, to make sure I understand the meaning of these parameters correctly:
When the card receives the first frame, it will be stored in some queue and then it waits until either a total CANFD_CTL_IRQ_CL_DEF frames have been received or CANFD_CTL_IRQ_TL_DEF x 100µs have passed before the interrupt is triggered. I assume this interrupt is needed so that the received frames are actually forwarded to the higher level application (e.g. candump or my motor control application)?

In my case the motor board is sending a sequence of four frames every millisecond, so ideally, I would set the parameters such that the interrupt is triggered after the fourth frame is received. Does that make sense?

[M.Heidemann]

You're welcome,
we are glad we could help you with this.

I'll clearify the usage of these defines:

Code: Select all

#define CANFD_CTL_IRQ_CL_DEF	16	/* Rx msg max nb per IRQ in Rx DMA */

This threshold determines how many messages will be in the queue
until a interrupt is sent. In most configurations the frequency of messages is alot higher.
The interrupt is needed to forward the frames to the application layer.

Code: Select all

#define CANFD_CTL_IRQ_TL_DEF	10	/* Time before IRQ if < CL (x100 µs) */

This threshold determines how long to wait to send an interrupt if at least 1 frame is in the queue, this is supposed to a provide adequate update rate to the application layer in case frames have long inter-frame times / infrequent cycles.

You can set CANFD_CTL_IRQ_CL_DEF to 4, if that reflects your application. Bear in mind however that the frequency of interrupts can have an effect on overall system perfomance/responsiveness. You can use both parameters in conjunction to find a balance between response times, stability and system load.

We'd apprecaite it if you share your results with us.

Best Regards

Marvin

[felixwidmaier]

Hello,

It's been a while (I was on vacation and busy with other projects) but unfortunately this is still not fully resolved.

I haven't decided on the final configuration for the driver yet but currently I'm using

Code: Select all

CANFD_CTL_IRQ_CL_DEF 1
CANFD_CTL_IRQ_TL_DEF 5

which seems to be okay for the moment (there are no more delays and I didn't observer any issues with system responsiveness so far).
While this indeed solves the unstable timing of the communication, I unfortunately still have the problem that the communication often gets interrupted (typically during or shortly after initialisation of the motor board). I assume this to be a separate, unrelated issue?
I switched to sending the initial commands manually with `cansend` for the tests to rule out any issue with our own software.
`candump` does not show any error, it seems the communication just stops at some point.

From the (admittedly limited) information I could get from the motor board, it seems to shut down the CAN controller, apparently because transmitted frames are not acknowledged (but I'm not sure if this is the original cause or just a follow-up issue).

Interestingly, I cannot reproduce the issue always. Sometimes it fails but on some days I can run it multiple times in a row without issue.
In the hope to get more helpful information, I now connected an additional computer to the bus, using a PCAN-USB device connected in listen-only mode and monitor it using pcanview.

The first time I tried this a few days ago I did two attempts. In both cases, the communication stopped but I got different error frames:

first run:

second run:


However, when I tried again today to gather more data, I could not reproduce the issue (i.e. the communication was working).
I also tried reverting the bit timing on the motor board back to the old values with sample point at 86.7%, while keeping the config of the interface on the PC at 66.7% (i.e. intentionally using mismatching values) and even then it was working well every time. So it seems to me that the configuration of the sample point is not the critical part.

As a next step, I'll try to get better error information from the motor board itself (this will need some work on the firmware, so might take some time).
Please let me know in case you can get any insight from the error frames shown above or if in general you have suggestions on how I could proceed debugging this.

[M.Heidemann]

Well,

there errors indicate an issue with the bitrate, but by all means:

If you do not have exact information regarding the bitrate from the motor-board manufacturer,
stick with a bitrate/Sample-Points that was tested successfully by you.

The best option would be to have a accurate information regarding the used bus-timing and use
the exact same settings on both sides.

Interestingly, I cannot reproduce the issue always. Sometimes it fails but on some days I can run it multiple times in a row without issue.

Did anything change between successful and unsuccessful tests? (Did you you move anything physically? Could it be a cable defect?)

Also, could you show us a small diagram how everything is interacting and which bitrate is used with which device?