2

I am running Ubuntu 18 on several servers with the low latency kernel.

Previously I have used chrt to raise the priority of interrupt threads for the interrupts related to a specific network interface.

Now, on new installations of Ubuntu 22, with the low latency kernel, I no longer see the interrupts for the network cards running as threads and so I cannot increase the priority of those with chrt.

What has changed, and is there a way to increase network card interrupt priority on Ubuntu 22 with lowlatenct kernel?

The reason I am doing this is because I have several instances of an application that syncs small amounts of data with other computers on a 10 ms interval which must not be delayed.

I did find this information for UbuntuStudio (although it is a bit old, from 2014) about the threadirqs kernel parameter, and I have confirmed that the kernel on my newer servers is built with CONFIG_IRQ_FORCED_THREADING=y, but not with CONFIG_IRQ_FORCED_THREADING_DEFAULT=y and not CONFIG_CMDLINE_BOOL=y and CONFIG_CMDLINE="threadirqs".

Is it as simple as booting the kernel with the threadirqs parameter, and if so, when and why did the lowlatency kernel change that it does not boot with threadirqs by default?

Details

The servers where my previous solution works run Ubuntu 18.04.3 LTS, with kernel 4.15.0-123-lowlatency

These are the steps I use to raise the priority of the network interface interrupts:

  1. I run grep eno1 /proc/interrupts to get the list of interrupts for network interface eno1

Output:

 46: 19627 486510106 1600726600 752055110 1667993661 1758831968 1471240837 895245906 658188895 458440425 918559817 1494584384 1650824561 1492836078 1214143068 920349210 IR-PCI-MSI 5767169-edge i40e-eno1-TxRx-0 47: 782983498 3867361290 1350470347 1488832628 1644524179 1306935370 2202471485 885906069 266079923 769050734 767000710 1199122348 1933889985 2475859304 880359145 1207562098 IR-PCI-MSI 5767170-edge i40e-eno1-TxRx-1 48: 487247958 557149515 2436993704 1327150410 2327637329 1692350382 1160439225 1234994234 425076063 640500055 1555726813 1469038634 2133556615 1260984085 1076406628 1141316582 IR-PCI-MSI 5767171-edge i40e-eno1-TxRx-2 49: 1143028283 802247216 842065588 1507113921 1450442442 1285188073 1414989498 1386700444 961900607 949612962 1176201048 996374473 1392801596 1501055021 1244839988 1568153886 IR-PCI-MSI 5767172-edge i40e-eno1-TxRx-3 50: 668676162 585775185 627035622 560691459 545566733 534335154 675644369 619390835 602253370 611090233 610683285 552978104 558929823 827257425 555723069 751315724 IR-PCI-MSI 5767173-edge i40e-eno1-TxRx-4 51: 293791983 293018539 290310300 284911609 302624294 171231086 295128963 277943781 299153596 305718023 306559090 265176440 265877422 330072276 275946899 322208126 IR-PCI-MSI 5767174-edge i40e-eno1-TxRx-5 52: 191406211 164689451 184092472 163124351 197416416 147238980 150711931 166047824 174857188 176597379 163302040 163989899 169136092 206831749 154774311 195388230 IR-PCI-MSI 5767175-edge i40e-eno1-TxRx-6 53: 154062755 152443945 151052334 133925954 161462702 130698860 164068832 88730046 146570114 152839498 140082079 134023798 139525722 178502808 136857972 164157076 IR-PCI-MSI 5767176-edge i40e-eno1-TxRx-7 54: 689819152 457848749 1706899545 1350792167 1009016882 750940295 1280794513 1964042727 4293839311 604650430 1394985726 1390827662 1403478542 1032557167 1258596213 1421573353 IR-PCI-MSI 5767177-edge i40e-eno1-TxRx-8 55: 561179546 1569329752 1142403865 1952025068 1831162021 1786111309 1819812537 1726432246 809350013 3510981566 1252176327 1188689474 2184195381 1689746731 1808756473 1998075218 IR-PCI-MSI 5767178-edge i40e-eno1-TxRx-9 60: 922953009 535440685 1433001681 1370014373 1542680223 1564958146 1905615201 2245129577 606607580 751437652 653813572 1648944839 1339662143 2748851364 1328493925 2197584768 IR-PCI-MSI 5767179-edge i40e-eno1-TxRx-10 61: 915173255 898646194 818454435 975220367 1436058787 1258590833 1222669978 1475644158 955884853 1048337605 1028501060 1229229809 1375764724 1619404100 1174614194 1626915747 IR-PCI-MSI 5767180-edge i40e-eno1-TxRx-11 62: 663129689 560699700 659990638 564915959 716701151 537167244 709668792 653540652 643175469 609563694 657141242 551578294 496276894 826134713 584425798 782807812 IR-PCI-MSI 5767181-edge i40e-eno1-TxRx-12 63: 301178832 272290237 297349011 271058986 300930906 219283819 308344403 269722445 297793600 300198211 271567742 257075242 257586638 275439840 258617424 331559250 IR-PCI-MSI 5767182-edge i40e-eno1-TxRx-13 64: 161271542 160600034 162911736 135003940 159159336 126317895 145054883 142236108 153342852 158784173 149385736 130987700 145150304 179008100 100121849 168255307 IR-PCI-MSI 5767183-edge i40e-eno1-TxRx-14 65: 174086138 162484310 163432318 142315712 164872478 136966716 168251078 137840006 150045153 156520410 143270315 134481889 152034193 184135755 140295604 167224365 IR-PCI-MSI 5767184-edge i40e-eno1-TxRx-15
  1. I run ps -el | grep irq to find the pid for each irq thread for eno1

Output: (I only show the relevant lines matching irqs 46-65, there are a lot more)

1 R 0 1255 2 1 9 - - 0 - ? 6-01:09:56 irq/46-i40e-eno 1 S 0 1256 2 1 9 - - 0 - ? 5-13:32:39 irq/47-i40e-eno 1 S 0 1257 2 0 9 - - 0 - ? 5-04:09:18 irq/48-i40e-eno 1 S 0 1259 2 0 9 - - 0 - ? 3-12:50:35 irq/49-i40e-eno 1 S 0 1261 2 0 9 - - 0 - ? 1-19:03:44 irq/50-i40e-eno 1 S 0 1262 2 0 9 - - 0 - ? 19:55:05 irq/51-i40e-eno 1 S 0 1263 2 0 9 - - 0 - ? 11:48:06 irq/52-i40e-eno 1 S 0 1264 2 0 9 - - 0 - ? 09:59:36 irq/53-i40e-eno 1 S 0 1265 2 0 9 - - 0 - ? 5-09:23:00 irq/54-i40e-eno 1 S 0 1266 2 0 9 - - 0 - ? 4-19:20:49 irq/55-i40e-eno 1 S 0 1267 2 0 9 - - 0 - ? 4-19:24:25 irq/60-i40e-eno 1 S 0 1268 2 0 9 - - 0 - ? 3-10:21:49 irq/61-i40e-eno 1 S 0 1269 2 0 9 - - 0 - ? 1-20:29:13 irq/62-i40e-eno 1 S 0 1270 2 0 9 - - 0 - ? 19:24:51 irq/63-i40e-eno 1 S 0 1271 2 0 9 - - 0 - ? 10:11:11 irq/64-i40e-eno 1 S 0 1273 2 0 9 - - 0 - ? 10:31:14 irq/65-i40e-eno
  1. For each of the irq threads for this network interface, I run chrt -p 80 <thread-pid>, e.g. chrt -p 80 1255 for irq/46.

I actually use a combination of grep, awk and xargs to perform these three steps in one:

grep $NIC /proc/interrupts \ | awk '{print substr($1, 0, length($1) - 1);}' \ | xargs -I% sh -c "ps -e | grep irq/%-" \ | awk '{print $1;}' \ | xargs -I% sh -c "echo irq PID %; chrt -p 80 %" 
me@myserver:~$ grep CONFIG_IRQ_FORCED_THREADING /boot/config-4.15.0-123-lowlatency CONFIG_IRQ_FORCED_THREADING=y CONFIG_IRQ_FORCED_THREADING_DEFAULT=y me@myserver:~$ grep CONFIG_CMDLINE /boot/config-4.15.0-123-lowlatency CONFIG_CMDLINE_PARTITION=y # CONFIG_CMDLINE_BOOL is not set

The newer servers where this solution does not work run Ubuntu 22.04.2 LTS with kernel 5.15.0-79-lowlatency

The results of the first two steps:

  1. grep eno4 /proc/interrupts

Output:

 240: 1 0 0 0 0 0 0 0 0 0 0 0 0 0 1030184410 0 IR-PCI-MSI 5249025-edge i40e-eno4-TxRx-0 241: 0 0 0 0 0 0 0 0 0 0 0 0 1342657199 0 0 0 IR-PCI-MSI 5249026-edge i40e-eno4-TxRx-1 242: 0 0 1 1007985709 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 5249027-edge i40e-eno4-TxRx-2 243: 0 995307412 0 21 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 5249028-edge i40e-eno4-TxRx-3 244: 0 0 0 0 16 0 1231037185 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 5249029-edge i40e-eno4-TxRx-4 245: 0 0 0 0 782521466 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 5249030-edge i40e-eno4-TxRx-5 246: 1035173010 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 5249031-edge i40e-eno4-TxRx-6 247: 0 0 0 0 0 0 0 0 0 0 0 930082293 0 0 0 0 IR-PCI-MSI 5249032-edge i40e-eno4-TxRx-7 248: 0 0 0 0 0 0 0 762559502 1 0 0 0 0 0 0 0 IR-PCI-MSI 5249033-edge i40e-eno4-TxRx-8 249: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 793943748 0 IR-PCI-MSI 5249034-edge i40e-eno4-TxRx-9 250: 0 0 0 0 0 0 0 0 0 0 12 0 998551326 0 0 0 IR-PCI-MSI 5249035-edge i40e-eno4-TxRx-10 251: 0 0 0 0 0 0 0 0 0 0 975747642 1 0 0 0 0 IR-PCI-MSI 5249036-edge i40e-eno4-TxRx-11 252: 0 1342889081 0 0 0 0 0 0 0 0 0 0 216 0 0 0 IR-PCI-MSI 5249037-edge i40e-eno4-TxRx-12 253: 0 0 0 0 0 0 0 0 0 0 0 0 0 14 0 818489097 IR-PCI-MSI 5249038-edge i40e-eno4-TxRx-13 254: 0 0 0 0 1177974058 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 5249039-edge i40e-eno4-TxRx-14 255: 0 0 942945250 0 0 0 0 0 0 0 0 0 0 0 0 27 IR-PCI-MSI 5249040-edge i40e-eno4-TxRx-15
  1. ps -el | grep irq

Output:

1 S 0 14 2 0 70 -10 - 0 - ? 00:06:33 ksoftirqd/0 1 S 0 23 2 0 70 -10 - 0 - ? 00:14:18 ksoftirqd/1 1 S 0 29 2 0 70 -10 - 0 - ? 00:30:44 ksoftirqd/2 1 S 0 35 2 0 70 -10 - 0 - ? 00:03:54 ksoftirqd/3 1 S 0 41 2 0 70 -10 - 0 - ? 00:09:49 ksoftirqd/4 1 S 0 47 2 0 70 -10 - 0 - ? 00:02:08 ksoftirqd/5 1 S 0 53 2 0 70 -10 - 0 - ? 00:06:32 ksoftirqd/6 1 S 0 59 2 0 70 -10 - 0 - ? 00:05:31 ksoftirqd/7 1 S 0 65 2 0 70 -10 - 0 - ? 00:03:34 ksoftirqd/8 1 S 0 71 2 0 70 -10 - 0 - ? 00:00:29 ksoftirqd/9 1 S 0 77 2 0 70 -10 - 0 - ? 00:04:11 ksoftirqd/10 1 S 0 83 2 0 70 -10 - 0 - ? 00:02:47 ksoftirqd/11 1 S 0 89 2 0 70 -10 - 0 - ? 00:13:01 ksoftirqd/12 1 S 0 95 2 0 70 -10 - 0 - ? 00:01:02 ksoftirqd/13 1 S 0 101 2 0 70 -10 - 0 - ? 00:06:02 ksoftirqd/14 1 S 0 107 2 0 70 -10 - 0 - ? 00:06:08 ksoftirqd/15 1 I 0 197 2 0 60 -20 - 0 - ? 00:00:00 vfio-irqfd-clea 4 S 0 924 1 0 80 0 - 20764 - ? 00:10:46 irqbalance

Kernel build parameters:

me@myserver:~$ grep CONFIG_IRQ_FORCED_THREADING /boot/config-5.15.0-79-lowlatency CONFIG_IRQ_FORCED_THREADING=y me@myserver:~$ grep CONFIG_CMDLINE_ /boot/config-5.15.0-79-lowlatency # CONFIG_CMDLINE_BOOL is not set CONFIG_CMDLINE_PARTITION=y
Improve this question

0

Reset to default

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.