Xenomai is a real-time OS using Linux as a background task. Linux is preempted as a simple task. With Xenomai, the idea of impossible preemption, handlers, is no longer valid. Tasks, handled by Xenomai, are guaranteed to have a hard real-time execution.
But running two Operating Systems has its load of issues. First you need two schedulers, which leads to issues again : how to share the hardware, how to make Xenomai’s and Linux’s tasks interact with each other.
As beginers ourselves, we will try to popularize what we can fetch from challenging tutorials and technical data about Xenomai.
Our Setup
This tutorial was tested on two laptops with different specifications :
Gigabyte P55WV7
- 16GB of RAM (DDR4)
- Intel Core i7-7700HQ CPU @ 2.80 GHz
- GP106M [GeForce GTX 1060 Mobile]
- Linux 5.3.0-7648-generic Kernel
- Ubuntu 18.04
- Architecture x86_64
ACER Aspire F5-571G
- 12GB of RAM (DDR4)
- Intel core i3-50005U CPU @ 2.00GHz
- HD Graphics 5500
- Linux 5.3.0-53-generic Kernel
- Ubuntu 18.04
- x86_64
Requirements :
For the rest of the tutorial I will assume your relative path ( i.e. ~/ ) is home/me. “me” being the name of the user for your machine.
First, make sure every packages you have on your machine are updated and clean. I tried to boot a Xenomai co-kernel on a dirty base once and it just didn’t work very well. Then create a working directory in which you will do everything. Let’s call it RTOS :
$ apt update
$ mkdir rtos
The Linux kernel from distributions such as Ubuntu being modified and somewhat polluted, you will need a fresh vanilla version of the Linux kernel here :
https://www.kernel.org/. We chose the 4.19.117 version, which is stable, recent, supported by Xenomai and maintained :
$ wget https://cdn.kernel.org/pub/linux/kernel/v4.x/linux-4.19.117.tar.xz
$ tar xvJf linux-4.19.117.tar.xz
We will now start working with Xenomai 3 :
$ cd ~/rtos
$ git clone git://git.xenomai.org/xenomai-3.git
Once the repository is cloned, make sure to bootstrap the autoconf system in the top-level directory (/home/me/rtos/xenomai-3). In order to do this, you will need the GNU autotools installed on your workstation.
$ apt install libncurses5 libncurses5-dev build-essential kernel-package fakeroot libssl-dev
$ cd ~/rtos/xenomai-3
$ ./scripts/bootstrap
You also need the interrupt pipeline (aka I-pipe) patch to apply against the kernel tree. The patch is already in the Xenomai package, but more recent ones can be available here This parameter can be omitted if the I-pipe has already been patched in, or the script shall suggest an appropriate one.
You will have to choose the patch corresponding to the linux kernel you chose (4.19.x here) :
cd ~/rtos
wget https://xenomai.org/downloads/ipipe/v4.x/x86/ipipe-core-4.19.114-cip24-x86-12.patch
Cobalt kernel
The first step is to build it as part of the target kernel ( being your vanilla linux ). The Xenomai 3 team already prepared a script to complete this step properly :
$ cd ~/rtos/xenomai-3
$ scripts/prepare-kernel.sh --linux=../linux-4.19.117 --ipipe=../ipipe-core-4.19.114-cip24-x86-12.patch --arch=x86_64`
Hunk #3 succeeded at 83 (offset -1 lines).
Hunk #4 succeeded at 96 (offset -3 lines).
Hunk #5 succeeded at 685 (offset -17 lines).
Hunk #6 succeeded at 912 (offset -17 lines).
checking file arch/x86/entry/common.c
...
...
...
checking file lib/bust_spinlocks.c
checking file lib/dump_stack.c
checking file lib/ioremap.c
checking file lib/smp_processor_id.c
checking file mm/memory.c
checking file mm/mlock.c
checking file mm/mmap.c
Hunk #2 succeeded at 2827 (offset -12 lines).
checking file mm/mmu_context.c
checking file mm/mprotect.c
checking file mm/vmalloc.c
You preferably do not get any Hunk ... messages. It means the patch was moved to be applied correctly. If you look at my kernel version and my patch version, they differ by 0.0.1. Ideally you will use the same version for both.
Kernel Configuration
You can now start configurating your Xenomai kernel :
$ cd ~/rtos/linux-4.19.117
$ make menuconfig
There are few things that needs to be removed in the kernel configuration to boot a Xenomai co-kernel. The kernel configuration options to check are listed below. You can edit directly the .config file, although I wouldn’t recommand it (mainly because I don’t trust myself tweaking something if I don’t understand what it depends on and what depends on it.) see http://www.tldp.org/ for more configuration details
- VISORBUS driver - Disable (might fail when building if activated)
- CONFIG_CPU_FREQ - Disable (in Power management->CPU Freq Scaling)
- CONFIG_CPU_IDLE - Disable (in Power management -> CPU idle PM support)
- CONFIG_ACPI_PROCESSOR - Disable(in Power management->ACPI->processor)
- CONFIG_INPUT_PCSPKR - Disable (I changed that directly in the .config file after being done with menuconfig because I couldn’t findthe option)
- DO NOT DISABLE MSI - It is now obsolete cf this. Under Real-time Subsystem, you may want to enable (mark with M to build as a kernel module) the real-time serial driver under Drivers.These and more Xenomai installation details can be found at official Xenomai install guide and explains how to configure the installation
- CONFIG_MIGRATION - Disable (in Memory management)
Tips for the GUi : use ‘/’ to research. If a option is marked as “-*-“ it means it’s selected by another one. Be careful when you deactivate the options that selects those above. Note that it is not good usage to go from the default kernel configuration and trim it down. What you want is go from a configuration as basic as possible and tailor it to what you need, it will be more time efficient in terms of compilation, and more power efficient for your target.
Build and install :
$ make -j8 && make -j8 modules_install install
Without modifying the configuration of modules, the kernel couldn’t build. The visorbus driver had some issues with defines in the include folder. Because the module was unnecessary to our end we simply deactivated the driver in the configuration of the kernel. It then built without issues.
Installing Xenomai Libraries
$ cd ~/rtos/xenomai-3
$ ./configure && make && make install
$ reboot
You now need to boot the right kernel. Depending on your grub theme or options or how you configured it things might change. But you need to run the kernel that was add when you built it in the previous step.
For more information about grub, refer to the Ubuntu installation post. Very basically, the make install command will end with the addition of your freshly built real-time kernel to your grub options, which are available in the bootloader menu on start-up, in the Advanced options for Ubuntu section.
I wouldn’t recommand putting Xenomai as the default grub option because… I did. And it was painful to come back from that. The freshly built kernel should be available in your grub bootloading options. Hopefully you managed to go this far without trouble, because we sure didn’t. Personally I tailored the kernel configuration over and over again until it finally booted. It was unbootable because it was missing the EFI option. Then it froze at the initramfs stage. Then it only took a while at the initramfs stage… to freeze at the “waiting for a pressed key” stage. I tried with different kernels and 4.19.117 was the most stable one.
You may encounter issues regarding NVIDIA drivers when booting (i.e. freeze at login) if so : Press CTRL+ALT+F2 on login page, and then run the following :
sudo add-apt-repository ppa:graphics-drivers/ppa
sudo apt-get update
sudo apt-get install nvidia-390
Testing and Debugging
It took time to tailor the configuration, and I had to update my packages and repair broken onces from the recovery mode terminal, but when it finally boots, you want to test that Xenomai is indeed there :
$ dmesg | grep -i xenomai
[ 8.267123] [Xenomai] scheduling class idle registered.
[ 8.267124] [Xenomai] scheduling class rt registered.
[ 8.267144] [Xenomai] SMI-enabled chipset found, but SMI workaround disabled
(see xenomai.smi parameter). You might encounter
[ 8.267171] I-pipe: head domain Xenomai registered.
[ 8.267565] [Xenomai] Cobalt v3.1
If the installation went smoothly, you should now be able to test for latency issues. This step will measure (by default) the latency of a periodic task. More explicitely, it’s measuring the difference in microseconds between the moment a task is expected to be active and the actual moment it is active. This test should be ran over a long time and in stressfull conditions.
To emulate stressfullness :
$ dd if=/dev/zero of=/dev/null
To run the tests :
$ sudo /usr/xenomai/bin/latency
== Sampling period: 100 us
== Test mode: periodic user-mode task
== All results in microseconds
warming up...
RTT| 00:00:01 (periodic user-mode task, 100 us period, priority 99)
RTH|----lat min|----lat avg|----lat max|-overrun|---msw|---lat best|--lat worst
RTD| 0.384| 0.715| 3.082| 0| 0| 0.384| 3.082
RTD| 0.259| 0.719| 5.336| 0| 0| 0.259| 5.336
RTD| 0.524| 0.722| 5.156| 0| 0| 0.259| 5.336
RTD| 0.453| 0.723| 3.669| 0| 0| 0.259| 5.336
RTD| 0.190| 0.723| 4.989| 0| 0| 0.190| 5.336
RTD| 0.570| 0.720| 5.116| 0| 0| 0.190| 5.336
RTD| 0.334| 0.725| 7.108| 0| 0| 0.190| 7.108
RTD| 0.409| 0.724| 5.942| 0| 0| 0.190| 7.108
RTD| 0.504| 0.717| 14.649| 0| 0| 0.190| 14.649
RTD| 0.436| 0.720| 26.437| 0| 0| 0.190| 26.437
RTD| 0.237| 0.722| 56.066| 0| 0| 0.190| 56.066
RTD| 0.191| 0.725| 21.825| 0| 0| 0.190| 56.066
RTD| 0.498| 0.723| 5.421| 0| 0| 0.190| 56.066
RTD| 0.321| 0.722| 5.381| 0| 0| 0.190| 56.066
RTD| 0.563| 0.722| 5.407| 0| 0| 0.190| 56.066
RTD| 0.193| 0.720| 5.898| 0| 0| 0.190| 56.066
RTD| 0.566| 0.724| 4.669| 0| 0| 0.190| 56.066
RTD| 0.159| 0.729| 41.120| 0| 0| 0.159| 56.066
RTD| 0.450| 0.723| 33.712| 0| 0| 0.159| 56.066
RTD| 0.271| 0.743| 95.302| 0| 0| 0.159| 95.302
RTD| 0.338| 0.735| 11.523| 0| 0| 0.159| 95.302
RTT| 00:00:22 (periodic user-mode task, 100 us period, priority 99)
RTH|----lat min|----lat avg|----lat max|-overrun|---msw|---lat best|--lat worst
RTD| 0.395| 0.742| 22.627| 0| 0| 0.159| 95.302
RTD| 0.392| 0.722| 8.102| 0| 0| 0.159| 95.302
RTD| 0.479| 0.725| 3.173| 0| 0| 0.159| 95.302
RTD| 0.246| 0.721| 5.015| 0| 0| 0.159| 95.302
RTD| 0.562| 0.725| 5.981| 0| 0| 0.159| 95.302
RTD| 0.504| 0.726| 6.034| 0| 0| 0.159| 95.302
RTD| 0.434| 0.728| 38.290| 0| 0| 0.159| 95.302
RTD| 0.135| 0.727| 93.103| 0| 0| 0.135| 95.302
The worst case gives a latency of 95µs. Over 100µs, the kernel should be investigated. You should expect a latency between 10µs and 50µs on a “traditional” x86 platform.
If the latency test runs, but the values are unexpectedly high :
- Make sure to read the troubleshooting guide regarding kernel configuration from the official Xenomai wiki.
- If running on a Raspberry Pi, make sure you read this
- If your high latencies occur while running the X server, try disabling hardware acceleration by modifying the xorg.conf by following those steps :
$ sudo apt install xserver-xorg-video-fbdev
$ nano /etc/X11/xorg.conf
Section "Device"
Identifier "Card0"
Driver "fbdev"
EndSection
Don’t hesitate to contact us if you have issues, we will be glad to (try) help you.
References
StackOverflow (March 2017). “Installation Steps for Xenomai 3 on Ubuntu 16.04”