linux-rtic

An RTIC implementation for real-time Linux.

How it Works

This implementation of RTIC is based on std::thread by spawning a thread for each task priority group. Threads are initialized with SCHED_FIFO real-time policy. Task priorities correspond 1:1 to Linux priorities and usually have a range of 1-99.

Scheduling

Scheduling of tasks is done by futex-queue, which cleverly utilizes futex syscall to wait on both immediate and scheduled (timed) tasks on a single syscall. No timer thread (and additional context switching) is required.

Resource Locking

Original cortex-m-rtic uses Stack Resource Policy (SRP), but it is difficult to emulate in userspace Linux. Firstly, setting thread priority for each lock/unlock involves an expensive syscall (~10us on Raspberry Pi 4). Secondly, setting thread priority does not guarantee that lower priority thread will not run. Lower priority thread can be executed on a different core, or when higher priority thread is suspended (i.e. I/O syscall). While it is possible to fix memory safety issues by a backup synchronisation mechanism (mutex), the syscall overhead is too high for real-time applications.

To solve the issue, a pcp-mutex library was written, which implements Original Priority Ceiling Protocol (OPCP). This allows preserving two important properties of SRP: bounding priority inversion and statically preventing deadlocks. This mutex is lock-free in the fast path. Technical details are in the pcp-mutex README.

Other Notes

Scheduling tasks in userspace threads is slow due to context switching overhead (~10us on Raspberry Pi 4) and other approaches were explored:

• POSIX signals are used in the older linux-rtfm implementation. They are faster than thread context switching, however, tasks are limited to reentrant (signal safe) functions only, which forces to use no_std. Also, resource locking slower, because of signal masking syscalls.
• Kernel threads are only marginally faster as most of the overhead seems to be in the scheduler itself. So losing userspace safety and std library didn't seem worth it.
• Hard interrupt context would be closest to what cortex-m-rtic does, but Linux does not support interrupt prioritization (only IRQ threads have priorities) and would require major kernel modifications.

Examples

Running examples requires Linux with PREEMPT-RT patched kernel for SCHED_FIFO and root privileges. This requirement can be lifted by compiling with --no-default-features, but then all tasks will share the same priority.

Build:

cargo build --release --example priority_inversion

Run (requires sudo for sched_setscheduler syscall):

sudo target/release/examples/priority_inversion

Single core:

sudo taskset -c 1 target/release/examples/priority_inversion

No real-time priorities:

cargo run --release --example priority_inversion --no-default-features

Tips to Make Real-Time More Real

• Apply PREEMPT-RT kernel patch and compile kernel with CONFIG_PREEMPT_RT_FULL to reduce non-preemptable sections in the kernel.
• Disable dynamic CPU frequency scaling. Either in kernel config or with cpufreq-set -g performance.
• Use isolcpus kernel parameter to run RTIC on an isolated core.
• Ensure that peripheral process (i.e. spi0) is scheduled with real-time priority: sudo chrt -f -p 50 $(pidof spi0).
• Try to limit scheduling between different priority tasks to reduce context switching overhead.
• Watch A Checklist for Writing Linux Real-Time Applications - John Ogness, Linutronix GmbH

Credits

This work was done as a part of my Thesis at University of Twente.