Real Time For the Masses

A concurrency framework for building real time systems

This book contains user level documentation for the Real Time For the Masses (RTFM) framework. The API reference can be found here.

There is a translation of this book in Russian.

HEADS UP This is an alpha pre-release; there may be breaking changes in the API and semantics before a proper release is made.

• Tasks as the unit of concurrency ¹. Tasks can be event triggered (fired in response to asynchronous stimuli) or spawned by the application on demand.

• Message passing between tasks. Specifically, messages can be passed to software tasks at spawn time.

• A timer queue ². Software tasks can be scheduled to run at some time in the future. This feature can be used to implement periodic tasks.

• Support for prioritization of tasks and, thus, preemptive multitasking.

• Efficient and data race free memory sharing through fine grained priority based critical sections ¹.

• Deadlock free execution guaranteed at compile time. This is an stronger guarantee than what's provided by the standard Mutex abstraction.

• Minimal scheduling overhead. The task scheduler has minimal software footprint; the hardware does the bulk of the scheduling.

• Highly efficient memory usage: All the tasks share a single call stack and there's no hard dependency on a dynamic memory allocator.

• All Cortex-M devices are supported. The core features of RTFM are supported on all Cortex-M devices. The timer queue is currently only supported on ARMv7-M devices.

• This task model is amenable to known WCET (Worst Case Execution Time) analysis and scheduling analysis techniques. (Though we haven't yet developed Rust friendly tooling for that.)

• Rust 1.36.0+

• Applications must be written using the 2018 edition.

This crate is based on the RTFM language created by the Embedded Systems group at Luleå University of Technology, led by Prof. Per Lindgren.

All source code (including code snippets) is licensed under either of

• Apache License, Version 2.0 (LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0)
• MIT license (LICENSE-MIT or https://opensource.org/licenses/MIT)

at your option.

The written prose contained within the book is licensed under the terms of the Creative Commons CC-BY-SA v4.0 license (LICENSE-CC-BY-SA or https://creativecommons.org/licenses/by-sa/4.0/legalcode).

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be licensed as above, without any additional terms or conditions.

This part of the book introduces the Real Time For the Masses (RTFM) framework to new users by walking them through examples of increasing complexity.

All examples in this part of the book can be found in the GitHub repository of the project, and most of the examples can be run on QEMU so no special hardware is required to follow along.

To run the examples on your laptop / PC you'll need the qemu-system-arm program. Check the embedded Rust book for instructions on how to set up an embedded development environment that includes QEMU.

This is the smallest possible RTFM application:

# #![allow(unused_variables)]
#fn main() {
//! examples/smallest.rs

#![deny(unsafe_code)]
#![deny(warnings)]
#![no_main]
#![no_std]

// panic-handler crate
extern crate panic_semihosting;

use rtfm::app;

#[app(device = lm3s6965)]
const APP: () = {
    #[init]
    fn init(_: init::Context) {}
};

#}

All RTFM applications use the app attribute (#[app(..)]). This attribute must be applied to a const item that contains items. The app attribute has a mandatory device argument that takes a path as a value. This path must point to a peripheral access crate (PAC) generated using svd2rust v0.14.x. The app attribute will expand into a suitable entry point so it's not required to use the cortex_m_rt::entry attribute.

ASIDE: Some of you may be wondering why we are using a const item as a module and not a proper mod item. The reason is that using attributes on modules requires a feature gate, which requires a nightly toolchain. To make RTFM work on stable we use the const item instead. When more parts of macros 1.2 are stabilized we'll move from a const item to a mod item and eventually to a crate level attribute (#![app]).

Within the pseudo-module the app attribute expects to find an initialization function marked with the init attribute. This function must have signature fn(init::Context) [-> init::LateResources].

This initialization function will be the first part of the application to run. The init function will run with interrupts disabled and has exclusive access to Cortex-M and device specific peripherals through the core and device variables fields of init::Context. Not all Cortex-M peripherals are available in core because the RTFM runtime takes ownership of some of them -- for more details see the rtfm::Peripherals struct.

static mut variables declared at the beginning of init will be transformed into &'static mut references that are safe to access.

The example below shows the types of the core and device variables and showcases safe access to a static mut variable.

# #![allow(unused_variables)]
#fn main() {
//! examples/init.rs

#![deny(unsafe_code)]
#![deny(warnings)]
#![no_main]
#![no_std]

extern crate panic_semihosting;

use cortex_m_semihosting::{debug, hprintln};

#[rtfm::app(device = lm3s6965)]
const APP: () = {
    #[init]
    fn init(c: init::Context) {
        static mut X: u32 = 0;

        // Cortex-M peripherals
        let _core: rtfm::Peripherals = c.core;

        // Device specific peripherals
        let _device: lm3s6965::Peripherals = c.device;

        // Safe access to local `static mut` variable
        let _x: &'static mut u32 = X;

        hprintln!("init").unwrap();

        debug::exit(debug::EXIT_SUCCESS);
    }
};

#}

Running the example will print init to the console and then exit the QEMU process.

$ cargo run --example init
init

A function marked with the idle attribute can optionally appear in the pseudo-module. This function is used as the special idle task and must have signature fn(idle::Context) - > !.

When present, the runtime will execute the idle task after init. Unlike init, idle will run with interrupts enabled and it's not allowed to return so it runs forever.

When no idle function is declared, the runtime sets the SLEEPONEXIT bit and then sends the microcontroller to sleep after running init.

Like in init, static mut variables will be transformed into &'static mut references that are safe to access.

The example below shows that idle runs after init.

# #![allow(unused_variables)]
#fn main() {
//! examples/idle.rs

#![deny(unsafe_code)]
#![deny(warnings)]
#![no_main]
#![no_std]

extern crate panic_semihosting;

use cortex_m_semihosting::{debug, hprintln};

#[rtfm::app(device = lm3s6965)]
const APP: () = {
    #[init]
    fn init(_: init::Context) {
        hprintln!("init").unwrap();
    }

    #[idle]
    fn idle(_: idle::Context) -> ! {
        static mut X: u32 = 0;

        // Safe access to local `static mut` variable
        let _x: &'static mut u32 = X;

        hprintln!("idle").unwrap();

        debug::exit(debug::EXIT_SUCCESS);

        loop {}
    }
};

#}

$ cargo run --example idle
init
idle

Just like you would do with the cortex-m-rt crate you can use the interrupt and exception attributes within the app pseudo-module to declare interrupt and exception handlers. In RTFM, we refer to interrupt and exception handlers as hardware tasks.

# #![allow(unused_variables)]
#fn main() {
//! examples/interrupt.rs

#![deny(unsafe_code)]
#![deny(warnings)]
#![no_main]
#![no_std]

extern crate panic_semihosting;

use cortex_m_semihosting::{debug, hprintln};
use lm3s6965::Interrupt;

#[rtfm::app(device = lm3s6965)]
const APP: () = {
    #[init]
    fn init(_: init::Context) {
        // Pends the UART0 interrupt but its handler won't run until *after*
        // `init` returns because interrupts are disabled
        rtfm::pend(Interrupt::UART0);

        hprintln!("init").unwrap();
    }

    #[idle]
    fn idle(_: idle::Context) -> ! {
        // interrupts are enabled again; the `UART0` handler runs at this point

        hprintln!("idle").unwrap();

        rtfm::pend(Interrupt::UART0);

        debug::exit(debug::EXIT_SUCCESS);

        loop {}
    }

    #[interrupt]
    fn UART0(_: UART0::Context) {
        static mut TIMES: u32 = 0;

        // Safe access to local `static mut` variable
        *TIMES += 1;

        hprintln!(
            "UART0 called {} time{}",
            *TIMES,
            if *TIMES > 1 { "s" } else { "" }
        )
        .unwrap();
    }
};

#}

$ cargo run --example interrupt
init
UART0 called 1 time
idle
UART0 called 2 times

So far all the RTFM applications we have seen look no different that the applications one can write using only the cortex-m-rt crate. In the next section we start introducing features unique to RTFM.

One of the limitations of the attributes provided by the cortex-m-rt crate is that sharing data (or peripherals) between interrupts, or between an interrupt and the entry function, requires a cortex_m::interrupt::Mutex, which always requires disabling all interrupts to access the data. Disabling all the interrupts is not always required for memory safety but the compiler doesn't have enough information to optimize the access to the shared data.

The app attribute has a full view of the application thus it can optimize access to static variables. In RTFM we refer to the static variables declared inside the app pseudo-module as resources. To access a resource the context (init, idle, interrupt or exception) one must first declare the resource in the resources argument of its attribute.

In the example below two interrupt handlers access the same resource. No Mutex is required in this case because the two handlers run at the same priority and no preemption is possible. The SHARED resource can only be accessed by these two handlers.

# #![allow(unused_variables)]
#fn main() {
//! examples/resource.rs

#![deny(unsafe_code)]
#![deny(warnings)]
#![no_main]
#![no_std]

extern crate panic_semihosting;

use cortex_m_semihosting::{debug, hprintln};
use lm3s6965::Interrupt;

#[rtfm::app(device = lm3s6965)]
const APP: () = {
    // A resource
    static mut SHARED: u32 = 0;

    #[init]
    fn init(_: init::Context) {
        rtfm::pend(Interrupt::UART0);
        rtfm::pend(Interrupt::UART1);
    }

    #[idle]
    fn idle(_: idle::Context) -> ! {
        debug::exit(debug::EXIT_SUCCESS);

        // error: `SHARED` can't be accessed from this context
        // SHARED += 1;

        loop {}
    }

    // `SHARED` can be access from this context
    #[interrupt(resources = [SHARED])]
    fn UART0(mut c: UART0::Context) {
        *c.resources.SHARED += 1;

        hprintln!("UART0: SHARED = {}", c.resources.SHARED).unwrap();
    }

    // `SHARED` can be access from this context
    #[interrupt(resources = [SHARED])]
    fn UART1(mut c: UART1::Context) {
        *c.resources.SHARED += 1;

        hprintln!("UART1: SHARED = {}", c.resources.SHARED).unwrap();
    }
};

#}

$ cargo run --example resource
UART0: SHARED = 1
UART1: SHARED = 2

The priority of each handler can be declared in the interrupt and exception attributes. It's not possible to set the priority in any other way because the runtime takes ownership of the NVIC peripheral thus it's also not possible to change the priority of a handler / task at runtime. Thanks to this restriction the framework has knowledge about the static priorities of all interrupt and exception handlers.

Interrupts and exceptions can have priorities in the range 1..=(1 << NVIC_PRIO_BITS) where NVIC_PRIO_BITS is a constant defined in the device crate. The idle task has a priority of 0, the lowest priority.

Resources that are shared between handlers that run at different priorities require critical sections for memory safety. The framework ensures that critical sections are used but only where required: for example, no critical section is required by the highest priority handler that has access to the resource.

The critical section API provided by the RTFM framework (see Mutex) is based on dynamic priorities rather than on disabling interrupts. The consequence is that these critical sections will prevent some handlers, including all the ones that contend for the resource, from starting but will let higher priority handlers, that don't contend for the resource, run.

In the example below we have three interrupt handlers with priorities ranging from one to three. The two handlers with the lower priorities contend for the SHARED resource. The lowest priority handler needs to lock the SHARED resource to access its data, whereas the mid priority handler can directly access its data. The highest priority handler is free to preempt the critical section created by the lowest priority handler.

# #![allow(unused_variables)]
#fn main() {
//! examples/lock.rs

#![deny(unsafe_code)]
#![deny(warnings)]
#![no_main]
#![no_std]

extern crate panic_semihosting;

use cortex_m_semihosting::{debug, hprintln};
use lm3s6965::Interrupt;

#[rtfm::app(device = lm3s6965)]
const APP: () = {
    static mut SHARED: u32 = 0;

    #[init]
    fn init(_: init::Context) {
        rtfm::pend(Interrupt::GPIOA);
    }

    // when omitted priority is assumed to be `1`
    #[interrupt(resources = [SHARED])]
    fn GPIOA(mut c: GPIOA::Context) {
        hprintln!("A").unwrap();

        // the lower priority task requires a critical section to access the data
        c.resources.SHARED.lock(|shared| {
            // data can only be modified within this critical section (closure)
            *shared += 1;

            // GPIOB will *not* run right now due to the critical section
            rtfm::pend(Interrupt::GPIOB);

            hprintln!("B - SHARED = {}", *shared).unwrap();

            // GPIOC does not contend for `SHARED` so it's allowed to run now
            rtfm::pend(Interrupt::GPIOC);
        });

        // critical section is over: GPIOB can now start

        hprintln!("E").unwrap();

        debug::exit(debug::EXIT_SUCCESS);
    }

    #[interrupt(priority = 2, resources = [SHARED])]
    fn GPIOB(mut c: GPIOB::Context) {
        // the higher priority task does *not* need a critical section
        *c.resources.SHARED += 1;

        hprintln!("D - SHARED = {}", *c.resources.SHARED).unwrap();
    }

    #[interrupt(priority = 3)]
    fn GPIOC(_: GPIOC::Context) {
        hprintln!("C").unwrap();
    }
};

#}

$ cargo run --example lock
A
B - SHARED = 1
C
D - SHARED = 2
E

One more note about priorities: choosing a priority higher than what the device supports (that is 1 << NVIC_PRIO_BITS) will result in a compile error. Due to limitations in the language the error message is currently far from helpful: it will say something along the lines of "evaluation of constant value failed" and the span of the error will not point out to the problematic interrupt value -- we are sorry about this!

Unlike normal static variables, which need to be assigned an initial value when declared, resources can be initialized at runtime. We refer to these runtime initialized resources as late resources. Late resources are useful for moving (as in transferring ownership) peripherals initialized in init into interrupt and exception handlers.

Late resources are declared like normal resources but that are given an initial value of () (the unit value). init must return the initial values of all late resources packed in a struct of type init::LateResources.

The example below uses late resources to stablish a lockless, one-way channel between the UART0 interrupt handler and the idle function. A single producer single consumer Queue is used as the channel. The queue is split into consumer and producer end points in init and then each end point is stored in a different resource; UART0 owns the producer resource and idle owns the consumer resource.

# #![allow(unused_variables)]
#fn main() {
//! examples/late.rs

#![deny(unsafe_code)]
#![deny(warnings)]
#![no_main]
#![no_std]

extern crate panic_semihosting;

use cortex_m_semihosting::{debug, hprintln};
use heapless::{
    consts::*,
    spsc::{Consumer, Producer, Queue},
};
use lm3s6965::Interrupt;

#[rtfm::app(device = lm3s6965)]
const APP: () = {
    // Late resources
    static mut P: Producer<'static, u32, U4> = ();
    static mut C: Consumer<'static, u32, U4> = ();

    #[init]
    fn init(_: init::Context) -> init::LateResources {
        // NOTE: we use `Option` here to work around the lack of
        // a stable `const` constructor
        static mut Q: Option<Queue<u32, U4>> = None;

        *Q = Some(Queue::new());
        let (p, c) = Q.as_mut().unwrap().split();

        // Initialization of late resources
        init::LateResources { P: p, C: c }
    }

    #[idle(resources = [C])]
    fn idle(c: idle::Context) -> ! {
        loop {
            if let Some(byte) = c.resources.C.dequeue() {
                hprintln!("received message: {}", byte).unwrap();

                debug::exit(debug::EXIT_SUCCESS);
            } else {
                rtfm::pend(Interrupt::UART0);
            }
        }
    }

    #[interrupt(resources = [P])]
    fn UART0(c: UART0::Context) {
        c.resources.P.enqueue(42).unwrap();
    }
};

#}

$ cargo run --example late
received message: 42

static variables can also be used as resources. Tasks can only get & (shared) references to these resources but locks are never required to access their data. You can think of static resources as plain static variables that can be initialized at runtime and have better scoping rules: you can control which tasks can access the variable, instead of the variable being visible to all the functions in the scope it was declared in.

In the example below a key is loaded (or created) at runtime and then used from two tasks that run at different priorities.

# #![allow(unused_variables)]
#fn main() {
//! examples/static.rs

#![deny(unsafe_code)]
#![deny(warnings)]
#![no_main]
#![no_std]

extern crate panic_semihosting;

use cortex_m_semihosting::{debug, hprintln};
use lm3s6965::Interrupt;

#[rtfm::app(device = lm3s6965)]
const APP: () = {
    static KEY: u32 = ();

    #[init]
    fn init(_: init::Context) -> init::LateResources {
        rtfm::pend(Interrupt::UART0);
        rtfm::pend(Interrupt::UART1);

        init::LateResources { KEY: 0xdeadbeef }
    }

    #[interrupt(resources = [KEY])]
    fn UART0(c: UART0::Context) {
        hprintln!("UART0(KEY = {:#x})", c.resources.KEY).unwrap();

        debug::exit(debug::EXIT_SUCCESS);
    }

    #[interrupt(priority = 2, resources = [KEY])]
    fn UART1(c: UART1::Context) {
        hprintln!("UART1(KEY = {:#x})", c.resources.KEY).unwrap();
    }
};

#}

$ cargo run --example static
UART1(KEY = 0xdeadbeef)
UART0(KEY = 0xdeadbeef)

RTFM treats interrupt and exception handlers as hardware tasks. Hardware tasks are invoked by the hardware in response to events, like pressing a button. RTFM also supports software tasks which can be spawned by the software from any execution context.

Software tasks can also be assigned priorities and are dispatched from interrupt handlers. RTFM requires that free interrupts are declared in an extern block when using software tasks; these free interrupts will be used to dispatch the software tasks. An advantage of software tasks over hardware tasks is that many tasks can be mapped to a single interrupt handler.

Software tasks are declared by applying the task attribute to functions. To be able to spawn a software task the name of the task must appear in the spawn argument of the context attribute (init, idle, interrupt, etc.).

The example below showcases three software tasks that run at 2 different priorities. The three tasks map to 2 interrupts handlers.

# #![allow(unused_variables)]
#fn main() {
//! examples/task.rs

#![deny(unsafe_code)]
#![deny(warnings)]
#![no_main]
#![no_std]

extern crate panic_semihosting;

use cortex_m_semihosting::{debug, hprintln};

#[rtfm::app(device = lm3s6965)]
const APP: () = {
    #[init(spawn = [foo])]
    fn init(c: init::Context) {
        c.spawn.foo().unwrap();
    }

    #[task(spawn = [bar, baz])]
    fn foo(c: foo::Context) {
        hprintln!("foo").unwrap();

        // spawns `bar` onto the task scheduler
        // `foo` and `bar` have the same priority so `bar` will not run until
        // after `foo` terminates
        c.spawn.bar().unwrap();

        // spawns `baz` onto the task scheduler
        // `baz` has higher priority than `foo` so it immediately preempts `foo`
        c.spawn.baz().unwrap();
    }

    #[task]
    fn bar(_: bar::Context) {
        hprintln!("bar").unwrap();

        debug::exit(debug::EXIT_SUCCESS);
    }

    #[task(priority = 2)]
    fn baz(_: baz::Context) {
        hprintln!("baz").unwrap();
    }

    // Interrupt handlers used to dispatch software tasks
    extern "C" {
        fn UART0();
        fn UART1();
    }
};

#}

$ cargo run --example task
foo
baz
bar

The other advantage of software tasks is that messages can be passed to these tasks when spawning them. The type of the message payload must be specified in the signature of the task handler.

The example below showcases three tasks, two of them expect a message.

# #![allow(unused_variables)]
#fn main() {
//! examples/message.rs

#![deny(unsafe_code)]
#![deny(warnings)]
#![no_main]
#![no_std]

extern crate panic_semihosting;

use cortex_m_semihosting::{debug, hprintln};

#[rtfm::app(device = lm3s6965)]
const APP: () = {
    #[init(spawn = [foo])]
    fn init(c: init::Context) {
        c.spawn.foo(/* no message */).unwrap();
    }

    #[task(spawn = [bar])]
    fn foo(c: foo::Context) {
        static mut COUNT: u32 = 0;

        hprintln!("foo").unwrap();

        c.spawn.bar(*COUNT).unwrap();
        *COUNT += 1;
    }

    #[task(spawn = [baz])]
    fn bar(c: bar::Context, x: u32) {
        hprintln!("bar({})", x).unwrap();

        c.spawn.baz(x + 1, x + 2).unwrap();
    }

    #[task(spawn = [foo])]
    fn baz(c: baz::Context, x: u32, y: u32) {
        hprintln!("baz({}, {})", x, y).unwrap();

        if x + y > 4 {
            debug::exit(debug::EXIT_SUCCESS);
        }

        c.spawn.foo().unwrap();
    }

    extern "C" {
        fn UART0();
    }
};

#}

$ cargo run --example message
foo
bar(0)
baz(1, 2)
foo
bar(1)
baz(2, 3)

Task dispatchers do not use any dynamic memory allocation. The memory required to store messages is statically reserved. The framework will reserve enough space for every context to be able to spawn each task at most once. This is a sensible default but the "inbox" capacity of each task can be controlled using the capacity argument of the task attribute.

The example below sets the capacity of the software task foo to 4. If the capacity is not specified then the second spawn.foo call in UART0 would fail.

# #![allow(unused_variables)]
#fn main() {
//! examples/capacity.rs

#![deny(unsafe_code)]
#![deny(warnings)]
#![no_main]
#![no_std]

extern crate panic_semihosting;

use cortex_m_semihosting::{debug, hprintln};
use lm3s6965::Interrupt;

#[rtfm::app(device = lm3s6965)]
const APP: () = {
    #[init]
    fn init(_: init::Context) {
        rtfm::pend(Interrupt::UART0);
    }

    #[interrupt(spawn = [foo, bar])]
    fn UART0(c: UART0::Context) {
        c.spawn.foo(0).unwrap();
        c.spawn.foo(1).unwrap();
        c.spawn.foo(2).unwrap();
        c.spawn.foo(3).unwrap();

        c.spawn.bar().unwrap();
    }

    #[task(capacity = 4)]
    fn foo(_: foo::Context, x: u32) {
        hprintln!("foo({})", x).unwrap();
    }

    #[task]
    fn bar(_: bar::Context) {
        hprintln!("bar").unwrap();

        debug::exit(debug::EXIT_SUCCESS);
    }

    // Interrupt handlers used to dispatch software tasks
    extern "C" {
        fn UART1();
    }
};

#}

$ cargo run --example capacity
foo(0)
foo(1)
foo(2)
foo(3)
bar

When the timer-queue feature is enabled the RTFM framework includes a global timer queue that applications can use to schedule software tasks to run at some time in the future.

NOTE: The timer-queue feature can't be enabled when the target is thumbv6m-none-eabi because there's no timer queue support for ARMv6-M. This may change in the future.

NOTE: When the timer-queue feature is enabled you will not be able to use the SysTick exception as a hardware task because the runtime uses it to implement the global timer queue.

To be able to schedule a software task the name of the task must appear in the schedule argument of the context attribute. When scheduling a task the Instant at which the task should be executed must be passed as the first argument of the schedule invocation.

The RTFM runtime includes a monotonic, non-decreasing, 32-bit timer which can be queried using the Instant::now constructor. A Duration can be added to Instant::now() to obtain an Instant into the future. The monotonic timer is disabled while init runs so Instant::now() always returns the value Instant(0 /* clock cycles */); the timer is enabled right before the interrupts are re-enabled and idle is executed.

The example below schedules two tasks from init: foo and bar. foo is scheduled to run 8 million clock cycles in the future. Next, bar is scheduled to run 4 million clock cycles in the future. bar runs before foo since it was scheduled to run first.

IMPORTANT: The examples that use the schedule API or the Instant abstraction will not properly work on QEMU because the Cortex-M cycle counter functionality has not been implemented in qemu-system-arm.

# #![allow(unused_variables)]
#fn main() {
//! examples/schedule.rs

#![deny(unsafe_code)]
#![deny(warnings)]
#![no_main]
#![no_std]

extern crate panic_semihosting;

use cortex_m_semihosting::hprintln;
use rtfm::Instant;

// NOTE: does NOT work on QEMU!
#[rtfm::app(device = lm3s6965)]
const APP: () = {
    #[init(schedule = [foo, bar])]
    fn init(c: init::Context) {
        let now = Instant::now();

        hprintln!("init @ {:?}", now).unwrap();

        // Schedule `foo` to run 8e6 cycles (clock cycles) in the future
        c.schedule.foo(now + 8_000_000.cycles()).unwrap();

        // Schedule `bar` to run 4e6 cycles in the future
        c.schedule.bar(now + 4_000_000.cycles()).unwrap();
    }

    #[task]
    fn foo(_: foo::Context) {
        hprintln!("foo  @ {:?}", Instant::now()).unwrap();
    }

    #[task]
    fn bar(_: bar::Context) {
        hprintln!("bar  @ {:?}", Instant::now()).unwrap();
    }

    extern "C" {
        fn UART0();
    }
};

#}

Running the program on real hardware produces the following output in the console:

init @ Instant(0)
bar  @ Instant(4000236)
foo  @ Instant(8000173)

Software tasks have access to the Instant at which they were scheduled to run through the scheduled variable. This information and the schedule API can be used to implement periodic tasks as shown in the example below.

# #![allow(unused_variables)]
#fn main() {
//! examples/periodic.rs

#![deny(unsafe_code)]
#![deny(warnings)]
#![no_main]
#![no_std]

extern crate panic_semihosting;

use cortex_m_semihosting::hprintln;
use rtfm::Instant;

const PERIOD: u32 = 8_000_000;

// NOTE: does NOT work on QEMU!
#[rtfm::app(device = lm3s6965)]
const APP: () = {
    #[init(schedule = [foo])]
    fn init(c: init::Context) {
        c.schedule.foo(Instant::now() + PERIOD.cycles()).unwrap();
    }

    #[task(schedule = [foo])]
    fn foo(c: foo::Context) {
        let now = Instant::now();
        hprintln!("foo(scheduled = {:?}, now = {:?})", c.scheduled, now).unwrap();

        c.schedule.foo(c.scheduled + PERIOD.cycles()).unwrap();
    }

    extern "C" {
        fn UART0();
    }
};

#}

This is the output produced by the example. Note that there is zero drift / jitter even though schedule.foo was invoked at the end of foo. Using Instant::now instead of scheduled would have resulted in drift / jitter.

foo(scheduled = Instant(8000000), now = Instant(8000196))
foo(scheduled = Instant(16000000), now = Instant(16000196))
foo(scheduled = Instant(24000000), now = Instant(24000196))

For the tasks scheduled from init we have exact information about their scheduled time. For hardware tasks there's no scheduled time because these tasks are asynchronous in nature. For hardware tasks the runtime provides a start time, which indicates the time at which the task handler started executing.

Note that start is not equal to the arrival time of the event that fired the task. Depending on the priority of the task and the load of the system the start time could be very far off from the event arrival time.

What do you think will be the value of scheduled for software tasks that are spawned instead of scheduled? The answer is that spawned tasks inherit the baseline time of the context that spawned it. The baseline of hardware tasks is start, the baseline of software tasks is scheduled and the baseline of init is start = Instant(0). idle doesn't really have a baseline but tasks spawned from it will use Instant::now() as their baseline time.

The example below showcases the different meanings of the baseline.

# #![allow(unused_variables)]
#fn main() {
//! examples/baseline.rs

#![deny(unsafe_code)]
#![deny(warnings)]
#![no_main]
#![no_std]

extern crate panic_semihosting;

use cortex_m_semihosting::{debug, hprintln};
use lm3s6965::Interrupt;

// NOTE: does NOT properly work on QEMU
#[rtfm::app(device = lm3s6965)]
const APP: () = {
    #[init(spawn = [foo])]
    fn init(c: init::Context) {
        hprintln!("init(baseline = {:?})", c.start).unwrap();

        // `foo` inherits the baseline of `init`: `Instant(0)`
        c.spawn.foo().unwrap();
    }

    #[task(schedule = [foo])]
    fn foo(c: foo::Context) {
        static mut ONCE: bool = true;

        hprintln!("foo(baseline = {:?})", c.scheduled).unwrap();

        if *ONCE {
            *ONCE = false;

            rtfm::pend(Interrupt::UART0);
        } else {
            debug::exit(debug::EXIT_SUCCESS);
        }
    }

    #[interrupt(spawn = [foo])]
    fn UART0(c: UART0::Context) {
        hprintln!("UART0(baseline = {:?})", c.start).unwrap();

        // `foo` inherits the baseline of `UART0`: its `start` time
        c.spawn.foo().unwrap();
    }

    extern "C" {
        fn UART1();
    }
};

#}

Running the program on real hardware produces the following output in the console:

init(baseline = Instant(0))
foo(baseline = Instant(0))
UART0(baseline = Instant(904))
foo(baseline = Instant(904))

The app attribute injects a context, a collection of variables, into every function. All these variables have predictable, non-anonymous types so you can write plain functions that take them as arguments.

The API reference specifies how these types are generated from the input. You can also generate documentation for you binary crate (cargo doc --bin <name>); in the documentation you'll find Context structs (e.g. init::Context and idle::Context).

The example below shows the different types generates by the app attribute.

# #![allow(unused_variables)]
#fn main() {
//! examples/types.rs

#![deny(unsafe_code)]
#![deny(warnings)]
#![no_main]
#![no_std]

extern crate panic_semihosting;

use cortex_m_semihosting::debug;
use rtfm::{Exclusive, Instant};

#[rtfm::app(device = lm3s6965)]
const APP: () = {
    static mut SHARED: u32 = 0;

    #[init(schedule = [foo], spawn = [foo])]
    fn init(c: init::Context) {
        let _: Instant = c.start;
        let _: rtfm::Peripherals = c.core;
        let _: lm3s6965::Peripherals = c.device;
        let _: init::Schedule = c.schedule;
        let _: init::Spawn = c.spawn;

        debug::exit(debug::EXIT_SUCCESS);
    }

    #[exception(schedule = [foo], spawn = [foo])]
    fn SVCall(c: SVCall::Context) {
        let _: Instant = c.start;
        let _: SVCall::Schedule = c.schedule;
        let _: SVCall::Spawn = c.spawn;
    }

    #[interrupt(resources = [SHARED], schedule = [foo], spawn = [foo])]
    fn UART0(c: UART0::Context) {
        let _: Instant = c.start;
        let _: resources::SHARED = c.resources.SHARED;
        let _: UART0::Schedule = c.schedule;
        let _: UART0::Spawn = c.spawn;
    }

    #[task(priority = 2, resources = [SHARED], schedule = [foo], spawn = [foo])]
    fn foo(c: foo::Context) {
        let _: Instant = c.scheduled;
        let _: Exclusive<u32> = c.resources.SHARED;
        let _: foo::Resources = c.resources;
        let _: foo::Schedule = c.schedule;
        let _: foo::Spawn = c.spawn;
    }

    extern "C" {
        fn UART1();
    }
};

#}

Send is a marker trait for "types that can be transferred across thread boundaries", according to its definition in core. In the context of RTFM the Send trait is only required where it's possible to transfer a value between tasks that run at different priorities. This occurs in a few places: in message passing, in shared static mut resources and in the initialization of late resources.

The app attribute will enforce that Send is implemented where required so you don't need to worry much about it. It's more important to know where you do not need the Send trait: on types that are transferred between tasks that run at the same priority. This occurs in two places: in message passing and in shared static mut resources.

The example below shows where a type that doesn't implement Send can be used.

# #![allow(unused_variables)]
#fn main() {
//! `examples/not-send.rs`

#![deny(unsafe_code)]
#![deny(warnings)]
#![no_main]
#![no_std]

extern crate panic_halt;

use core::marker::PhantomData;

use cortex_m_semihosting::debug;
use rtfm::app;

pub struct NotSend {
    _0: PhantomData<*const ()>,
}

#[app(device = lm3s6965)]
const APP: () = {
    static mut SHARED: Option<NotSend> = None;

    #[init(spawn = [baz, quux])]
    fn init(c: init::Context) {
        c.spawn.baz().unwrap();
        c.spawn.quux().unwrap();
    }

    #[task(spawn = [bar])]
    fn foo(c: foo::Context) {
        // scenario 1: message passed to task that runs at the same priority
        c.spawn.bar(NotSend { _0: PhantomData }).ok();
    }

    #[task]
    fn bar(_: bar::Context, _x: NotSend) {
        // scenario 1
    }

    #[task(priority = 2, resources = [SHARED])]
    fn baz(mut c: baz::Context) {
        // scenario 2: resource shared between tasks that run at the same priority
        *c.resources.SHARED = Some(NotSend { _0: PhantomData });
    }

    #[task(priority = 2, resources = [SHARED])]
    fn quux(mut c: quux::Context) {
        // scenario 2
        let _not_send = c.resources.SHARED.take().unwrap();

        debug::exit(debug::EXIT_SUCCESS);
    }

    extern "C" {
        fn UART0();
        fn UART1();
    }
};

#}

It's important to note that late initialization of resources is effectively a send operation where the initial value is sent from idle, which has the lowest priority of 0, to a task with will run with a priority greater than or equal to 1. Thus all late resources need to implement the Send trait.

Sharing a resource with init can be used to implement late initialization, see example below. For that reason, resources shared with init must also implement the Send trait.

# #![allow(unused_variables)]
#fn main() {
//! `examples/shared-with-init.rs`

#![deny(unsafe_code)]
#![deny(warnings)]
#![no_main]
#![no_std]

extern crate panic_halt;

use cortex_m_semihosting::debug;
use lm3s6965::Interrupt;
use rtfm::app;

pub struct MustBeSend;

#[app(device = lm3s6965)]
const APP: () = {
    static mut SHARED: Option<MustBeSend> = None;

    #[init(resources = [SHARED])]
    fn init(c: init::Context) {
        // this `message` will be sent to task `UART0`
        let message = MustBeSend;
        *c.resources.SHARED = Some(message);

        rtfm::pend(Interrupt::UART0);
    }

    #[interrupt(resources = [SHARED])]
    fn UART0(c: UART0::Context) {
        if let Some(message) = c.resources.SHARED.take() {
            // `message` has been received
            drop(message);

            debug::exit(debug::EXIT_SUCCESS);
        }
    }
};

#}

Similarly, Sync is a marker trait for "types for which it is safe to share references between threads", according to its definition in core. In the context of RTFM the Sync trait is only required where it's possible for two, or more, tasks that run at different priority to hold a shared reference to a resource. This only occurs with shared static resources.

The app attribute will enforce that Sync is implemented where required but it's important to know where the Sync bound is not required: in static resources shared between tasks that run at the same priority.

The example below shows where a type that doesn't implement Sync can be used.

# #![allow(unused_variables)]
#fn main() {
//! `examples/not-sync.rs`

#![deny(unsafe_code)]
#![deny(warnings)]
#![no_main]
#![no_std]

extern crate panic_halt;

use core::marker::PhantomData;

use cortex_m_semihosting::debug;

pub struct NotSync {
    _0: PhantomData<*const ()>,
}

#[rtfm::app(device = lm3s6965)]
const APP: () = {
    static SHARED: NotSync = NotSync { _0: PhantomData };

    #[init]
    fn init(_: init::Context) {
        debug::exit(debug::EXIT_SUCCESS);
    }

    #[task(resources = [SHARED])]
    fn foo(c: foo::Context) {
        let _: &NotSync = c.resources.SHARED;
    }

    #[task(resources = [SHARED])]
    fn bar(c: bar::Context) {
        let _: &NotSync = c.resources.SHARED;
    }

    extern "C" {
        fn UART0();
    }
};

#}

Now that you have learned about the main features of the RTFM framework you can try it out on your hardware by following these instructions.

1. Instantiate the cortex-m-quickstart template.

$ # for example using `cargo-generate`
$ cargo generate \
    --git https://github.com/rust-embedded/cortex-m-quickstart \
    --name app

$ # follow the rest of the instructions

2. Add a peripheral access crate (PAC) that was generated using svd2rust v0.14.x, or a board support crate that depends on one such PAC as a dependency. Make sure that the rt feature of the crate is enabled.

In this example, I'll use the lm3s6965 device crate. This device crate doesn't have an rt Cargo feature; that feature is always enabled.

This device crate provides a linker script with the memory layout of the target device so memory.x and build.rs need to be removed.

$ cargo add lm3s6965 --vers 0.1.3

$ rm memory.x build.rs

3. Add the cortex-m-rtfm crate as a dependency and, if you need it, enable the timer-queue feature.

$ cargo add cortex-m-rtfm --allow-prerelease

4. Write your RTFM application.

Here I'll use the init example from the cortex-m-rtfm crate.

$ curl \
    -L https://github.com/japaric/cortex-m-rtfm/raw/v0.5.0-alpha.1/examples/init.rs \
    > src/main.rs

That example depends on the panic-semihosting crate:

$ cargo add panic-semihosting

5. Build it, flash it and run it.

$ # NOTE: I have uncommented the `runner` option in `.cargo/config`
$ cargo run
init

Resources shared between two or more tasks implement the Mutex trait in all contexts, even on those where a critical section is not required to access the data. This lets you easily write generic code that operates on resources and can be called from different tasks. Here's one such example:

# #![allow(unused_variables)]
#fn main() {
//! examples/generics.rs

#![deny(unsafe_code)]
#![deny(warnings)]
#![no_main]
#![no_std]

extern crate panic_semihosting;

use cortex_m_semihosting::{debug, hprintln};
use lm3s6965::Interrupt;
use rtfm::Mutex;

#[rtfm::app(device = lm3s6965)]
const APP: () = {
    static mut SHARED: u32 = 0;

    #[init]
    fn init(_: init::Context) {
        rtfm::pend(Interrupt::UART0);
        rtfm::pend(Interrupt::UART1);
    }

    #[interrupt(resources = [SHARED])]
    fn UART0(c: UART0::Context) {
        static mut STATE: u32 = 0;

        hprintln!("UART0(STATE = {})", *STATE).unwrap();

        advance(STATE, c.resources.SHARED);

        rtfm::pend(Interrupt::UART1);

        debug::exit(debug::EXIT_SUCCESS);
    }

    #[interrupt(priority = 2, resources = [SHARED])]
    fn UART1(mut c: UART1::Context) {
        static mut STATE: u32 = 0;

        hprintln!("UART1(STATE = {})", *STATE).unwrap();

        // just to show that `SHARED` can be accessed directly and ..
        *c.resources.SHARED += 0;
        // .. also through a (no-op) `lock`
        c.resources.SHARED.lock(|shared| *shared += 0);

        advance(STATE, c.resources.SHARED);
    }
};

fn advance(state: &mut u32, mut shared: impl Mutex<T = u32>) {
    *state += 1;

    let (old, new) = shared.lock(|shared| {
        let old = *shared;
        *shared += *state;
        (old, *shared)
    });

    hprintln!("SHARED: {} -> {}", old, new).unwrap();
}

#}

$ cargo run --example generics
UART1(STATE = 0)
SHARED: 0 -> 1
UART0(STATE = 0)
SHARED: 1 -> 2
UART1(STATE = 1)
SHARED: 2 -> 4

This also lets you change the static priorities of tasks without having to rewrite code. If you consistently use locks to access the data behind shared resources then your code will continue to compile when you change the priority of tasks.

You can use conditional compilation (#[cfg]) on resources (static [mut] items) and tasks (fn items). The effect of using #[cfg] attributes is that the resource / task will not be available through the corresponding Context struct if the condition doesn't hold.

The example below logs a message whenever the foo task is spawned, but only if the program has been compiled using the dev profile.

# #![allow(unused_variables)]
#fn main() {
//! examples/cfg.rs

#![deny(unsafe_code)]
#![deny(warnings)]
#![no_main]
#![no_std]

extern crate panic_semihosting;

#[cfg(debug_assertions)]
use cortex_m_semihosting::hprintln;

#[rtfm::app(device = lm3s6965)]
const APP: () = {
    #[cfg(debug_assertions)] // <- `true` when using the `dev` profile
    static mut COUNT: u32 = 0;

    #[init]
    fn init(_: init::Context) {
        // ..
    }

    #[task(priority = 3, resources = [COUNT], spawn = [log])]
    fn foo(c: foo::Context) {
        #[cfg(debug_assertions)]
        {
            *c.resources.COUNT += 1;

            c.spawn.log(*c.resources.COUNT).ok();
        }

        // this wouldn't compile in `release` mode
        // *resources.COUNT += 1;

        // ..
    }

    #[cfg(debug_assertions)]
    #[task]
    fn log(_: log::Context, n: u32) {
        hprintln!(
            "foo has been called {} time{}",
            n,
            if n == 1 { "" } else { "s" }
        )
        .ok();
    }

    extern "C" {
        fn UART0();
        fn UART1();
    }
};

#}

The main goal of moving the specification of RTFM applications to attributes in RTFM v0.4.0 was to allow inter-operation with other attributes. For example, the link_section attribute can be applied to tasks to place them in RAM; this can improve performance in some cases.

IMPORTANT: In general, the link_section, export_name and no_mangle attributes are very powerful but also easy to misuse. Incorrectly using any of these attributes can cause undefined behavior; you should always prefer to use safe, higher level attributes around them like cortex-m-rt's interrupt and exception attributes.

In the particular case of RAM functions there's no safe abstraction for it in cortex-m-rt v0.6.5 but there's an RFC for adding a ramfunc attribute in a future release.

The example below shows how to place the higher priority task, bar, in RAM.

# #![allow(unused_variables)]
#fn main() {
//! examples/ramfunc.rs

#![deny(unsafe_code)]
#![deny(warnings)]
#![no_main]
#![no_std]

extern crate panic_semihosting;

use cortex_m_semihosting::{debug, hprintln};

#[rtfm::app(device = lm3s6965)]
const APP: () = {
    #[init(spawn = [bar])]
    fn init(c: init::Context) {
        c.spawn.bar().unwrap();
    }

    #[inline(never)]
    #[task]
    fn foo(_: foo::Context) {
        hprintln!("foo").unwrap();

        debug::exit(debug::EXIT_SUCCESS);
    }

    // run this task from RAM
    #[inline(never)]
    #[link_section = ".data.bar"]
    #[task(priority = 2, spawn = [foo])]
    fn bar(c: bar::Context) {
        c.spawn.foo().unwrap();
    }

    extern "C" {
        fn UART0();

        // run the task dispatcher from RAM
        #[link_section = ".data.UART1"]
        fn UART1();
    }
};

#}

Running this program produces the expected output.

$ cargo run --example ramfunc
foo

One can look at the output of cargo-nm to confirm that bar ended in RAM (0x2000_0000), whereas foo ended in Flash (0x0000_0000).

$ cargo nm --example ramfunc --release | grep ' foo::'
20000100 B foo::FREE_QUEUE::ujkptet2nfdw5t20
200000dc B foo::INPUTS::thvubs85b91dg365
000002c6 T foo::sidaht420cg1mcm8

$ cargo nm --example ramfunc --release | grep ' bar::'
20000100 B bar::FREE_QUEUE::lk14244m263eivix
200000dc B bar::INPUTS::mi89534s44r1mnj1
20000000 T bar::ns9009yhw2dc2y25

You can give hardware tasks more task-like names using the binds argument: you name the function as you wish and specify the name of the interrupt / exception in the binds argument. Types like Spawn will be placed in a module named after the function, not the interrupt / exception. Example below:

# #![allow(unused_variables)]
#fn main() {
//! examples/binds.rs

#![deny(unsafe_code)]
#![deny(warnings)]
#![no_main]
#![no_std]

extern crate panic_semihosting;

use cortex_m_semihosting::{debug, hprintln};
use lm3s6965::Interrupt;

// `examples/interrupt.rs` rewritten to use `binds`
#[rtfm::app(device = lm3s6965)]
const APP: () = {
    #[init]
    fn init(_: init::Context) {
        rtfm::pend(Interrupt::UART0);

        hprintln!("init").unwrap();
    }

    #[idle]
    fn idle(_: idle::Context) -> ! {
        hprintln!("idle").unwrap();

        rtfm::pend(Interrupt::UART0);

        debug::exit(debug::EXIT_SUCCESS);

        loop {}
    }

    #[interrupt(binds = UART0)]
    fn foo(_: foo::Context) {
        static mut TIMES: u32 = 0;

        *TIMES += 1;

        hprintln!(
            "foo called {} time{}",
            *TIMES,
            if *TIMES > 1 { "s" } else { "" }
        )
        .unwrap();
    }
};

#}

$ cargo run --example binds
init
foo called 1 time
idle
foo called 2 times

Message passing always involves copying the payload from the sender into a static variable and then from the static variable into the receiver. Thus sending a large buffer, like a [u8; 128], as a message involves two expensive memcpys. To minimize the message passing overhead one can use indirection: instead of sending the buffer by value, one can send an owning pointer into the buffer.

One can use a global allocator to achieve indirection (alloc::Box, alloc::Rc, etc.), which requires using the nightly channel as of Rust v1.34.0, or one can use a statically allocated memory pool like heapless::Pool.

Here's an example where heapless::Pool is used to "box" buffers of 128 bytes.

# #![allow(unused_variables)]
#fn main() {
//! examples/pool.rs

#![deny(unsafe_code)]
#![deny(warnings)]
#![no_main]
#![no_std]

extern crate panic_semihosting;

use cortex_m_semihosting::{debug, hprintln};
use heapless::{
    pool,
    pool::singleton::{Box, Pool},
};
use lm3s6965::Interrupt;
use rtfm::app;

// Declare a pool of 128-byte memory blocks
pool!(P: [u8; 128]);

#[app(device = lm3s6965)]
const APP: () = {
    #[init]
    fn init(_: init::Context) {
        static mut MEMORY: [u8; 512] = [0; 512];

        // Increase the capacity of the memory pool by ~4
        P::grow(MEMORY);

        rtfm::pend(Interrupt::I2C0);
    }

    #[interrupt(priority = 2, spawn = [foo, bar])]
    fn I2C0(c: I2C0::Context) {
        // claim a memory block, leave it uninitialized and ..
        let x = P::alloc().unwrap().freeze();

        // .. send it to the `foo` task
        c.spawn.foo(x).ok().unwrap();

        // send another block to the task `bar`
        c.spawn.bar(P::alloc().unwrap().freeze()).ok().unwrap();
    }

    #[task]
    fn foo(_: foo::Context, x: Box<P>) {
        hprintln!("foo({:?})", x.as_ptr()).unwrap();

        // explicitly return the block to the pool
        drop(x);

        debug::exit(debug::EXIT_SUCCESS);
    }

    #[task(priority = 2)]
    fn bar(_: bar::Context, x: Box<P>) {
        hprintln!("bar({:?})", x.as_ptr()).unwrap();

        // this is done automatically so we can omit the call to `drop`
        // drop(x);
    }

    extern "C" {
        fn UART0();
        fn UART1();
    }
};

#}

$ cargo run --example binds
bar(0x2000008c)
foo(0x20000110)

#[rtfm::app] is a procedural macro that produces support code. If for some reason you need to inspect the code generated by this macro you have two options:

You can inspect the file rtfm-expansion.rs inside the target directory. This file contains the expansion of the #[rtfm::app] item (not your whole program!) of the last built (via cargo build or cargo check) RTFM application. The expanded code is not pretty printed by default so you'll want to run rustfmt over it before you read it.

$ cargo build --example foo

$ rustfmt target/rtfm-expansion.rs

$ tail -n30 target/rtfm-expansion.rs

#[doc = r" Implementation details"]
const APP: () = {
    use lm3s6965 as _;
    #[no_mangle]
    unsafe fn main() -> ! {
        rtfm::export::interrupt::disable();
        let mut core = rtfm::export::Peripherals::steal();
        let late = init(
            init::Locals::new(),
            init::Context::new(rtfm::Peripherals {
                CBP: core.CBP,
                CPUID: core.CPUID,
                DCB: core.DCB,
                DWT: core.DWT,
                FPB: core.FPB,
                FPU: core.FPU,
                ITM: core.ITM,
                MPU: core.MPU,
                SCB: &mut core.SCB,
                SYST: core.SYST,
                TPIU: core.TPIU,
            }),
        );
        core.SCB.scr.modify(|r| r | 1 << 1);
        rtfm::export::interrupt::enable();
        loop {
            rtfm::export::wfi()
        }
    }
};

Or, you can use the cargo-expand subcommand. This subcommand will expand all the macros, including the #[rtfm::app] attribute, and modules in your crate and print the output to the console.

$ # produces the same output as before
$ cargo expand --example smallest | tail -n30

This section describes the internals of the RTFM framework at a high level. Low level details like the parsing and code generation done by the procedural macro (#[app]) will not be explained here. The focus will be the analysis of the user specification and the data structures used by the runtime.

We highly suggest that you read the embedonomicon section on concurrency before you dive into this material.

Interrupts are core to the operation of RTFM applications. Correctly setting interrupt priorities and ensuring they remain fixed at runtime is a requisite for the memory safety of the application.

The RTFM framework exposes interrupt priorities as something that is declared at compile time. However, this static configuration must be programmed into the relevant registers during the initialization of the application. The interrupt configuration is done before the init function runs.

This example gives you an idea of the code that the RTFM framework runs:

# #![allow(unused_variables)]
#fn main() {
#[rtfm::app(device = ..)]
const APP: () = {
    #[init]
    fn init(c: init::Context) {
        // .. user code ..
    }

    #[idle]
    fn idle(c: idle::Context) -> ! {
        // .. user code ..
    }

    #[interrupt(binds = UART0, priority = 2)]
    fn foo(c: foo::Context) {
        // .. user code ..
    }
};
#}

The framework generates an entry point that looks like this:

// the real entry point of the program
#[no_mangle]
unsafe fn main() -> ! {
    // transforms a logical priority into a hardware / NVIC priority
    fn logical2hw(priority: u8) -> u8 {
        // this value comes from the device crate
        const NVIC_PRIO_BITS: u8 = ..;

        // the NVIC encodes priority in the higher bits of a bit
        // also a bigger numbers means lower priority
        ((1 << NVIC_PRIORITY_BITS) - priority) << (8 - NVIC_PRIO_BITS)
    }

    cortex_m::interrupt::disable();

    let mut core = cortex_m::Peripheral::steal();

    core.NVIC.enable(Interrupt::UART0);

    // value specified by the user
    let uart0_prio = 2;

    // check at compile time that the specified priority is within the supported range
    let _ = [(); (1 << NVIC_PRIORITY_BITS) - (uart0_prio as usize)];

    core.NVIC.set_priority(Interrupt::UART0, logical2hw(uart0_prio));

    // call into user code
    init(/* .. */);

    // ..

    cortex_m::interrupt::enable();

    // call into user code
    idle(/* .. */)
}

In RTFM, tasks handlers are not reentrant. Reentering a task handler can break Rust aliasing rules and lead to undefined behavior. A task handler can be reentered in one of two ways: in software or by hardware.

To reenter a task handler in software its underlying interrupt handler must be invoked using FFI (see example below). FFI requires unsafe code so end users are discouraged from directly invoking an interrupt handler.

# #![allow(unused_variables)]
#fn main() {
#[rtfm::app(device = ..)]
const APP: () = {
    static mut X: u64 = 0;

    #[init]
    fn init(c: init::Context) { .. }

    #[interrupt(binds = UART0, resources = [X])]
    fn foo(c: foo::Context) {
        let x: &mut u64 = c.resources.X;

        *x = 1;

        //~ `bar` can preempt `foo` at this point

        *x = 2;

        if *x == 2 {
            // something
        }
    }

    #[interrupt(binds = UART1, priority = 2)]
    fn bar(c: foo::Context) {
        extern "C" {
            fn UART0();
        }

        // this interrupt handler will invoke task handler `foo` resulting
        // in mutable aliasing of the static variable `X`
        unsafe { UART0() }
    }
};
#}

The RTFM framework must generate the interrupt handler code that calls the user defined task handlers. We are careful in making these handlers unsafe and / or impossible to call from user code.

The above example expands into:

# #![allow(unused_variables)]
#fn main() {
fn foo(c: foo::Context) {
    // .. user code ..
}

fn bar(c: bar::Context) {
    // .. user code ..
}

const APP: () = {
    // everything in this block is not visible to user code

    #[no_mangle]
    unsafe fn USART0() {
        foo(..);
    }

    #[no_mangle]
    unsafe fn USART1() {
        bar(..);
    }
};
#}

A task handler can also be reentered without software intervention. This can occur if the same handler is assigned to two or more interrupts in the vector table but there's no syntax for this kind of configuration in the RTFM framework.

One of the core foundations of RTFM is access control. Controlling which parts of the program can access which static variables is instrumental to enforcing memory safety.

Static variables are used to share state between interrupt handlers, or between interrupts handlers and the bottom execution context, main. In normal Rust code it's hard to have fine grained control over which functions can access a static variable because static variables can be accessed from any function that resides in the same scope in which they are declared. Modules give some control over how a static variable can be accessed by they are not flexible enough.

To achieve the fine-grained access control where tasks can only access the static variables (resources) that they have specified in their RTFM attribute the RTFM framework performs a source code level transformation. This transformation consists of placing the resources (static variables) specified by the user inside a const item and the user code outside the const item. This makes it impossible for the user code to refer to these static variables.

Access to the resources is then given to each task using a Resources struct whose fields correspond to the resources the task has access to. There's one such struct per task and the Resources struct is initialized with either a mutable reference (&mut) to the static variables or with a resource proxy (see section on critical sections).

The code below is an example of the kind of source level transformation that happens behind the scenes:

# #![allow(unused_variables)]
#fn main() {
#[rtfm::app(device = ..)]
const APP: () = {
    static mut X: u64: 0;
    static mut Y: bool: 0;

    #[init(resources = [Y])]
    fn init(c: init::Context) {
        // .. user code ..
    }

    #[interrupt(binds = UART0, resources = [X])]
    fn foo(c: foo::Context) {
        // .. user code ..
    }

    #[interrupt(binds = UART1, resources = [X, Y])]
    fn bar(c: bar::Context) {
        // .. user code ..
    }

    // ..
};
#}

The framework produces codes like this:

fn init(c: init::Context) {
    // .. user code ..
}

fn foo(c: foo::Context) {
    // .. user code ..
}

fn bar(c: bar::Context) {
    // .. user code ..
}

// Public API
pub mod init {
    pub struct Context<'a> {
        pub resources: Resources<'a>,
        // ..
    }

    pub struct Resources<'a> {
        pub Y: &'a mut bool,
    }
}

pub mod foo {
    pub struct Context<'a> {
        pub resources: Resources<'a>,
        // ..
    }

    pub struct Resources<'a> {
        pub X: &'a mut u64,
    }
}

pub mod bar {
    pub struct Context<'a> {
        pub resources: Resources<'a>,
        // ..
    }

    pub struct Resources<'a> {
        pub X: &'a mut u64,
        pub Y: &'a mut bool,
    }
}

/// Implementation details
const APP: () = {
    // everything inside this `const` item is hidden from user code

    static mut X: u64 = 0;
    static mut Y: bool = 0;

    // the real entry point of the program
    unsafe fn main() -> ! {
        interrupt::disable();

        // ..

        // call into user code; pass references to the static variables
        init(init::Context {
            resources: init::Resources {
                X: &mut X,
            },
            // ..
        });

        // ..

        interrupt::enable();

        // ..
    }

    // interrupt handler that `foo` binds to
    #[no_mangle]
    unsafe fn UART0() {
        // call into user code; pass references to the static variables
        foo(foo::Context {
            resources: foo::Resources {
                X: &mut X,
            },
            // ..
        });
    }

    // interrupt handler that `bar` binds to
    #[no_mangle]
    unsafe fn UART1() {
        // call into user code; pass references to the static variables
        bar(bar::Context {
            resources: bar::Resources {
                X: &mut X,
                Y: &mut Y,
            },
            // ..
        });
    }
};

Some resources are initialized at runtime after the init function returns. It's important that these resources (static variables) are fully initialized before tasks are allowed to run, that is they must be initialized while interrupts are disabled.

The example below shows the kind of code that the framework generates to initialize late resources.

# #![allow(unused_variables)]
#fn main() {
#[rtfm::app(device = ..)]
const APP: () = {
    // late resource
    static mut X: Thing = {};

    #[init]
    fn init() -> init::LateResources {
        // ..

        init::LateResources {
            X: Thing::new(..),
        }
    }

    #[task(binds = UART0, resources = [X])]
    fn foo(c: foo::Context) {
        let x: &mut Thing = c.resources.X;

        x.frob();

        // ..
    }

    // ..
};
#}

The code generated by the framework looks like this:

fn init(c: init::Context) -> init::LateResources {
    // .. user code ..
}

fn foo(c: foo::Context) {
    // .. user code ..
}

// Public API
pub mod init {
    pub struct LateResources {
        pub X: Thing,
    }

    // ..
}

pub mod foo {
    pub struct Resources<'a> {
        pub X: &'a mut Thing,
    }

    pub struct Context<'a> {
        pub resources: Resources<'a>,
        // ..
    }
}

/// Implementation details
const APP: () = {
    // uninitialized static
    static mut X: MaybeUninit<Thing> = MaybeUninit::uninit();

    #[no_mangle]
    unsafe fn main() -> ! {
        cortex_m::interrupt::disable();

        // ..

        let late = init(..);

        // initialization of late resources
        X.write(late.X);

        cortex_m::interrupt::enable(); //~ compiler fence

        // exceptions, interrupts and tasks can preempt `main` at this point

        idle(..)
    }

    #[no_mangle]
    unsafe fn UART0() {
        foo(foo::Context {
            resources: foo::Resources {
                // `X` has been initialized at this point
                X: &mut *X.as_mut_ptr(),
            },
            // ..
        })
    }
};

An important detail here is that interrupt::enable behaves like a compiler fence, which prevents the compiler from reordering the write to X to after interrupt::enable. If the compiler were to do that kind of reordering there would be a data race between that write and whatever operation foo performs on X.

Architectures with more complex instruction pipelines may need a memory barrier (atomic::fence) instead of a compiler fence to fully flush the write operation before interrupts are re-enabled. The ARM Cortex-M architecture doesn't need a memory barrier in single-core context.

When a resource (static variable) is shared between two, or more, tasks that run at different priorities some form of mutual exclusion is required to access the memory in a data race free manner. In RTFM we use priority-based critical sections to guarantee mutual exclusion (see the Immediate Priority Ceiling Protocol).

The critical section consists of temporarily raising the dynamic priority of the task. While a task is within this critical section all the other tasks that may request the resource are not allowed to start.

How high must the dynamic priority be to ensure mutual exclusion on a particular resource? The ceiling analysis is in charge of answering that question and will be discussed in the next section. This section will focus on the implementation of the critical section.

For simplicity, let's look at a resource shared by two tasks that run at different priorities. Clearly one of the task can preempt the other; to prevent a data race the lower priority task must use a critical section when it needs to modify the shared memory. On the other hand, the higher priority task can directly modify the shared memory because it can't be preempted by the lower priority task. To enforce the use of a critical section on the lower priority task we give it a resource proxy, whereas we give a mutable reference (&mut-) to the higher priority task.

The example below shows the different types handed out to each task:

# #![allow(unused_variables)]
#fn main() {
#[rtfm::app(device = ..)]
const APP: () = {
    static mut X: u64 = 0;

    #[interrupt(binds = UART0, priority = 1, resources = [X])]
    fn foo(c: foo::Context) {
        // resource proxy
        let mut x: resources::X = c.resources.X;

        x.lock(|x: &mut u64| {
            // critical section
            *x += 1
        });
    }

    #[interrupt(binds = UART1, priority = 2, resources = [X])]
    fn bar(c: foo::Context) {
        let mut x: &mut u64 = c.resources.X;

        *x += 1;
    }

    // ..
};
#}

Now let's see how these types are created by the framework.

# #![allow(unused_variables)]
#fn main() {
fn foo(c: foo::Context) {
    // .. user code ..
}

fn bar(c: bar::Context) {
    // .. user code ..
}

pub mod resources {
    pub struct X {
        // ..
    }
}

pub mod foo {
    pub struct Resources {
        pub X: resources::X,
    }

    pub struct Context {
        pub resources: Resources,
        // ..
    }
}

pub mod bar {
    pub struct Resources<'a> {
        pub X: rtfm::Exclusive<'a, u64>, // newtype over `&'a mut u64`
    }

    pub struct Context {
        pub resources: Resources,
        // ..
    }
}

const APP: () = {
    static mut X: u64 = 0;

    impl rtfm::Mutex for resources::X {
        type T = u64;

        fn lock<R>(&mut self, f: impl FnOnce(&mut u64) -> R) -> R {
            // we'll check this in detail later
        }
    }

    #[no_mangle]
    unsafe fn UART0() {
        foo(foo::Context {
            resources: foo::Resources {
                X: resources::X::new(/* .. */),
            },
            // ..
        })
    }

    #[no_mangle]
    unsafe fn UART1() {
        bar(bar::Context {
            resources: bar::Resources {
                X: rtfm::Exclusive(&mut X),
            },
            // ..
        })
    }
};
#}

Let's now zoom into the critical section itself. In this example, we have to raise the dynamic priority to at least 2 to prevent a data race. On the Cortex-M architecture the dynamic priority can be changed by writing to the BASEPRI register.

The semantics of the BASEPRI register are as follows:

• Writing a value of 0 to BASEPRI disables its functionality.
• Writing a non-zero value to BASEPRI changes the priority level required for interrupt preemption. However, this only has an effect when the written value is lower than the priority level of current execution context, but note that a lower hardware priority level means higher logical priority

Thus the dynamic priority at any point in time can be computed as

# #![allow(unused_variables)]
#fn main() {
dynamic_priority = max(hw2logical(BASEPRI), hw2logical(static_priority))
#}

Where static_priority is the priority programmed in the NVIC for the current interrupt, or a logical 0 when the current context is idle.

In this particular example we could implement the critical section as follows:

NOTE: this is a simplified implementation

# #![allow(unused_variables)]
#fn main() {
impl rtfm::Mutex for resources::X {
    type T = u64;

    fn lock<R, F>(&mut self, f: F) -> R
    where
        F: FnOnce(&mut u64) -> R,
    {
        unsafe {
            // start of critical section: raise dynamic priority to `2`
            asm!("msr BASEPRI, 192" : : : "memory" : "volatile");

            // run user code within the critical section
            let r = f(&mut implementation_defined_name_for_X);

            // end of critical section: restore dynamic priority to its static value (`1`)
            asm!("msr BASEPRI, 0" : : : "memory" : "volatile");

            r
        }
    }
}
#}

Here it's important to use the "memory" clobber in the asm! block. It prevents the compiler from reordering memory operations across it. This is important because accessing the variable X outside the critical section would result in a data race.

It's important to note that the signature of the lock method prevents nesting calls to it. This is required for memory safety, as nested calls would produce multiple mutable references (&mut-) to X breaking Rust aliasing rules. See below:

# #![allow(unused_variables)]
#fn main() {
#[interrupt(binds = UART0, priority = 1, resources = [X])]
fn foo(c: foo::Context) {
    // resource proxy
    let mut res: resources::X = c.resources.X;

    res.lock(|x: &mut u64| {
        res.lock(|alias: &mut u64| {
            //~^ error: `res` has already been mutably borrowed
            // ..
        });
    });
}
#}

Nesting calls to lock on the same resource must be rejected by the compiler for memory safety but nesting lock calls on different resources is a valid operation. In that case we want to make sure that nesting critical sections never results in lowering the dynamic priority, as that would be unsound, and we also want to optimize the number of writes to the BASEPRI register and compiler fences. To that end we'll track the dynamic priority of the task using a stack variable and use that to decide whether to write to BASEPRI or not. In practice, the stack variable will be optimized away by the compiler but it still provides extra information to the compiler.

Consider this program:

# #![allow(unused_variables)]
#fn main() {
#[rtfm::app(device = ..)]
const APP: () = {
    static mut X: u64 = 0;
    static mut Y: u64 = 0;

    #[init]
    fn init() {
        rtfm::pend(Interrupt::UART0);
    }

    #[interrupt(binds = UART0, priority = 1, resources = [X, Y])]
    fn foo(c: foo::Context) {
        let mut x = c.resources.X;
        let mut y = c.resources.Y;

        y.lock(|y| {
            *y += 1;

            *x.lock(|x| {
                x += 1;
            });

            *y += 1;
        });

        // mid-point

        x.lock(|x| {
            *x += 1;

            y.lock(|y| {
                *y += 1;
            });

            *x += 1;
        })
    }

    #[interrupt(binds = UART1, priority = 2, resources = [X])]
    fn bar(c: foo::Context) {
        // ..
    }

    #[interrupt(binds = UART2, priority = 3, resources = [Y])]
    fn baz(c: foo::Context) {
        // ..
    }

    // ..
};
#}

The code generated by the framework looks like this:

# #![allow(unused_variables)]
#fn main() {
// omitted: user code

pub mod resources {
    pub struct X<'a> {
        priority: &'a Cell<u8>,
    }

    impl<'a> X<'a> {
        pub unsafe fn new(priority: &'a Cell<u8>) -> Self {
            X { priority }
        }

        pub unsafe fn priority(&self) -> &Cell<u8> {
            self.priority
        }
    }

    // repeat for `Y`
}

pub mod foo {
    pub struct Context {
        pub resources: Resources,
        // ..
    }

    pub struct Resources<'a> {
        pub X: resources::X<'a>,
        pub Y: resources::Y<'a>,
    }
}

const APP: () = {
    #[no_mangle]
    unsafe fn UART0() {
        // the static priority of this interrupt (as specified by the user)
        const PRIORITY: u8 = 1;

        // take a snashot of the BASEPRI
        let initial: u8;
        asm!("mrs $0, BASEPRI" : "=r"(initial) : : : "volatile");

        let priority = Cell::new(PRIORITY);
        foo(foo::Context {
            resources: foo::Resources::new(&priority),
            // ..
        });

        // roll back the BASEPRI to the snapshot value we took before
        asm!("msr BASEPRI, $0" : : "r"(initial) : : "volatile");
    }

    // similarly for `UART1`

    impl<'a> rtfm::Mutex for resources::X<'a> {
        type T = u64;

        fn lock<R>(&mut self, f: impl FnOnce(&mut u64) -> R) -> R {
            unsafe {
                // the priority ceiling of this resource
                const CEILING: u8 = 2;

                let current = self.priority().get();
                if current < CEILING {
                    // raise dynamic priority
                    self.priority().set(CEILING);
                    let hw = logical2hw(CEILING);
                    asm!("msr BASEPRI, $0" : : "r"(hw) : "memory" : "volatile");

                    let r = f(&mut X);

                    // restore dynamic priority
                    let hw = logical2hw(current);
                    asm!("msr BASEPRI, $0" : : "r"(hw) : "memory" : "volatile");
                    self.priority().set(current);

                    r
                } else {
                    // dynamic priority is high enough
                    f(&mut X)
                }
            }
        }
    }

    // repeat for `Y`
};
#}

At the end the compiler will optimize the function foo into something like this:

# #![allow(unused_variables)]
#fn main() {
fn foo(c: foo::Context) {
    // NOTE: BASEPRI contains the value `0` (its reset value) at this point

    // raise dynamic priority to `3`
    unsafe { asm!("msr BASEPRI, 160" : : : "memory" : "volatile") }

    // the two operations on `Y` are merged into one
    Y += 2;

    // BASEPRI is not modified to access `X` because the dynamic priority is high enough
    X += 1;

    // lower (restore) the dynamic priority to `1`
    unsafe { asm!("msr BASEPRI, 224" : : : "memory" : "volatile") }

    // mid-point

    // raise dynamic priority to `2`
    unsafe { asm!("msr BASEPRI, 192" : : : "memory" : "volatile") }

    X += 1;

    // raise dynamic priority to `3`
    unsafe { asm!("msr BASEPRI, 160" : : : "memory" : "volatile") }

    Y += 1;

    // lower (restore) the dynamic priority to `2`
    unsafe { asm!("msr BASEPRI, 192" : : : "memory" : "volatile") }

    // NOTE: it would be sound to merge this operation on X with the previous one but
    // compiler fences are coarse grained and prevent such optimization
    X += 1;

    // lower (restore) the dynamic priority to `1`
    unsafe { asm!("msr BASEPRI, 224" : : : "memory" : "volatile") }

    // NOTE: BASEPRI contains the value `224` at this point
    // the UART0 handler will restore the value to `0` before returning
}
#}

An invariant that the RTFM framework has to preserve is that the value of the BASEPRI at the start of an interrupt handler must be the same value it has when the interrupt handler returns. BASEPRI may change during the execution of the interrupt handler but running an interrupt handler from start to finish should not result in an observable change of BASEPRI.

This invariant needs to be preserved to avoid raising the dynamic priority of a handler through preemption. This is best observed in the following example:

# #![allow(unused_variables)]
#fn main() {
#[rtfm::app(device = ..)]
const APP: () = {
    static mut X: u64 = 0;

    #[init]
    fn init() {
        // `foo` will run right after `init` returns
        rtfm::pend(Interrupt::UART0);
    }

    #[task(binds = UART0, priority = 1)]
    fn foo() {
        // BASEPRI is `0` at this point; the dynamic priority is currently `1`

        // `bar` will preempt `foo` at this point
        rtfm::pend(Interrupt::UART1);

        // BASEPRI is `192` at this point (due to a bug); the dynamic priority is now `2`
        // this function returns to `idle`
    }

    #[task(binds = UART1, priority = 2, resources = [X])]
    fn bar() {
        // BASEPRI is `0` (dynamic priority = 2)

        X.lock(|x| {
            // BASEPRI is raised to `160` (dynamic priority = 3)

            // ..
        });

        // BASEPRI is restored to `192` (dynamic priority = 2)
    }

    #[idle]
    fn idle() -> ! {
        // BASEPRI is `192` (due to a bug); dynamic priority = 2

        // this has no effect due to the BASEPRI value
        // the task `foo` will never be executed again
        rtfm::pend(Interrupt::UART0);

        loop {
            // ..
        }
    }

    #[task(binds = UART2, priority = 3, resources = [X])]
    fn baz() {
        // ..
    }

};
#}

IMPORTANT: let's say we forget to roll back BASEPRI in UART1 -- this would be a bug in the RTFM code generator.

# #![allow(unused_variables)]
#fn main() {
// code generated by RTFM

const APP: () = {
    // ..

    #[no_mangle]
    unsafe fn UART1() {
        // the static priority of this interrupt (as specified by the user)
        const PRIORITY: u8 = 2;

        // take a snashot of the BASEPRI
        let initial: u8;
        asm!("mrs $0, BASEPRI" : "=r"(initial) : : : "volatile");

        let priority = Cell::new(PRIORITY);
        bar(bar::Context {
            resources: bar::Resources::new(&priority),
            // ..
        });

        // BUG: FORGOT to roll back the BASEPRI to the snapshot value we took before
        // asm!("msr BASEPRI, $0" : : "r"(initial) : : "volatile");
    }
};
#}

The consequence is that idle will run at a dynamic priority of 2 and in fact the system will never again run at a dynamic priority lower than 2. This doesn't compromise the memory safety of the program but affects task scheduling: in this particular case tasks with a priority of 1 will never get a chance to run.

A resource priority ceiling, or just ceiling, is the dynamic priority that any task must have to safely access the resource memory. Ceiling analysis is relatively simple but critical to the memory safety of RTFM applications.

To compute the ceiling of a resource we must first collect a list of tasks that have access to the resource -- as the RTFM framework enforces access control to resources at compile time it also has access to this information at compile time. The ceiling of the resource is simply the highest logical priority among those tasks.

init and idle are not proper tasks but they can access resources so they need to be considered in the ceiling analysis. idle is considered as a task that has a logical priority of 0 whereas init is completely omitted from the analysis -- the reason for that is that init never uses (or needs) critical sections to access static variables.

In the previous section we showed that a shared resource may appear as a mutable reference or behind a proxy depending on the task that has access to it. Which version is presented to the task depends on the task priority and the resource ceiling. If the task priority is the same as the resource ceiling then the task gets a mutable reference to the resource memory, otherwise the task gets a proxy -- this also applies to idle. init is special: it always gets a mutable reference to resources.

An example to illustrate the ceiling analysis:

# #![allow(unused_variables)]
#fn main() {
#[rtfm::app(device = ..)]
const APP: () = {
    // accessed by `foo` (prio = 1) and `bar` (prio = 2)
    // CEILING = 2
    static mut X: u64 = 0;

    // accessed by `idle` (prio = 0)
    // CEILING = 0
    static mut Y: u64 = 0;

    #[init(resources = [X])]
    fn init(c: init::Context) {
        // mutable reference because this is `init`
        let x: &mut u64 = c.resources.X;

        // mutable reference because this is `init`
        let y: &mut u64 = c.resources.Y;

        // ..
    }

    // PRIORITY = 0
    #[idle(resources = [Y])]
    fn idle(c: idle::Context) -> ! {
        // mutable reference because priority (0) == resource ceiling (0)
        let y: &'static mut u64 = c.resources.Y;

        loop {
            // ..
        }
    }

    #[interrupt(binds = UART0, priority = 1, resources = [X])]
    fn foo(c: foo::Context) {
        // resource proxy because task priority (1) < resource ceiling (2)
        let x: resources::X = c.resources.X;

        // ..
    }

    #[interrupt(binds = UART1, priority = 2, resources = [X])]
    fn bar(c: foo::Context) {
        // mutable reference because task priority (2) == resource ceiling (2)
        let x: &mut u64 = c.resources.X;

        // ..
    }

    // ..
};
#}

RTFM supports software tasks and hardware tasks. Each hardware task is bound to a different interrupt handler. On the other hand, several software tasks may be dispatched by the same interrupt handler -- this is done to minimize the number of interrupts handlers used by the framework.

The framework groups spawn-able tasks by priority level and generates one task dispatcher per priority level. Each task dispatcher runs on a different interrupt handler and the priority of said interrupt handler is set to match the priority level of the tasks managed by the dispatcher.

Each task dispatcher keeps a queue of tasks which are ready to execute; this queue is referred to as the ready queue. Spawning a software task consists of adding an entry to this queue and pending the interrupt that runs the corresponding task dispatcher. Each entry in this queue contains a tag (enum) that identifies the task to execute and a pointer to the message passed to the task.

The ready queue is a SPSC (Single Producer Single Consumer) lock-free queue. The task dispatcher owns the consumer endpoint of the queue; the producer endpoint is treated as a resource shared by the tasks that can spawn other tasks.

Let's first take a look the code generated by the framework to dispatch tasks. Consider this example:

# #![allow(unused_variables)]
#fn main() {
#[rtfm::app(device = ..)]
const APP: () = {
    // ..

    #[interrupt(binds = UART0, priority = 2, spawn = [bar, baz])]
    fn foo(c: foo::Context) {
        foo.spawn.bar().ok();

        foo.spawn.baz(42).ok();
    }

    #[task(capacity = 2, priority = 1)]
    fn bar(c: bar::Context) {
        // ..
    }

    #[task(capacity = 2, priority = 1, resources = [X])]
    fn baz(c: baz::Context, input: i32) {
        // ..
    }

    extern "C" {
        fn UART1();
    }
};
#}

The framework produces the following task dispatcher which consists of an interrupt handler and a ready queue:

# #![allow(unused_variables)]
#fn main() {
fn bar(c: bar::Context) {
    // .. user code ..
}

const APP: () = {
    use heapless::spsc::Queue;
    use cortex_m::register::basepri;

    struct Ready<T> {
        task: T,
        // ..
    }

    /// `spawn`-able tasks that run at priority level `1`
    enum T1 {
        bar,
        baz,
    }

    // ready queue of the task dispatcher
    // `U4` is a type-level integer that represents the capacity of this queue
    static mut RQ1: Queue<Ready<T1>, U4> = Queue::new();

    // interrupt handler chosen to dispatch tasks at priority `1`
    #[no_mangle]
    unsafe UART1() {
        // the priority of this interrupt handler
        const PRIORITY: u8 = 1;

        let snapshot = basepri::read();

        while let Some(ready) = RQ1.split().1.dequeue() {
            match ready.task {
                T1::bar => {
                    // **NOTE** simplified implementation

                    // used to track the dynamic priority
                    let priority = Cell::new(PRIORITY);

                    // call into user code
                    bar(bar::Context::new(&priority));
                }

                T1::baz => {
                    // we'll look at `baz` later
                }
            }
        }

        // BASEPRI invariant
        basepri::write(snapshot);
    }
};
#}

The spawn API is exposed to the user as the methods of a Spawn struct. There's one Spawn struct per task.

The Spawn code generated by the framework for the previous example looks like this:

# #![allow(unused_variables)]
#fn main() {
mod foo {
    // ..

    pub struct Context<'a> {
        pub spawn: Spawn<'a>,
        // ..
    }

    pub struct Spawn<'a> {
        // tracks the dynamic priority of the task
        priority: &'a Cell<u8>,
    }

    impl<'a> Spawn<'a> {
        // `unsafe` and hidden because we don't want the user to tamper with it
        #[doc(hidden)]
        pub unsafe fn priority(&self) -> &Cell<u8> {
            self.priority
        }
    }
}

const APP: () = {
    // ..

    // Priority ceiling for the producer endpoint of the `RQ1`
    const RQ1_CEILING: u8 = 2;

    // used to track how many more `bar` messages can be enqueued
    // `U2` is the capacity of the `bar` task; a max of two instances can be queued
    // this queue is filled by the framework before `init` runs
    static mut bar_FQ: Queue<(), U2> = Queue::new();

    // Priority ceiling for the consumer endpoint of `bar_FQ`
    const bar_FQ_CEILING: u8 = 2;

    // a priority-based critical section
    //
    // this run the given closure `f` at a dynamic priority of at least
    // `ceiling`
    fn lock(priority: &Cell<u8>, ceiling: u8, f: impl FnOnce()) {
        // ..
    }

    impl<'a> foo::Spawn<'a> {
        /// Spawns the `bar` task
        pub fn bar(&self) -> Result<(), ()> {
            unsafe {
                match lock(self.priority(), bar_FQ_CEILING, || {
                    bar_FQ.split().1.dequeue()
                }) {
                    Some(()) => {
                        lock(self.priority(), RQ1_CEILING, || {
                            // put the taks in the ready queue
                            RQ1.split().1.enqueue_unchecked(Ready {
                                task: T1::bar,
                                // ..
                            })
                        });

                        // pend the interrupt that runs the task dispatcher
                        rtfm::pend(Interrupt::UART0);
                    }

                    None => {
                        // maximum capacity reached; spawn failed
                        Err(())
                    }
                }
            }
        }
    }
};
#}

Using bar_FQ to limit the number of bar tasks that can be spawned may seem like an artificial limitation but it will make more sense when we talk about task capacities.

We have omitted how message passing actually works so let's revisit the spawn implementation but this time for task baz which receives a u64 message.

# #![allow(unused_variables)]
#fn main() {
fn baz(c: baz::Context, input: u64) {
    // .. user code ..
}

const APP: () = {
    // ..

    // Now we show the full contents of the `Ready` struct
    struct Ready {
        task: Task,
        // message index; used to index the `INPUTS` buffer
        index: u8,
    }

    // memory reserved to hold messages passed to `baz`
    static mut baz_INPUTS: [MaybeUninit<u64>; 2] =
        [MaybeUninit::uninit(), MaybeUninit::uninit()];

    // the free queue: used to track free slots in the `baz_INPUTS` array
    // this queue is initialized with values `0` and `1` before `init` is executed
    static mut baz_FQ: Queue<u8, U2> = Queue::new();

    // Priority ceiling for the consumer endpoint of `baz_FQ`
    const baz_FQ_CEILING: u8 = 2;

    impl<'a> foo::Spawn<'a> {
        /// Spawns the `baz` task
        pub fn baz(&self, message: u64) -> Result<(), u64> {
            unsafe {
                match lock(self.priority(), baz_FQ_CEILING, || {
                    baz_FQ.split().1.dequeue()
                }) {
                    Some(index) => {
                        // NOTE: `index` is an ownining pointer into this buffer
                        baz_INPUTS[index as usize].write(message);

                        lock(self.priority(), RQ1_CEILING, || {
                            // put the task in the ready queu
                            RQ1.split().1.enqueue_unchecked(Ready {
                                task: T1::baz,
                                index,
                            });
                        });

                        // pend the interrupt that runs the task dispatcher
                        rtfm::pend(Interrupt::UART0);
                    }

                    None => {
                        // maximum capacity reached; spawn failed
                        Err(message)
                    }
                }
            }
        }
    }
};
#}

And now let's look at the real implementation of the task dispatcher:

# #![allow(unused_variables)]
#fn main() {
const APP: () = {
    // ..

    #[no_mangle]
    unsafe UART1() {
        const PRIORITY: u8 = 1;

        let snapshot = basepri::read();

        while let Some(ready) = RQ1.split().1.dequeue() {
            match ready.task {
                Task::baz => {
                    // NOTE: `index` is an ownining pointer into this buffer
                    let input = baz_INPUTS[ready.index as usize].read();

                    // the message has been read out so we can return the slot
                    // back to the free queue
                    // (the task dispatcher has exclusive access to the producer
                    // endpoint of this queue)
                    baz_FQ.split().0.enqueue_unchecked(ready.index);

                    let priority = Cell::new(PRIORITY);
                    baz(baz::Context::new(&priority), input)
                }

                Task::bar => {
                    // looks just like the `baz` branch
                }

            }
        }

        // BASEPRI invariant
        basepri::write(snapshot);
    }
};
#}

INPUTS plus FQ, the free queue, is effectively a memory pool. However, instead of using the usual free list (linked list) to track empty slots in the INPUTS buffer we use a SPSC queue; this lets us reduce the number of critical sections. In fact, thanks to this choice the task dispatching code is lock-free.

The RTFM framework uses several queues like ready queues and free queues. When the free queue is empty trying to spawn a task results in an error; this condition is checked at runtime. Not all the operations performed by the framework on these queues check if the queue is empty / full. For example, returning an slot to the free queue (see the task dispatcher) is unchecked because there's a fixed number of such slots circulating in the system that's equal to the capacity of the free queue. Similarly, adding an entry to the ready queue (see Spawn) is unchecked because of the queue capacity chosen by the framework.

Users can specify the capacity of software tasks; this capacity is the maximum number of messages one can post to said task from a higher priority task before spawn returns an error. This user-specified capacity is the capacity of the free queue of the task (e.g. foo_FQ) and also the size of the array that holds the inputs to the task (e.g. foo_INPUTS).

The capacity of the ready queue (e.g. RQ1) is chosen to be the sum of the capacities of all the different tasks managed by the dispatcher; this sum is also the number of messages the queue will hold in the worst case scenario of all possible messages being posted before the task dispatcher gets a chance to run. For this reason, getting a slot from the free queue in any spawn operation implies that the ready queue is not yet full so inserting an entry into the ready queue can omit the "is it full?" check.

In our running example the task bar takes no input so we could have omitted both bar_INPUTS and bar_FQ and let the user post an unbounded number of messages to this task, but if we did that it would have not be possible to pick a capacity for RQ1 that lets us omit the "is it full?" check when spawning a baz task. In the section about the timer queue we'll see how the free queue is used by tasks that have no inputs.

The queues internally used by the spawn API are treated like normal resources and included in the ceiling analysis. It's important to note that these are SPSC queues and that only one of the endpoints is behind a resource; the other endpoint is owned by a task dispatcher.

Consider the following example:

# #![allow(unused_variables)]
#fn main() {
#[rtfm::app(device = ..)]
const APP: () = {
    #[idle(spawn = [foo, bar])]
    fn idle(c: idle::Context) -> ! {
        // ..
    }

    #[task]
    fn foo(c: foo::Context) {
        // ..
    }

    #[task]
    fn bar(c: bar::Context) {
        // ..
    }

    #[task(priority = 2, spawn = [foo])]
    fn baz(c: baz::Context) {
        // ..
    }

    #[task(priority = 3, spawn = [bar])]
    fn quux(c: quux::Context) {
        // ..
    }
};
#}

This is how the ceiling analysis would go:

• idle (prio = 0) and baz (prio = 2) contend for the consumer endpoint of foo_FQ; this leads to a priority ceiling of 2.

• idle (prio = 0) and quux (prio = 3) contend for the consumer endpoint of bar_FQ; this leads to a priority ceiling of 3.

• idle (prio = 0), baz (prio = 2) and quux (prio = 3) all contend for the producer endpoint of RQ1; this leads to a priority ceiling of 3

The timer queue functionality lets the user schedule tasks to run at some time in the future. Unsurprisingly, this feature is also implemented using a queue: a priority queue where the scheduled tasks are kept sorted by earliest scheduled time. This feature requires a timer capable of setting up timeout interrupts. The timer is used to trigger an interrupt when the scheduled time of a task is up; at that point the task is removed from the timer queue and moved into the appropriate ready queue.

Let's see how this in implemented in code. Consider the following program:

# #![allow(unused_variables)]
#fn main() {
#[rtfm::app(device = ..)]
const APP: () = {
    // ..

    #[task(capacity = 2, schedule = [foo])]
    fn foo(c: foo::Context, x: u32) {
        // schedule this task to run again in 1M cycles
        c.schedule.foo(c.scheduled + Duration::cycles(1_000_000), x + 1).ok();
    }

    extern "C" {
        fn UART0();
    }
};
#}

Let's first look at the schedule API.

# #![allow(unused_variables)]
#fn main() {
mod foo {
    pub struct Schedule<'a> {
        priority: &'a Cell<u8>,
    }

    impl<'a> Schedule<'a> {
        // unsafe and hidden because we don't want the user to tamper with this
        #[doc(hidden)]
        pub unsafe fn priority(&self) -> &Cell<u8> {
            self.priority
        }
    }
}

const APP: () = {
    use rtfm::Instant;

    // all tasks that can be `schedule`-d
    enum T {
        foo,
    }

    struct NotReady {
        index: u8,
        instant: Instant,
        task: T,
    }

    // The timer queue is a binary (min) heap of `NotReady` tasks
    static mut TQ: TimerQueue<U2> = ..;
    const TQ_CEILING: u8 = 1;

    static mut foo_FQ: Queue<u8, U2> = Queue::new();
    const foo_FQ_CEILING: u8 = 1;

    static mut foo_INPUTS: [MaybeUninit<u32>; 2] =
        [MaybeUninit::uninit(), MaybeUninit::uninit()];

    static mut foo_INSTANTS: [MaybeUninit<Instant>; 2] =
        [MaybeUninit::uninit(), MaybeUninit::uninit()];

    impl<'a> foo::Schedule<'a> {
        fn foo(&self, instant: Instant, input: u32) -> Result<(), u32> {
            unsafe {
                let priority = self.priority();
                if let Some(index) = lock(priority, foo_FQ_CEILING, || {
                    foo_FQ.split().1.dequeue()
                }) {
                    // `index` is an owning pointer into these buffers
                    foo_INSTANTS[index as usize].write(instant);
                    foo_INPUTS[index as usize].write(input);

                    let nr = NotReady {
                        index,
                        instant,
                        task: T::foo,
                    };

                    lock(priority, TQ_CEILING, || {
                        TQ.enqueue_unchecked(nr);
                    });
                } else {
                    // No space left to store the input / instant
                    Err(input)
                }
            }
        }
    }
};
#}

This looks very similar to the Spawn implementation. In fact, the same INPUTS buffer and free queue (FQ) are shared between the spawn and schedule APIs. The main difference between the two is that schedule also stores the Instant at which the task was scheduled to run in a separate buffer (foo_INSTANTS in this case).

TimerQueue::enqueue_unchecked does a bit more work that just adding the entry into a min-heap: it also pends the system timer interrupt (SysTick) if the new entry ended up first in the queue.

The system timer interrupt (SysTick) takes cares of two things: moving tasks that have become ready from the timer queue into the right ready queue and setting up a timeout interrupt to fire when the scheduled time of the next task is up.

Let's see the associated code.

# #![allow(unused_variables)]
#fn main() {
const APP: () = {
    #[no_mangle]
    fn SysTick() {
        const PRIORITY: u8 = 1;

        let priority = &Cell::new(PRIORITY);
        while let Some(ready) = lock(priority, TQ_CEILING, || TQ.dequeue()) {
            match ready.task {
                T::foo => {
                    // move this task into the `RQ1` ready queue
                    lock(priority, RQ1_CEILING, || {
                        RQ1.split().0.enqueue_unchecked(Ready {
                           task: T1::foo,
                           index: ready.index,
                        })
                    });

                    // pend the task dispatcher
                    rtfm::pend(Interrupt::UART0);
                }
            }
        }
    }
};
#}

This looks similar to a task dispatcher except that instead of running the ready task this only places the task in the corresponding ready queue, that way it will run at the right priority.

TimerQueue::dequeue will set up a new timeout interrupt when it returns None. This ties in with TimerQueue::enqueue_unchecked, which pends this handler; basically, enqueue_unchecked delegates the task of setting up a new timeout interrupt to the SysTick handler.

In the current implementation the DWT's (Data Watchpoint and Trace) cycle counter is used as a monotonic timer. Instant::now returns a snapshot of this timer; these DWT snapshots (Instants) are used to sort entries in the timer queue. The cycle counter is a 32-bit counter clocked at the core clock frequency. This counter wraps around every (1 << 32) clock cycles; there's no interrupt associated to this counter so nothing worth noting happens when it wraps around.

To order Instants in the queue we need to compare two 32-bit integers. To account for the wrap-around behavior we use the difference between two Instants, a - b, and treat the result as a 32-bit signed integer. If the result is less than zero then b is a later Instant; if the result is greater than zero then b is an earlier Instant. This means that scheduling a task at an Instant that's (1 << 31) - 1 cycles greater than the scheduled time (Instant) of the first (earliest) entry in the queue will cause the task to be inserted at the wrong place in the queue. There some debug assertions in place to prevent this user error but it can't be avoided because the user can write (instant + duration_a) + duration_b and overflow the Instant.

The system timer, SysTick, is a 24-bit counter also clocked at the core clock frequency. When the next scheduled task is more than 1 << 24 clock cycles in the future an interrupt is set to go off in 1 << 24 cycles. This process may need to happen several times until the next scheduled task is within the range of the SysTick counter.

In conclusion, both Instant and Duration have a resolution of 1 core clock cycle and Duration effectively has a (half-open) range of 0..(1 << 31) (end not included) core clock cycles.

The capacity of the timer queue is chosen to be the sum of the capacities of all schedule-able tasks. Like in the case of the ready queues, this means that once we have claimed a free slot in the INPUTS buffer we are guaranteed to be able to insert the task in the timer queue; this lets us omit runtime checks.

The priority of the system timer can't set by the user; it is chosen by the framework. To ensure that lower priority tasks don't prevent higher priority tasks from running we choose the priority of the system timer to be the maximum of all the schedule-able tasks.

To see why this is required consider the case where two previously scheduled tasks with priorities 2 and 3 become ready at about the same time but the lower priority task is moved into the ready queue first. If the system timer priority was, for example, 1 then after moving the lower priority (2) task it would run to completion (due to it being higher priority than the system timer) delaying the execution of the higher priority (3) task. To prevent scenarios like these the system timer must match the highest priority of the schedule-able tasks; in this example that would be 3.

The timer queue is a resource shared between all the tasks that can schedule a task and the SysTick handler. Also the schedule API contends with the spawn API over the free queues. All this must be considered in the ceiling analysis.

To illustrate, consider the following example:

# #![allow(unused_variables)]
#fn main() {
#[rtfm::app(device = ..)]
const APP: () = {
    #[task(priority = 3, spawn = [baz])]
    fn foo(c: foo::Context) {
        // ..
    }

    #[task(priority = 2, schedule = [foo, baz])]
    fn bar(c: bar::Context) {
        // ..
    }

    #[task(priority = 1)]
    fn baz(c: baz::Context) {
        // ..
    }
};
#}

The ceiling analysis would go like this:

• foo (prio = 3) and baz (prio = 1) are schedule-able task so the SysTick must run at the highest priority between these two, that is 3.

• foo::Spawn (prio = 3) and bar::Schedule (prio = 2) contend over the consumer endpoind of baz_FQ; this leads to a priority ceiling of 3.

• bar::Schedule (prio = 2) has exclusive access over the consumer endpoint of foo_FQ; thus the priority ceiling of foo_FQ is effectively 2.

• SysTick (prio = 3) and bar::Schedule (prio = 2) contend over the timer queue TQ; this leads to a priority ceiling of 3.

• SysTick (prio = 3) and foo::Spawn (prio = 3) both have lock-free access to the ready queue RQ3, which holds foo entries; thus the priority ceiling of RQ3 is effectively 3.

• The SysTick has exclusive access to the ready queue RQ1, which holds baz entries; thus the priority ceiling of RQ1 is effectively 3.

When the "timer-queue" feature is enabled the spawn implementation changes a bit to track the baseline of tasks. As you saw in the schedule implementation there's an INSTANTS buffers used to store the time at which a task was scheduled to run; this Instant is read in the task dispatcher and passed to the user code as part of the task context.

# #![allow(unused_variables)]
#fn main() {
const APP: () = {
    // ..

    #[no_mangle]
    unsafe UART1() {
        const PRIORITY: u8 = 1;

        let snapshot = basepri::read();

        while let Some(ready) = RQ1.split().1.dequeue() {
            match ready.task {
                Task::baz => {
                    let input = baz_INPUTS[ready.index as usize].read();
                    // ADDED
                    let instant = baz_INSTANTS[ready.index as usize].read();

                    baz_FQ.split().0.enqueue_unchecked(ready.index);

                    let priority = Cell::new(PRIORITY);
                    // CHANGED the instant is passed as part the task context
                    baz(baz::Context::new(&priority, instant), input)
                }

                Task::bar => {
                    // looks just like the `baz` branch
                }

            }
        }

        // BASEPRI invariant
        basepri::write(snapshot);
    }
};
#}

Conversely, the spawn implementation needs to write a value to the INSTANTS buffer. The value to be written is stored in the Spawn struct and its either the start time of the hardware task or the scheduled time of the software task.

# #![allow(unused_variables)]
#fn main() {
mod foo {
    // ..

    pub struct Spawn<'a> {
        priority: &'a Cell<u8>,
        // ADDED
        instant: Instant,
    }

    impl<'a> Spawn<'a> {
        pub unsafe fn priority(&self) -> &Cell<u8> {
            &self.priority
        }

        // ADDED
        pub unsafe fn instant(&self) -> Instant {
            self.instant
        }
    }
}

const APP: () = {
    impl<'a> foo::Spawn<'a> {
        /// Spawns the `baz` task
        pub fn baz(&self, message: u64) -> Result<(), u64> {
            unsafe {
                match lock(self.priority(), baz_FQ_CEILING, || {
                    baz_FQ.split().1.dequeue()
                }) {
                    Some(index) => {
                        baz_INPUTS[index as usize].write(message);
                        // ADDED
                        baz_INSTANTS[index as usize].write(self.instant());

                        lock(self.priority(), RQ1_CEILING, || {
                            RQ1.split().1.enqueue_unchecked(Ready {
                                task: Task::foo,
                                index,
                            });
                        });

                        rtfm::pend(Interrupt::UART0);
                    }

                    None => {
                        // maximum capacity reached; spawn failed
                        Err(message)
                    }
                }
            }
        }
    }
};
#}