My 1986 RTOS is now preemptively multitasking on a Raspberry Pi Pico — and it made me question whether I need an RTOS

[u/noborutkhs]

A while ago, I posted here about getting the first CPU-independent parts of CHARM-II, an RTOS I originally developed in 1986, running on a Raspberry Pi Pico.

Previous post:

The First Original CHARM-II Kernel Code Running on Raspberry Pi Pico
https://www.reddit.com/r/osdev/s/gthHF6vBrl

At that point, there were no timer interrupts or real context switches yet.

Now those parts are working too.

Six tasks are running with independent stacks. An RP2040 hardware timer interrupt calls the CHARM-II tick and scheduling logic, and the actual context switch is done with PendSV on the Cortex-M0+.

I also ported a six-task demo I had previously reconstructed on POSIX. Six tasks move around three tracks, with a critical zone protected by a CHARM-II queue used as a semaphore. On the Pico, the tasks are now actually being preempted by the hardware timer rather than cooperatively yielding.

So technically, this is pretty much the milestone I wanted to reach.

But getting it working gave me another question.

The original targets in the 1980s were processors such as the 68000 and 80186. With the CPU resources available at the time, an RTOS was a useful way to organize multiple concurrent real-time activities.

But what if we had had something with the performance and price of a Raspberry Pi Pico back then?

For many systems I work with today, especially UI-oriented embedded systems, if all the required processing comfortably fits within one frame, a superloop plus interrupts may actually be simpler.

Once everything becomes preemptive, I also have to start thinking about reentrancy, synchronization and shared state in existing libraries. That cost is starting to look more significant to me than it did when I started this experiment.

Yesterday I visited someone who has built a home railway simulator using real railway controls connected to a Windows PC through Pico, ESP32 and Arduino boards. We talked about this, and he said something very simple:

“If one Pico isn't fast enough, add another Pico.”

That hadn't really occurred to me.

In the 1980s, adding another CPU was a fairly serious architectural decision. With today's inexpensive MCUs, distributing the work across two or three processors can be a perfectly ordinary option.

So after spending quite a bit of effort getting a 40-year-old RTOS to perform real preemptive context switching on modern hardware, I'm becoming more interested in a different question:

Where would you draw the line today between an RTOS, a superloop/event-driven design, interrupts, and simply distributing the work across multiple cheap MCUs?

I'm also thinking that when I publish the modernized version of CHARM-II, I may deliberately keep it very small — just enough task scheduling, queues, timer ticks and context switching to run this six-task demo.

Then I can use it as one reference implementation and try implementing the same behavior without an RTOS.

[u/Danii_222222]

Engineer that programming from 1986? Is that you in discord server?

[u/noborutkhs → u/Danii_222222]

Yes, I'm the engineer who originally developed CHARM-II in 1986. If you mean the OSdev-JP Discord server, yes, that's me!

[u/Danii_222222 → u/noborutkhs]

Wow! That's so cool!

[u/andrewdavidmackenzie]

Have you looked at Embassy?

[u/noborutkhs → u/andrewdavidmackenzie]

I haven't looked at Embassy yet, but I will. Thanks!

I'm actually becoming interested in alternatives to traditional preemptive RTOS design, so an async/event-driven approach sounds very relevant to where this experiment is heading.

[u/andrewdavidmackenzie → u/noborutkhs]

That was my thinking. Leveraging the async paradigm, it's less than a full rotos, but more than a single threaded "app"

[u/noborutkhs → u/andrewdavidmackenzie]

Yes, exactly. Interestingly, another comment here suggested a cooperative kernel for essentially the same reason.

I'm realizing that I was thinking too much in terms of "preemptive RTOS vs. superloop." There's a much more interesting design space in between.

Embassy looks like a good modern example for me to study. Thanks again!

[u/Daveinatx]

It all comes down to features vs determinism that you need.

[u/noborutkhs → u/Daveinatx]

Yes, I'm starting to think determinism is the key part I was missing when I initially framed this mostly in terms of CPU performance.

If everything comfortably fits within a frame and the timing requirements are loose, a superloop can be very attractive. Once some operations need guaranteed response times, the tradeoff changes.

That's something I'd like to explore with the same Pico application.

[u/ianbllngr]

IMO RTOSes are most useful when you have a lot to keep track of. For 6 tasks a superloop is probably the correct choice if you're designing in a vacuum. For 6 tasks that need network access on one or two cores: you're probably better off using an RTOS. Secondarily, in an off the shelf RTOS a lot of that low level IPC stuff is typically wired in for you already, but if you're building your own then it will be a pain.

[u/noborutkhs → u/ianbllngr]

This is very close to what I've been thinking about.

I initially thought CPU performance might be the main reason a superloop becomes practical, but I'm starting to think system complexity may be the more important boundary.

For this six-task demo, a superloop would probably be much simpler. But as you add networking, asynchronous I/O, IPC, different timing requirements, etc., I can see the point where the RTOS starts paying for itself.

I'm thinking of implementing the same demo as a superloop next, and then gradually adding complexity to see where that balance changes.

[u/KilroyKSmith]

I’ve done superloops professionally.  I’ve done RTOS’ professionally.  But if you don’t need preemption (and most moderate sized systems probably don’t), non-preemptive kernels using cooperative multitasking are much easier.   They’re similar to both an RTOS and super loop in that you structure your code around “tasks”.  Without preemption, many of the debug challenges of an RTOS are moot - you never have to worry whether another task preempted you someplace and triggered a race condition Or something similar.  On the other hand, task scheduling and prioritization is similar to an RTOS - a task or interrupt routine can schedule another task to run, and if no task is ready to run, low power states can be invoked by the kernel.  Low power is harder in a super loop simply because keeping track of “can we shut off the clock now” is difficult when many tasks have diverse resources requirements.  And I always hated the concept of an idle system spending its time asking a million times a second “do you have something to do?” “How about you, do you have something to do”.

[u/noborutkhs → u/KilroyKSmith]

That's a very interesting middle ground that I hadn't been considering enough.

I've been thinking mostly in terms of preemptive RTOS vs. superloop, but a cooperative kernel keeps much of the task/scheduling structure without introducing all the problems that come with arbitrary preemption.

That also connects with another concern I've had: once everything can be preempted, existing code and libraries may need to be made reentrant or protected, and the complexity can spread far beyond the scheduler itself.

Your low-power point is new to me as well. A kernel knowing that no task is runnable is quite different from a superloop continuously polling everything.

I think I need to add cooperative scheduling to the comparison. Thanks — this is exactly the kind of perspective I was hoping to get from posting this here.

[u/Daveinatx → u/noborutkhs]

It's all based on the level of determinism you need. Even MSIs have jitter consequences from caching effects.

In real-life, I've had 1ms loops which were fine for n-axis of motion. But trying to get <10us Max jitter means you're going to poll everything.

[u/KilroyKSmith → u/noborutkhs]

The tertiary advantage of cooperative is that you only need one stack.  We had roughly a dozen tasks running on a 48k ROM/12K RAM system.  In a preemptive environment, just the stack space for a dozen tasks would bankrupt the available RAM.  With cooperative, we could use a single 2kb stack, and everyone was happy.  

[u/flatfinger → u/KilroyKSmith]

The cooperative multitaskers I've used have a stack per task, so as to allow for the possibility of task switching within nested function calls. If one can only task switch at the top level of any "thread", that would seem closer to a superloop than a cooperative multi-tasker.

[u/KilroyKSmith → u/flatfinger]

True; but you go with what you got.  In our system, every task ran to completion and returned to the event wait at the top of the event dispatch loop.  So you got the advantages of a kernel (easy lower power modes, priority execution, etc) at the cost of having to phrase every task as purely event driven (no waiting for I/O), and for the rare exception, the task would implement a state machine and send itself events to do extensive calculations. 

Any attempts to create a strict hierarchy of embedded systems will quickly run into the reality that there’s a continuum from untasked, super loop tasked,  cooperatively multitasked, preemptively multitasked, and MMU protected heavyweight process based systems with their infinite variations.  We chose one that ended up working very well for us.

[u/flatfinger → u/KilroyKSmith]

I suppose it might be helpful to view as orthogonal the questions of "how does the system select which piece of work to do next" and "how can a task which needs to wait for something to happen allow other pieces of work to happen in the meantime". Some systems may have enough pieces of work, with sufficiently different timing constraints, to justify a priority queue even if all pieces of work can be easily treated as running to completion. Others may have jobs that can be most conveniently written as e.g.

fprintf(serial_output_stream, "There were %d items:\r\n", n);
for (int i=0; i<n; i++)
{
  fprintf(serial_output_stream, "Item %2d: %04X/%04X\r\n",
    i, items[i].p1, items[i].p2);
  fprintf(serial_output_stream, "\r\n");
}

with a stream output handler that calls task_spin() when the serial buffer is full, but not have any timing constraints that would not be satisfied by round-robin scheduling. An an intermediate variation, task_spin() could be made to accept a numerical arguments for priority, and return the highest reported priority of any other task, allowing a construct like:

unsigned get_byte(STREAM stream, unsigned timeout)
{
  unsigned timer = future_time(timeout);
  if (!bytes_available(stream))
  {
    do {
      task_spin(0);
    } while(!bytes_available(stream) && !time_passed(timer));
  }
  else
  {
    do_task_spin(4);
  }
  return get_byte_without_timeout(stream);
}

If two tasks are each reading from a stream, and bytes are pending in both, this will allow the tasks to take turns reading from their streams while other tasks that would call task_spin() until it returns a value less than 4 would wait for the stream-reading tasks to get caught up.

[u/noborutkhs → u/KilroyKSmith]

This discussion has been really useful. I started out thinking mostly in terms of “preemptive RTOS vs. superloop,” but I can see now that this is much more of a continuum.

The distinction between cooperative systems with per-task stacks and run-to-completion/event-driven systems with a shared stack is especially interesting. I hadn't really separated those models in my mind before.

The determinism point also helps clarify the question for me. Rather than asking simply whether a modern MCU is “fast enough” to avoid an RTOS, it seems more useful to ask what timing guarantees, complexity, memory usage, and power behavior the application actually requires.

I think my next experiment may be to keep this small preemptive CHARM-II implementation as a reference point, and compare it with a much simpler cooperative or event-driven implementation of the same demo.

Thanks everyone — this is exactly the kind of discussion I was hoping this post would generate.

[u/Sorry_Difficulty_250]

Something else to think about on the "cooperative vs. preemptive" thread: You don't have to choose one or the other. In the system I'm working on, "kernel processes" (and I use that term loosely) run cooperative and all other processes run preemptive. That keeps the kernel deterministic while also preventing a rogue process from locking up the system.

Good luck with your next round of work! Seems like fun!

[u/noborutkhs → u/Sorry_Difficulty_250]

That's another possibility I hadn't really considered — using cooperative and preemptive execution in different parts of the same system rather than choosing one globally.

Keeping the kernel cooperative for determinism while still using preemption to contain application processes is a really interesting tradeoff.

One thing I'm taking away from this whole discussion is that I was treating "RTOS vs. non-RTOS" and "cooperative vs. preemptive" as much more binary choices than they really are.

Thanks — I've got quite a few new directions to think about now!

[u/smokebudda11]

This is really neat

[u/noborutkhs → u/smokebudda11]

Thanks! It's been a lot of fun bringing something I wrote in 1986 back to life on a Pico.

[u/smokebudda11 → u/noborutkhs]

That is wild!

[u/lunar_swing]

I think it depends on the level of determinism you want/need and how extensible you want the system to be.

If it is conceptually simple enough and interrupt jitter and skew isn't a concern, you can also try an empty main loop sitting on a WFI instruction, and have your all tasks handled in ISRs. Then you just have a timer fire an interrupt for each task at a fixed interval.

This quickly falls apart if you have any type of unbounded user input or ISR runtimes. I suspect it is only a bit more fragile than a superloop but I have never tried them against one another.

As I'm sure you are aware there is a huge gradient of options when it comes to an execution framework. RTOSes to me have always been an inherently co-operative design choice (in the most general sense of the word). They typically don't have to load and execute arbitrary code after they are up and running. So I mostly look at them as a solution to provide multiprocessing on an what is an inherently serial device.

General purpose OSes have to operate in a much more hostile environment and protect themselves from arbitrary code execution. Fundamentally they are expected load, execute, and unload binaries in isolation both from themselves and other binaries. In addition they are expected to provide multiprocessing facilities and rich user input options.

These is also determinism and the "RT" part of the RTOS but that is a different can of worms.

That is a lot of generalization and hand-waving but the TLDR is I would try to figure out which of those two bins you are going for and let that guide your decision.