Comment by Animats | Hacker News Reader

Comment by Animats | Hacker News Reader

Parent Full thread

So this is basically a CPU with an upper bound on the horrible cases. That's useful for real time. A standard test for real time is to run a hard real time OS like QNX, and have a simple program which receives an interrupt from an input pin and restarts a high priority task that's waiting for the interrupt. The high priority task turns on an output pin. You hook the input and output pins to an oscilloscope. You want to see all the output spikes about the same distance from the input spikes. You don't want to see output outliers way out there, late. If you see that, it's not a hard real time system. You can do this with a standalone program to test the CPU by itself, but you really need to test it when the CPU is also doing other things.

Sources of CPU-level trouble include 1) rarely used CPU instructions that run slow microcode, 2) cases where the pipeline needs a total flush and that's slow, 3) the board manufacturer doing something in system management mode and not telling you. Drivers locking out interrupts too long is the usual Linux problem. It's assumed that all real time code is locked in memory; you do not page real time processes.

Is there any upper bound on latency for something to qualify as a realtime system? For example, a system that is guaranteed to compute its output at the end of each week?

To put a slightly different spin on qwertox's reply: What matters is that the OS can guarantee how long it will take for, say, a high-priority task to become active after it is activated. It's not a matter of the absolute numbers; it's the ability to make a real guarantee. Of course, the bigger the number, the harder it is to make a guarantee at all, because there are more things going on, and are you sure you got all of them? But with a real time OS, there are far fewer things that happen in such a case, and so it's easier to calculate and to prove that yes, it really will respond within X cycles.

There's a lot of subjectivity in these definitions. But an overnight batch job run once a week would usually not be considered a realtime task. The duration of the job and the time resolution of its deadline is pretty large compared to the CPU scheduler timeslices used by general-purpose operating systems, and you probably don't need any special hardware support. You just need to provision enough CPU time overall, and not allow the job to be starved by higher-priority tasks consuming all the available CPU time.

A task would usually be considered realtime to some degree if it cannot automatically get its work done on time even when competing with a background-priority long-running job—for example, if your deadlines are operating on similar or smaller timescales to the operating system's CPU scheduler. Then you need to start taking extra measures to ensure adequate responsiveness, not just adequate throughput.

But there's also the aspect of consequences of missing a deadline. Most realtime tasks are concerned with millisecond or shorter timescales, but a job that must be performed once every 108 minutes to prevent the end of the world would also be reasonably considered a realtime task.

Yeah, it would be nice to have a real-life example with a relatively lengthy latency. Perhaps with delivering an organ for a transplant or something.

At least wrt soft realtime scheduling for audio, I think users/programmers tend to get severely tripped up because the latencies are simply too small for them to make a clear mental model of the underlying system. That, along with their ineluctable desire for their code to "go faster" drives them to request features or design algorithms which would have to travel "back to the future" in order to work at all.

So it would be nice to decouple "ultra-low latency" from "realtime" to make it clear that, say, you can't put the heart on ice after the helicopter already lifted off to fly it to the hospital.

If you're actually paying attention to guaranteed worst-case runtimes to make sure that something completes within a week, then it would be fair to call that a real-time system and real-time techniques would be relevant. If you're running it on a backend with no firm worst-case guarantees and trusting that it won't take too much longer than the average time then that's not a real-time system.

If you're expecting to always get the data in 5 cycles but you sporadically get in 250 cycles, you're no longer on a realtime system. The latency per se doesn't matter, as long as it is what you expect it to be.

Its application-specific:

i.e. A spacecraft that requires a thruster to be fired with accuracy of a few hundred milliseconds in two years time has mission/safety critical timing requirements. Therefore this is hard-real-time

Alternatively, a payroll that has to run at the end of every month will not cause people to miss payments if that deadline is missed by any relatively large amount of time (minutes/hours). The timing requirements are not mission critical at this point. They could be considered soft-realtime tho as there is still an upper bound to desired behaviour.

Maybe you can clear this up for me.

To me, realtime means "responds faster than the fastest it needs to in order to control a feedback loop" which is super subjective since different system dynamics are so different.

When I tried to use RTOS it appeared to be that better than 1ms resolution was not remotely possible. Just hardcoding the interrupts resulted in 1us resolution easily.

I grant that the RTOS provides a scheduler, but when this chip says that it is intended for "realtime" does it mean that it's microsecond fast even when running RTOS, that it's capable of running fast with handwritten interrupts and code (which to me seems to be most things), or just that it has a spec for an upper bound on op timing?

I guess the last is important for correctness proofs but the prior two are how I interpret it as a confused person reading marketing materials put out by conflicting sources.

Actually realtime doesn't mean "responds faster than the fastest it needs to" but rather it means always responds faster than some predetermined threshold. A lot of the easier ways to make things fast (like caches, or even branch prediction) are inherently probabilistic; if the thing is in the cache you get wonderfully fast performance; if not performance falls off a cliff. A realtime system avoids that, making the average case much slower but the slowest case much faster. It basically flattens the latency distribution.

Does this mean realtime doesn't use caches and branch prediction a all? Or worst case of miss for cache or prediction miss is lower than realtime threshold?

A lot of hardware designed to support realtime applications will let you opt in or out of some of those features. So you have some ranges of memory that go through the usual cache hierarchy to off-chip DRAM, but you also have some memory that is directly mapped to on-chip SRAM that isn't a cache and therefore cannot suffer the latency penalty of a cache miss.

Stuff like branch prediction probably isn't too useful to implement as a switchable feature. Instead, the core is designed so that mispredict latency is still reasonably short (short pipeline, in-order execution or OoO with a short reorder window). This means you won't get as much benefit from speculative work as you would on a desktop-class core, but you're probably not going to be as vulnerable to Spectre-style attacks, either.

Its perfectly possible to have realtime requirements and still have variable latency as long as all the variability is bounded. Think video playback or gaming, both of which run on traditional OS's because they tend to be fast enough to meet the frame deadlines. Of course as we all know, the requirements aren't "hard" so dropping a frame/etc is a non critical failure.

So, there is a spectrum of realtime and how "hard" it needs to be. There are a lot of cases where simply having a real time OS on a traditional processor is fine as long as everything can be sufficiently bounded.

In one application I worked on, the caches were turned off completely. Slower, but more predictable latencies, and most importantly, without the huge latency spikes we would occasionally get with caches enabled. (Because of large cache misses.)

> When I tried to use RTOS it appeared to be that better than 1ms resolution was not remotely possible.

I've only worked with one RTOS, and a lightweight one at that.

The scheduler would run every time an interrupt handler finished running.

One such interrupt was the 'systick' - you can choose if you want it to run every 1ms, or 100us, or a different rate (depending on how fast your CPU is)

The systick was how you timed anything you didn't want to use a dedicated hardware timer for. If you want to wait 10ms and your systick happened every 1ms, you waited until the 10th systick.

If you want to time something with finer resolution than 1ms, you either made your systick faster, or used a hardware timer with the interrupt triggering the scheduler.

> Just hardcoding the interrupts resulted in 1us resolution easily.

If your CPU runs at 180MHz, 1us is enough time for 180 clock cycles. Plenty of time to run one reasonably carefully coded interrupt handler, not much time to run 10 interrupt handlers.

As you can imagine, running your systick at 1us with that clock rate would be challenging!

You can also have a 'tick less' scheduler for the systick style timer that won't have a regular period, but instead will dynamically reset the timer in each expiry ISR for the next scheduled event.

Realtime is a fluid (AFAIK) definition. In general, realtime concerns reaction time to inputs - how much (sic) of inputs are processed "in time". On one hand, you may want the system to react "quickly" to 99% of inputs on a pin. On the other hand, you may not care (within reason) when the system responds to UDP packet. To recap, realtimeness concerns amount of events on the same input processed "in time" and number of inputs subject to timeliness. Hence general confusion and multiple definitions.

I would say there are two (2) sets of realtime definitions.

1) concerning amount of events

Hard realtime system always produces output to an input in predefined time (subject to clocking). This time can vary between different inputs, but is fixed for a given input.

Soft realtime systems are bound - output time is bounded to some number.

2) concerning number of inputs

Hard realtime system has bound reaction time to every possible input

Soft realtime system has bound reaction time to "critical" inputs.

---

Definitions used in practice mix the two in some ways. The most prevalent definition I happen to come across in the general internet is something like "realtime system has bounded, low-variation reaction time to critical inputs".

I don't get the impression that in this case "realtime" means anything surprising – that is, it means "responds within a well-known and deterministic time." As you point out, that time is usually slower than a non-realtime system would offer.

Specifically with the R-family ARM chips, there is support for things like tightly-coupled memory (https://developer.arm.com/documentation/ddi0338/g/level-one-...) to provide latency guarantees for real-time code, deterministic interrupts, that kind of thing.

I see, thanks. I can see an advantage to being able to say that your average performance isn't as good but your variation is small, you can prove you'll hit your metric. I guess I usually over-engineer by 10x but maybe a better coder would do a better job of running up against the limits.

As others said, the difference is very significant.

Real-time systems show up especially in industrial control and safety systems. For example, take a logging harvester, which cuts down trees and strips their branches and bark. It has a control panel for moving incredibly violent, dangerous machinery. It is truly critical that you can guarantee that, for example, sensors which detect a problem can abort something within a very tight latency budget.

> average performance isn't as good but your variation is small, you can prove you'll hit your metric

I think you are missing something here.

Real Time systems aren't designed for high performance at all and instead they are designed for predictability.

You aren't aiming to "hit your metric", you are aiming to never, even go below it.

This is useful for something like old-style spinning hard drives where you had to calculate where to move the head of the disk within the time of the rotation. You have a instructions/second budget, and you know it will execute that.

On non-realtime hardware (and software) you know how many instructions/second it can do, but things like speculative execution of caching strategies or non-RTOS mean those instructions are not guaranteed at all.

I'm confused, aren't "don't fall below your spec" and "hit your metric" synonyms? In what field are they not?

"Hit your metric" commonly refers to average (or some percentage of average) performance being within bounds.

It's pretty rare in software for it to define minimum performance. There's good reasons for this - software is interacting with intrinsicly messy things (like humans) and so performance outliers are a big factor.

It's pretty common in software for the worse case to be 10x the "hit your metric" case, and this isn't viewed as bad: the worst case happens rarely enough for the trade off to be worth it.

Ah, thanks for the clarification.

I am coming from the hardware world, through raw assembly, to C so my path is very much the assumption that if the microcontroller is going to control a feedback loop that stabilizes at 10MHz it will be guaranteed to do so.

But I suppose that I actually do that by overspecifying things nowadays since it's much too unpredictable as you say to predict how long things will take.

To me if you say you sample at 100kHz you sample at a minimum of 100kHz. Very very good to know that if I work with a firmware engineer that they won't hear the same thing as I mean if I say that it needs to hit a 100kHz sampling rate.

It would be very disappointing if they wrote a loop designed to average a 100kHz sampling rate with random jitter... (kidding mostly, I'd assume they would at least not do that).

Firmware engineers would probably think in the same terms as you about sampling frequency.

"realtime" means "latency is bounded to some documented value". "Hard realtime" means it never exceeds that value, and has the same latency every time. "Soft realtime" means it never exceeds that value but the actual latency varies.

That value doesn't have to be anything in particular. Could be 1ms. Could be 100ms. Could be a year. It just has to exist and be documented, and never, ever be exceeded by whatever operations are covered by its guarantee.

Your description of hard vs soft realtime doesn't match any that I've previously encountered.

Generally, hard realtime means missing a deadline in unacceptable, but does not necessarily imply that latency is exactly fixed—just bounded. Soft realtime means that the consequences of missing a deadline are less severe, so the system may allow for some missed deadlines: eg. dropped packets for a video conference or delayed frames for a video game are noticeable especially if they become common, but they don't crash the whole system and the system remains useful even after missing a deadline.

>and has the same latency every time.

There's no such requirement.

All hard realtime needs is the guarantee that the value will never be exceeded.

Right. Should have written "same maximum every time". Went stupid for a minute.

Reply on news.ycombinator.com