Hacker News Reader

Comment by tremon | Hacker News Reader

We'll have to wait and see if they publish a public post-mortem, but this is a textbook real-time scheduling problem and it surprises me. Scheduling theory teaches to abandon processing (a frame/image in this case) if the result is late, both to avoid doing unnecessary work and to recover from sync/latency issues.

From the minimal descriptions I've seen so far, it seems like they were using producer/consumer work queues for the image processing without further synchronization.

View on HN

Is it "just a scheduling problem"? For example if the system is designed to expect every frame being delivered and the algorithms can't handle "incomplete" data (a frame drop), then you have to redesign your algorithms. From the descriptions of the problem it sounded to me like a similar situation - it expected the N+1-th frame to actually be the N-th frame and treated it as such.

Say we would assume this is a complete real-time system. How would a result be late? It seems like a failure mode that doesn't exist in that domain (given that all guarantees are verified).

Now, I get that this is probably not a real-time system.

It’s the first time they are using commodity hardware on mars.

I believe the CPU is the same that has been used in a few smartphones.

So it’s not a real-time system.

https://9to5google.com/2021/04/19/nasa-ingenuity-qualcomm-sn...

Can’t you have a real-time kernel?

There is no single CPU; it's a multitiered control system with at least two of the tiers automotive/military grade components for realtime systems.

ref. https://news.ycombinator.com/item?id=26907669 for more details for example.

There's 2 copies of memory and a RAD hardened FPGA that compares them and looks for bitflips. If it detects one then it restarts, so quickly it can do this midflight with no issues.

Why would a smartphone CPU preclude the use of an RTOS?

IT doesn't apriori preclude an RTOS, but multitasking OSs from unix heritage have so much 'free stuff' (networking, instrumentation, high-quality compilers, experience, etc) they are often the choice. For reasons I don't fully understand, and I'm in 'the business', true real-time embedded systems tend to have relatively simple TCP/UDP/IP stacks, simple https, and by simple I mean 'less robust than those on unix'. I think a fair amount of that is due to power budgets - embedded tend to be small or tiny. However, in this case, having that super beefy motor system to drive the blades taking many many watts, there would seem to be little point in saving microwatts in the MPU. So, one could do an RTOS on a smartphone chip, but it's 'fast enough' for human interactions - so the extra work in getting finite and well-defined real-time response doesn't really make much difference in smartphone apps - if the chip is 'fast enough'.

For RTOS such a chip is always fast enough. The problem is IO latency and software.

A RTOS is typically a much slower and much smaller OS. FreeRTOS e.g consists of 3 small source files, and the control loop might be from 1KHz to 10KHz (for extremely high-dynamic systems). Compare that to the snapdragon loop of 2GHz. Factor 1e6.

https in real-time is obviously a joke. There exist proper networking protocols for hard realtime, not randomized and spiky Ethernet based protocols. Telcos use such proper realtime, slicing protocols.

When conditions under which the code is verified were violated? Like camera chip was supposed to return pictures, CPU was supposed to increment PC reliably, the board was supposed to not experience more than 5000G, etc? I’m not an expert though

Yes, it seems the control system isn't real-time anyway, so this discussion is moot. But to answer your question:

A real-time system can still have inputs that can not be guaranteed, and internal processes can have an upper and lower bound for their execution time. So a result would be late if either:

  1) $T_{input} + t_{process_min} > T_{deadline}$
  2) $T_{input} + t_{process_actual} > T_{deadline}$

(where $T$ is a timestamp given by some time source and $t$ is process duration). Condition 1 can be determined on arrival of the data, and the input can be discarded immediately. Condition 2 cannot be determined beforehand, but if the processing isn't finished at $T_{deadline}$, the process/thread could be killed without waiting for it to complete.

Of course, this requires that an accurate deadline can be determined for each input packet. The textbook use case is for rendering live video streams, where stream latency is more important than rendering each frame accurately. This flight control system is a similar use case, since the utility of the camera feed for determining location or drift rapidly declines as the picture ages.

But if the timestamp-keeping isn't accurate as it seems in this case, it really doesn't matter if the system was real-time or not.

It seems to me that the problem was too much scheduling.

If it really were just producer/consumer workflows one frame would have been lost, the program would be comparing images two frames apart and seen it moving twice as fast as it was supposed to, this would have caused a minor excursion but then the next frame would be right and it would very quickly have settled back down to proper flight.

The problem is that it wasn't simply comparing frames, but it had an internal clock the frames were being compared against.

> this is a textbook real-time scheduling problem

Drive by question: do you have recommendations on good literature for real time systems?

Reply on news.ycombinator.com