Defensive Programming
interrupt.memfault.com
interrupt.memfault.com
I'm actually really curious what you all do in C/C++ to prevent bad operations from ever being performed in the first place.
For example, should we just change our internal malloc / free to use double pointers instead?
bool malloc(size_t n, void **ptr) {
*ptr = <memory>
}
void free(void **ptr) {
... <free>
*ptr = NULL
}
This way, if anyone actually tries to use that pointer, it will crash the system (hardfault), instead of potentially using corrupted memory, which IMO is much worse.1. Go as far as you can with static memory. Don't call the allocator: its slow, sometimes requires inter-thread communications.
2. Dynamic uses often can use the stack: almost always in L1 of your local core. Stack allocations are self-cleaning, just don't pass those pointers to anyone else.
3. unique_ptr<blah> covers most of the true dynamic issues.
4. shared_ptr<blah> covers the rest of them.
---------
The above advice is just general-purpose / low-performance code.
If you're entering high performance coding (aka: you're actually worried about fragmentation, memory sizes, and such), then things get more complicated. Embedded is often RAM-restricted, so you end up in this complicated place where you need to think of highly-efficient techniques.
That's almost always a complexity / resource tradeoff. Efficient techniques are just innately more complex than general purpose techniques. Try not to reach for efficient coding unless you know its something you need to do
But they should also be very careful when deciding when to use the stack. Uncontrolled input + alloca = a bad time.
void *p1, *p2;
malloc(100, &p1);
p2 = p1;
...
free(&p1);
*p2; // oops!
Obviously its trivial in an example like this, but when p2 leaves the lexical scope then there's a problem. p2 = p1; p1 = NULL;
However if you are intentionally doing multiple ownership, then yes you can still have problems.For performance critical code running on a more powerful system with an OS, it's still good to stick with static allocation, or malloc and free only during initialization time.
Over the last decade I've written firmware for space ships, self driving cars, and autonomous aircraft, and I've never needed more than an allocate only pool or an O(n) LRU cache data structure. I've written plenty of software that was hard on the heap, but rarely does memory need to be freed during runtime, and it's rare to need raw pointers in C++ when writing such code.
Ideally, your "firmware" is written in a way that abstracts away the target hardware enough that it's basically just weird software. Then, you can unit test it, and even run linters and memory sanitizers on it.
Oh you sweet summer child. Null pointer deference is undefined behaviour, the compiler will happily turn it into corrupting memory under the right circumstances.
Should have clarified. On the embedded systems I work on, generally Cortex-M4, you set an MPU region on address 0x0 and it turns into a hardfault, so it's pretty easy to catch.
That said the compiler isn't infinitely smart (thank god) and complex null derefs will "safely" make it to runtime. (What a sad world we're in.)
Nim for example does that, and has compiler options for embedded systems.
https://nim-lang.org/docs/nimc.html#nim-for-embedded-systems
A very valuable concept either way: if a function encounters an error and can't somehow resolve it and still do its work successfully, then it should fail and not do something else.
IMHO defensive techniques including recovery should be placed in runtime monitors and supervisors, not in underpinning code which should have a minimum set of behaviors. It should also be kept as simple as possible to avoid code bloat and help reviewers and tools like static analyzers do their job.
Old computing hardware was designed like that. Then software attempted to replicate hardware "exceptions", however not realizing that the model was intrinsically different and completely exploding in intricacies dealing with interrupts, stacks, memory management and threads.
In a similar way, supervisors should be as decoupled as possible from the target code to provide reliability and avoid overall system complexity, with their own set of narrowed down requirements.
Sadly our industry is plagued with historical, ad-hoc programming techniques that are not relevant anymore or were plain empirical bad advice in first place. For example, blindly relying on stateful paradigms, or side effects by design or as an acceptable consequence of certain programming idioms.
What's unsaid in this article is that your ability to break noisily varies a lot based on what the end user's expectations are and how much of the device you own.
If you're building a single-purpose embedded device, sure, you can probably halt and catch fire if its use-case allows doing so safely. This might involve making sure the outside world is safely brought to a halt, but that might be feasible if you're building a device that works on a production line.
On the other hand, you almost certainly don't want to bugcheck some random big corporate file server that might be doing goodness knows what else to get a crash dump. At least not without the blessing of the client.
Like so many other things in engineering, how to handle errors depends greatly on the use case, the users, and the maintainers (which might or might not overlap with the users). You can't baldly claim that one approach or another is best without understanding those.
I've spent some time with "offensive" programming in the context of both security (like key leakage) and safety (like firmware integrity). It was just kinda funny thinking about meetings spent determining what we could do if i2c or something wasn't responding, and different approaches to try and fix it, and then a year later going back and saying, ok, if something isn't working, we've got an easy answer. Lockdown and halt boot. Anything seems weird, just stop trying.
Absolutely. Probably the most common difference I've seen in "Debug" and "Release" builds relying on the standard library headers is when NDEBUG is defined as a macro when <assert.h> is included, which causes assert to become a no-op [0].
This has interesting implications when developers accidentally rely on the side effects of functions and assert on the return value of that helper. On "Release" builds that side effect would be compiled out. For that case, clang-tidy has the bugprone-assert-side-effect [1] checker if you want to enforce this in your CI.
[0] https://en.cppreference.com/w/c/error/assert
[1] https://clang.llvm.org/extra/clang-tidy/checks/bugprone-asse...
But it is really nocive in the context of real products. This is how you finish with the samsung blu ray player that became suddenly a brick. Unusable and unupdatable to be fixed.
Also, there used to be a lot of Android phone that became unusable bricks because they were stuck in boot loops and nothing could be done to go to a recovery or update stage.
That case was common for example when there was a few corrupted or dead bits in internal ram or storage.
Static assertions are enabled by dependent typing. But unlike dependent typing, static assertions are only assertions, for example they do not enable you to write a function that returns different types based on its arguments. Static assertions do not allow you to write a compile-time vector type.
- Dependent types aren't easily available everywhere. If you don't have them then you can't use them.
- The compiler can't prove some statements (admittedly not a big issue for the examples from TFA). Supposing you have a dependent type system which allows you to state that a property holds, it could still be prudent to have a runtime check. If the type system doesn't allow such statements then it isn't sufficiently expressive to solve the problem in the first place.
- Assertions are a local change which one can make incrementally, whereas changing the types being used will immediately require larger changes (if the types are used more than once) or necessitate some extra wrapper code.
Defensive programming does not mean that you ignore errors. It means that the dev should always expect the worse and decide right away how to deal with it early.
Imagine you have a car that suddenly block and explode each time an unexpected error is encountered. So that you are sure to catch it.
So, you are driving on the highway, then a there a window fuse that blow and because of that your car decide to brake and explode...
Especially in hardware and embedded, you have to expect the unexpected to happen.
Imagine the car will suddenly stop when a lightbulb break or there is a shortcut in windows control. Sure you will know, but you will have 300 000 cars all around the world bricked!
If the light sensor or the braking module goes into a completely unexpected state where it can no longer be communicated with, it has corruption, etc, you need to do something about it.
The solution isn't to reboot the car, it's to reboot the individual module and quickly get it back into working order. Rebooting the braking or lighting module should take on the order of milliseconds (if done correctly), and that is much, much better than having a piece of software in a corrupted state for seconds/minutes/hours.
In a car, there is a mothership board that is orchestrating everything, pinging every module, and ensuring everything is in working order. Just like a kubernetes setup, if any one piece fails to ping or looks off, just reboot it and get it working ASAP.
Defensive programming can sometimes look like silently ignoring errors. A haphazard test of defensively written code (that, for example falls back to a different encryption scheme or io device) could pass incorrectly. But there is a place for defensive programming too. I think most important is being aware of which is which.
My first experience with that behavior was with AIX in the early '90s: allocation always succeeds (unless you've exhausted your address space) because it is backed by virtual memory. If you do run low on physical memory, every process gets a SIGDANGER and is expected to free up memory. If that fails, the kernel starts sending SIGKILLs.
Unfortunately, exactly none of the normal utilities understood the protocol and upon running low, the kernel just started shooting processes in the head until the situation improved. For some reason, one of the first processes chosen was usually inetd, meaning you could no longer remotely log in to see what was going on.
When I complained, I was told that the machines were frequently used for scientific computing, where they would allocate giant arrays which were then used sparsely, so it was the intended behavior. And that the X server at the time made use of the behavior.
So, ... yech.
$ uname -a
Linux mimeomia 5.9.11 #1-NixOS SMP Tue Nov 24 12:39:15 UTC 2020 x86_64 GNU/Linux
$ cat main.c
#include <stdio.h>
#include <stdlib.h>
int main() {
printf("malloc: %lx\n", malloc(128LL << 30)); // 128GiB
perror("malloc err");
}
$ gcc main.c -o main && ./main
malloc: 0
malloc err: Cannot allocate memoryYou can't adopt a defensive / offensive stance with regards to out-of-memory issues, because on Linux... malloc doesn't fail when you run out of physical memory.
Your "error" occurs on a pointer-dereference, which could be anywhere in your program. Once more memory is dereferenced than the system has capacity for, the OOM-killer is activated. There's no defensive stance possible, aside from setting overcommit to "Type2" (disabling the overcommit behavior of the Kernel).
But that's somewhat non-standard, and possibly non-helpful, since some applications depend on overcommit.
Yes, overcommit makes things tricker, but it is hardly “impossible”
All this does is run the OOM killer, which doesn't necessarily kill your process. Maybe its a process you didn't care about, maybe its a process you actually are communicating with and suddenly you're now in a weird deadlock state. Etc. etc.
Even if YOU touch all your memory like you're supposed to, there's no guarantee that OTHER processes have touched their memory.
So you touch memory unnecessarily: which means you're now officially reserving RAM (wasting it). Then, some OTHER process touches their memory and runs the OOM-killer. That wouldn't have happened if you just didn't touch your RAM.
-----------
If Linux Overcommit is on default, you live with it and die with it. No point trying to fight OS-policy. Change the policy (aka: Overcommit to 2), and then carefully select applications to run with that follow the same overcommit policy as you.
- Set your watchdog and logging to "never kill". Set watchdog to reboot on OOM, after uploading error report. This way, if the system OOMs, you can log the problems to your logging server, and then watchdog will reboot the system to recover.
- If you have a "UI" process which can also eat all the memory, set it to "kill me first". Yes, the UI will crash, but the rest of system will restart it, and your updater would still work.
- If you have separate resource-intensive process, you can set it to "kill me first" -- then the rest of the system can stay up and maybe even show a descriptive error message.
- The latter approach works pretty nicely for general systems. Is your OOM killer activating because your build system started too many gcc instances? run it with "kill me first" setting -- it inherits to children processes, so you only build processes would be killed, the rest of the system (UI, browser, etc..) would all stay alive.
Firmware is just a pain in the ass to deal with. Wish there was a better way.
Despite the lack of static typing, python has memory safety and a GIL; eliminating a much larger class of bugs, so it's not that hard to believe.
You'd be surprised how many c++ programmers think python is a terrible language just simply due to the lack of type checking and indentations.
A lot of them have no idea how much easier and how much more productive life is outside of the c++ world. Take a look at the other replier. He's thinking "perhaps" it's true, he has no idea.
That same pain-in-the-arse requirement is true of other software, albeit higher-level platforms can perhaps save your arse when bugs are encountered
This had nothing to do with a language saving your arse. This has everything to do with the simple non-existence of a type of bug in other languages, for example memory safety in rust and python.
Additionally c++ often simply does not tell you where an error occurs and you have to use a coredump to figure everything out. It's not an offensive programming strategy by default. Python on the other hand makes every single type of error read like a c++ syntax error giving you a line number and an error type by default.
Another thing to consider is raw speed and lower memory requirements. These are the only advantages of c++. If your hardware isn't strict on the above requirements consider micropython.
I'm finding c++ programmers have this very narrow and cocky view of the world. They turn their nose up at languages like python without truly experiencing how much life is easier and more productive in those worlds.
Not being cocky and not turning my nose up at Python as I know it can be productive and allow quick creation of solutions.
But pointing out that my experience has also encompassed high-resilience, high-availability, low-latency, fault-tolerance, high-security, air-gapped and embedded systems along the way. For the latter of those especially, the rigour of defining requirements and proving them during design and test phases, are key facets that go beyond the choice of any programming language whether modern or old.