mark_undoio

> Sure, it doesn't help much for some scenarios (one I've heard people mention is multithreaded code, where logs are better?), but for most people it's not that far from a superpower.

Debuggers can be great for understanding multithreaded code - and you can potentially freeze threads and continue others in order to provoke a particular race condition.

However they're potentially quite weak at stepping through a concurrency bug - stopping after each line to understand the sequence of events has a good chance of making your bug go away.

I'd say you want Time Travel Debugging if you need to capture and step through a rare event: you get to record the bug happening (without interrupting it) and then step through the recording.

On Linux, Undo.io (disclaimer: where I work) and rr (open source) are good at this.

On Windows, you have Microsoft's own Time Travel Debug solution: https://learn.microsoft.com/en-us/windows-hardware/drivers/d...

(nb. there's also GDB's built-in process record technology but I'd recommend against that for any non-trivial software as the overheads are very high)

You could also use GDB's Dynamic Printf (https://sourceware.org/gdb/current/onlinedocs/gdb.html/Dynam...) to do the logging directly from GDB.

Essentially you set it like a breakpoint (attaching a printf style string to a code location) and then just "continue" until you've gathered what you want.

Even indoor bouldering has had an effect for me - my thicker skin just doesn't always register well on my phone screen.

Tangential but I keep wishing for an MS Space Simulator revival - I realise there are other games that scratch that itch these days but I did enjoy how very seriously it took accurate simulation (plus occasional silly touches like the various futuristic spacecraft).

Yes, this is true. Genuine parallelism will absolutely be a problem there - similarly to how it is for user-level record/replay tech. And serialising is going to have more impact when you're recording a whole OS vs just one application.

Microsoft's TTD (a user-level tech) does allow genuine parallelism, I believe. But my understanding is that it has slower (though still very impressive) single-threaded performance as a result, so there's a tradeoff. But perhaps you could do something more like that.

We believe it's possible to handle parallelism in record/replay tech while still having good single-threaded performance - but it implies a much more complex implementation to do so safely.

> Presumably rr, focusing on processes, offers a more constrained environment that's easier to log & replay.

As someone who's worked elsewhere on time travel debug, I'm really curious on @roca's take on this - because I'd have expected a full-VM solution to be easier to make reliable.

Hardware-level behaviour sounds harder but it's well-constrained. The behaviour an OS can rely on from the hardware is large but well-documented and slow to change.

In contrast, the process boundary is really ill-defined and permeable. Also, when you need things to be precisely the same, you notice bits of kernel behaviour leaking into the user space ABI in unexpected ways.

> Even then, you still want snapshots to jump to a point in time. The only annoying thing is the case where you want to step back one, meaning a naive implementation would jump back to the last snapshot and play forward.

At Undo.io we use similar techniques to time travel debug C/C++ (and other languages).

> An 80% solution is to keep the last N states in memory.

We use a slightly more specialised version of what you describe - we maintain a set of snapshots at wide spacings throughout history (for big time jumps the user might want to do) and ones just a small gap before the "current time" the user is viewing, so small steps backwards can be fast too.

It's been a complex process figuring out an algorithm that balances these concerns without using up unreasonable amounts of RAM.

The other way to tackle slowness - though this is for larger jumps in history - is to search the timeline in parallel. Since you've got deterministic re-execution you can play several chunks of history at once while looking for a key event. It can't help for small jumps in time but if you are looking far into the past it can be a significant speed-up.

I've heard if the company really wants stuff to get done efficiently then they need to put a lawyer on the product team - i.e. change the incentive from "prevent engineering creating risks, even at the cost of them failing" to "help engineering manage risk whilst succeeding".

GDB is happy to deal with separate debug info files from the executable code - and has been approximately "always", as far as I'm aware. But it's not particularly common / well-understood how to actually achieve it.

Some info here on how to configure GDB to use it: https://sourceware.org/gdb/current/onlinedocs/gdb.html/Separ...

The old-school way appears to be to extract the debug information from the binaries after compilation, then strip the binaries. As described here: https://stackoverflow.com/questions/866721/how-to-generate-g...

The new way is to use gcc's ability to generate split DWARF directly: https://interrupt.memfault.com/blog/dealing-with-large-symbo...

This will work with debuginfod but you don't have to have that running to use these - you can just supply the symbol directory when you want to debug.

I've seen gcc do an enthusiastic job on `static` functions within a C compilation unit - specifically where they only have one call site and so can be fully inlined into the caller.

In that case, the code did become pretty hard to debug due to the extensive inlining and reordering it had allowed. Unfortunate because the only reason such functions exist is to make the structure of the code more apparent!

Maybe that's an exception (and / or maybe it's easier with C than for C++).

Maybe the less is that it's still always worth trying function call boundaries, in case the compiler has been conservative!

This one, I believe: https://github.com/libunwind/libunwind

ETA: Thinking about it, I'm not really sure what it'd do for C++ - I guess you'd end up with mangled names, so if you want sensible names you might need to demangle (either as a post-processing step or within the dumper) too.

I don't think you'll get any decoded argument values out of it either, so I guess it depends what backtrace info is needed.

If you're on *NIX have you tried just invoking gstack or similar as an external process? https://linux.die.net/man/1/gstack

Or, indeed, getting a core dump and applying GDB to it. GDB seems generally pretty good at reconstructing stacks at arbitrary points in application runtime.

We've also used a combination of libunwind and https://linux.die.net/man/1/addr2line to produce good crash dumps when GDB is not necessarily available.

There's `debuginfod` on Linux: https://developers.redhat.com/blog/2019/10/14/introducing-de...

It builds a lot on quite a simple conceptual base, benefiting from native support in gcc / clang (for embedding unique build IDs) and in GDB (for contacting the server). It can serve up both source and symbol information.

I would like to see this adopted more - e.g. build infrastructure automatically populating a debuginfod server so debugging is seamless.

The other trick I've found really helpful for "optimized out" values is to find places where they cross boundaries that block optimizations (e.g. procedure calls to another translation unit, so long as you're not doing some kind of link-time optimisation).

e.g. if the value you're interested in is being passed to / returned from a function then inspecting it around the call / return site should have the value available.

> It’s a massive pain to have everything be marked as “optimized out” and reassemble the things you want from other variables or by using a disassembler to manually track which register the value is hiding in.

If you've got a time traveling / reversible debugger than you can (sometimes) go back to a point where the value was being written / used, at which point it'll often reappear in scope and be accessible.

I believe DWARF's built-in virtual machine should be able to recompute missing values in many cases but I don't think compilers are great at putting the relevant info in, even where it should be possible to compute the right value fairly easily.

Good to see this discussed - debuggability is not talked about enough but, done right, it could be a superpower.

Setting the build for an old x64 machine (https://dhashe.com/how-to-build-highly-debuggable-c-binaries...) for reversible / time travel debuggers seems unnecessarily restrictive to me. I'd expect a modern time travel debug tool (e.g. either rr or Undo - disclaimer, which I work on) to cope fine with most modern instructions (I believe GDB's built-in record / replay debugging tends to be further behind the curve on new CPU instructions - but if you're doing anything at scale it's not the right choice anyhow).

Regarding compilation (https://dhashe.com/how-to-build-highly-debuggable-c-binaries...) - we generally advise customers to use -Og rather than -O0. As the article states, this will still optimise out some code but should be a good trade-off without being too slow. (NB. last I checked, clang currently uses -Og as an alias for -O1, so it may behave less satisfactorily than under GCC).

It's also not said enough but: you don't need a special debug build to be able to debug. It's less user-friendly to debug a fully-optimised release build but it's totally possible. You just need to retain the DWARF debug info (instead of throwing it away). This is really important to know if you're debugging on a customer system or analysing a bug that's only in release builds.

Debuggers, though "one more thing to think about" whilst learning do actually make things easier to understand for a beginner.

I try not to introduce them too early because lots of concepts at once gets frustrating - but eventually you get to a point where it clearly would save stress to be able to step and inspect state.

> Go was originally pitched as a systems programming language, a better C, like you said.

I've always understood this claim as a particular (rather restricted) interpretation of "systems" code - meaning network servers, basically (maybe also CLI applications?)

I'd always thought of "systems" as including things like writing standard libraries, kernels and embedded code. But I guess there is a decent slice of other systemsy stuff that Go is good for.

Whereas something like Zig feels like it could really be a better systems programming language in general that C, in all of its different application areas.

That flashed me back to a vague memory that the film 10,000BC showed mammoths in use building the pyramids, which I thought was incredibly silly (well, it was...). But it's kind of satisfying to know that the building at least overlapped with the mammoths in real life.

One thing that struck me about the Z1 was how closely it paralleled things I'd expect to see in a silicon implementation of a CPU:

I recall seeing: different physical blocks for different logic, a big regular blog for the memory and other blocks for ALU, etc, addressing handled by column and row selection lines, evaluating bit values by putting a signal onto the data line and seeing whether the value changes...

But all implemented with mechanical linkages (so a signal is a pull of a rod, a connection is a hooking of a rod onto some other linkage, etc.

A little over 10 years ago I was doing some very resource-constrained embedded programming. We had been using custom chip with an 8051-compatible instruction set (plus some special purpose analogue circuitry) with a few hundred bytes of RAM. For a new project we used an ARM Cortex M0, plus some external circuitry for analogue parts.

The difference was ridiculous - we were actually porting a prototype algorithm from a powerful TI device with hardware floating point. It turned out viable to simply compile the same algorithm with software emulation of floating point - the Cortex M0 could keep up.

Having said all that though: the 8051 solution was so much physically smaller that the ARM just wouldn't have been viable in some products (this was more significant because having the analogue circuitry on-chip limited how small the feature size for the digital part of the silicon could be).

Obviously that was quite a while ago! But even at the time, I was amazed how much difference the simpler chip made actually made to the size of the solution. The ARM would have been a total deal breaker for that first project, it would just have been too big. I could certainly believe people are still programming for applications like that where a modern CPU doesn't get a look in.

> I think the only scenario where it makes sense is a difficult-to-reproduce bug that you're actively looking for.

That's a great use case for time travel debugging - do you have a sense why you've not found an opportunity for it?

IMO there are a few other good use cases. One is just to understand code, especially if you're working on a very large project with other people - seeing how / why you got to a source line can be very helpful.

I think the implementation in C++ put it out of fashion, as later languages (e.g. Java) deliberately restricted it to avoid the complexity. The main criticism I saw was the potential for (variants of) the "diamond" where A is subclassed by B and C, then both of those are subclassed by D. Does D get two copies of A's state? It's hard to come up with an intuitive behaviour.

More recently, the move seems to be away from class based object orientation (including inheritance) entirely.

On the other side of things, I've never heard people talk about Python's multiple inheritance with the same tone used for C++ - but then there are cultural differences in the language communities too.

I think it's a shame this wasn't highlighted more explicitly in, at least, my computer science degree.

We did finite state machines in quite a bit of detail, vaguely referred to pushdown automata but then concentrated on lambda calculus for quite a lot, with Turing machines less so.

The Turing machine description, without showing it in this hierarchy, seems like a really weird abstraction. Fitting it in here (and combining it with the different Chomsky classes of grammar) makes it all feel more joined up.

Agreed. Very different gameplay, setting, everything really. Just two good games, weirdly sharing a name!

On a holiday with friends I once spent some time on a beach trying to demonstrate a computer could be built with sandcastles and tennis balls.

IIRC, we managed to get AND, OR, NOT, XOR working and I set about building a half adder. It needed a lot of beach space, since it was a fairly shallow slope and you needed the balls to get up enough speed to work the logic.

The end result was, unfortunately, that the sea started washing away my design before I quite managed to debug it.

Ultimately I'd have needed to find a way to build a bigger chunk of logic and keep the balls circulating around it - didn't get that far but felt ready to do so if a post-apocalyptic scenario called for it.

We converted our product (which I'd say is medium-sized - it's not a sprawling enterprise codebase but it's the work of a team over many years) to use Meson for its C / C++ components and I've been very pleased with it.

It's pretty straightforward to understand, has good documentation and it's fast.

EarthSiege 2 was amazing - many hours spent there. I used to multiplayer with a friend: one of us would steer the HERC's legs and one the turret.

When I bought Free space it was actually titled Descent: Freespace - The Great War.

I wish the series had carried on, I really wanted to find out where the confusing plot threads of Freespace 2 went!

I remember hearing that it was Ninja in the rest of the world at some point as a child and being very confused by it.

Some discussion on that here: https://www.reddit.com/r/TMNT/comments/11phg3p/did_anybody_k...