sidkshatriya

How to record/replay and debug a program using reverse debugging through a small example program written in C.

It's very important for both the compiler tools chains of go to continue working well for redundancy and feature design validation purposes. However, I'm generally curious -- do people / organizations use gcc-go for some use cases ?

fish is amazing. I use it as my primary shell.

But for writing scripts I would reach for Elvish/Nushell. More powerful.

The shell prompt is also a small interface. How your shell responds to tab autocomplete, provides suggestions etc. can be quite helpful. Here I just like the way fish suggests filenames, provides an underline for filenames that exist and so on.

The language is what you write in an $EDITOR. Here Elvish scripts can be nice, succinct and powerful. I like how I don't have to worry about strange "bashisms" like argument quoting etc. Everything feels consistent.

nushell vs Elvish

The Nushell and Elvish scripting languages are similar in many ways. I personally find the "shell" experience better in Nushell than Elvish.

Nushell

- Bigger community and more contributors

- Bigger feature set than Elvish

- Built in Rust (Yay :-)! )

Elvish

- Mostly developed by one person

- Built in golang

- Amazing documentation and general attention to detail

- Less features than Nushell

- Feels more stable, polished, complete than Nushell. Your script written today more likely to work unaltered in Elvish a year down the line. However this is an impression. Nushell must have settled down since I last looked at it.

For "one off" scripts I prefer Elvish.

I would recommend both projects. They are excellent. Elvish feels less ambitious which is precisely why I like it to write scripts. It does fewer things and I think does them better.

Nushell feels like what a future scripting language and shell might be. It feels more futuristic than Elvish. But as mentioned earlier both languages have a lot of similarities.

I don't use Elvish daily (I use fish) but writing scripts in Elvish is a great experience. The elvish executable can serve as an LSP server and that makes writing Elvish scripts a bit easier.

I don't care much for the Elvish shell experience, rather I like the Elvish scripting language. The documentation is top notch and the language evolves slowly and feels stable.

> It just shows the mindset of its devs was a little behind the realities of the industry, or they simply didn't care about concurrency.

OCaml cared about concurrency (e.g. Lwt, Async are old libraries that provide concurrency -- they didn't need multicore Ocaml). OCaml didn't care so much about true _parallelism_ in threads until recently. Parallelism was to be obtained via processes and not threads in pre OCaml 5.0. True parallelism in threads is available in OCaml >= 5.0

Python is actually trying to go multicore too ! OCaml however beat it to the punch. The strengths of Python are elsewhere though, a topic for another day.

Would you rather have a HFT trade go correctly and a few nanoseconds slower or a few nanoseconds faster but with some edge case bugs related to variable initialisation ?

You might claim that that you can have both but bugs are more inevitable in the uninitialised by default scenario. I doubt that variable initialisation is the thing that would slow down HFT. I would posit is it things like network latency that would dominate.

TL;DR Your dad's PHP is no longer the current PHP. The language itself has been thoroughly modernized, made more consistent, bug free and performant. There is nothing better than a piece of software that keeps getting improved decade after decade borne out of real developer needs and experience.

- Type annotations are integrated and work well with PHP now. This results in a kind of scripty Java OOP that is more succinct and type checked ! (it is possible to write PHP without types when you want metaprogramming features also)

- Many inconsistencies in the design of the language (OOP, function naming etc.) have been resolved/smoothed out

- The language recognises the inherent problems with `null` and allows you to make type annotations that rule out null. This is my smell test for a language BTW -- golang for instance does not seem to have a good story for null.

- PHP Language developers have been really good about increasing performance year after year

- Lots of software like web frameworks and CMSes continue to be based on PHP

And -- the greatest part of the design -- PHP remains the best example of the "shared nothing" architecture. An architecture that allows you to scale your application easily.

A good way to penalize planned obsolescence would be to charge a decreasing penalty if the goods are recycled/disposed earlier. So if I return the fridge for recycling after a couple of years (bad fridge) then the company gets charged automatically 5% of the fridge cost. If I recycle after 10 years then the company gets charged zero (as an example).

Maybe instead of a charge this could be a credit. If the recycling happens after a long time the company gets a bigger payback than if it happens before. The money is collected on checkout so the company can't claim bankruptcy or low profits to make the payment.

This is a good design. The company that manufactures the product does not need to be necessarily responsible for the cleanup. The cleanup is done by another company and the costs are added on customer checkout. But this is open to abuse as you mentioned -- some companies may take short cuts or the cleanup companies may become an oligopoly and charge unreasonable prices that add a lot of cost to the products.

Also, what happens if you order a product online from another company in a different country ? Does Norway still get to add tax for cleanup on these imported goods ? I would guess that this would be a powerful incentive for customers to skirt these regulations for lower prices.

This is a fair and valid view of looking at things. Though the trick is to not get too heavy handed. At what point does regulation become too stifling ?

The American way is to possibly put too few responsibilities on manufacturers. The European way seems to be to saddle them with just too many regulations -- possibly killing so much innovation.

One way to approach this would be to put more responsibilities on large established companies and less on smaller companies. But then the problem is that larger companies will want to arbitrage this somehow by indirectly "owning" these smaller companies with less environmental responsibilities.

This area is far more complex than we think it is.

Also what do we do about totally new materials that are thought to be benign when introduced but then are proved to have harmful effects many years later. Does the company that introduced them now have huge open ended costs and now go bankrupt ?

The solution is as always in the middle ground. Society as a whole bears some cost of cleanup (a kind of insurance policy for all companies) and companies bear some of costs.

You should not discuss politics with friends.

You should however discuss politics with close friends -- they probably got close to you because you both share a worldview or they like hearing your worldview (even if it differs from yours).

Closeness means more sharing. That always comes with risks and rewards.

Probably not at the moment.

_Software Counter mode_ rr requires robust support for debuginfod in your Linux distribution.

NixOS to my knowledge does not provide debuginfo via a dedicated debuginfod server for all packages so this will make things unreliable for debugging via _Software Counters mode_ rr for packages for which debuginfo is not available via debuginfod.

I recently learned about https://github.com/symphorien/nixseparatedebuginfod for nix (note: I haven't used this so I don't know how reliable or good this is). Anyways, this project requires setting `separateDebuginfo = true;` for the derivation for the debuginfo to be available via debuginfod. This is an opt in approach but we need pervasive support like it exists in Fedora (and some others).

It's very difficult for a broad based record/replay software like rr to exist for macOS in my opinion. macOS system interfaces are quite basic in terms of functionality compared to Linux and increasingly locked down.

rr uses many advanced features of Linux `ptrace`. Compare `man ptrace` on Linux with that on macOS for example and you will notice that Linux gives a lot of features to `ptrace` that macOS simply does not.

There are a large number of other features required for practical record and replay -- I dont think macOS simply provides them also.

It's probably possible to build _some_ record/replay system on macOS with constraints, restrictions, workarounds and compromises -- never say never as they say. But I don't think it can be as capable/generic as rr on Linux.

rr and Gdb are very DWARF debugging focussed. As long as Haskell has only basic DWARF debugging support I wonder how much rr/gdb can do.

Though I do see a lot of promise in the future. rr can help make premature evaluation (many expressions are evaluated earlier than they might typically happen in a real Haskell program because a user may want to inspect a value) in the debugger not matter so much because that evaluation can be executed in a diversion session.

Thanks for the encouraging words! Please do try it out and report back if it worked well or not for you on the issue tracker.

With sufficient usage I think we can make a good case to get merged upstream. This patch introduces dynamic/static instrumentation for ticks counting which is quite different to how things have happened till now on rr. If there are many success stories a stronger case for upstream merge can be made. The rr maintainers are aware of this project but it is early days yet for an upstream merge PR attempt yet

Tough to say right now. Here is a long answer: https://github.com/sidkshatriya/rr.soft?tab=readme-ov-file#w...

Any gdb integration scripts for Python to get stack frames etc. should work fine in rr and Software Counters Mode rr.

`rr replay` and (the software counters equivalent) `rr replay -W` invokes gdb.

Yes, io_uring is still not supported due to fundamental issues in the overall rr architecture which my modification does not resolve. My modification only addresses the HW counter requirement of upstream rr and the other core aspects of rr remain the same.

Normal system calls transition to kernel space and return back from kernel space. They will change your program's memory/process state as soon as they complete. This gives rr an easy boundary when it "can do its thing" to record memory/process state changes or insert results (during replay).

When does an io_uring request/response complete ? That's difficult to say. The kernel/userspace when using io_uring communicate with each other by checking a queue head or tail with memory accesses to see if something got added/removed from request/response ring buffer.

Think of io_uring and userspace cooperating via memory. (Yes, sometimes "proper" traditional ring crossing system calls are made but what makes io_uring so fast is communicating via memory and not via system calls most of the time). Anyways all this makes it difficult for rr to intervene on the boundary between kernel and userspace because this boundary is elusive when it comes to io_uring. The memory writes cannot be caught by ptrace ! This explanation is simplified of course.

There are some plans to deal with io_uring by rr maintainers https://github.com/rr-debugger/rr/issues/2613

rr has always worked with Python in the sense that it can record and replay Python programs.

However, when you try to debug the program you can only debug the C code the Python interpreter is written in.

I suppose you want to be able to debug the Python code itself. Here is something that could do this https://pypy.org/posts/2016/07/reverse-debugging-for-python-... . I don't think the project is active nowadays though. Also I haven't used it so can't say whether it is good or not.

It should be possible to built a Python reverse debugger on top of rr. I know this should be possible because I built something for PHP https://github.com/sidkshatriya/dontbug .

There are other fancy (and possibly better) things that are possible -- instead of building a Python debugger atop rr you can record the full trace of the Python program and then for e.g. store the values of important variables at each executed line of the Python program in a database. This would again use rr as the record/replay substrate but with a slightly different approach. This is an area which I've done some work internally but nothing public released yet :-) !

I've not used Pernosco but I am generally aware of it. My guess is that Pernosco would need to be technically modified to support recordings that have "soft ticks" i.e. use Software Counters.

I don't see any compelling reason why this should _not_ be possible from a broad level technical point of view. Pernosco engineers of course would be able to give a more authoritative reply.

Good suggestion. Added to the TL;DR section of the blog post !

Author here. Yes there are some tradeoffs:

- Performance: slower than using rr with HW counters. Both dynamic and static instrumentation employed by _Software Counter mode_ rr slow things down

- Potential fragility: Dynamic and static instrumentation can often make record/replay a bit more fragile

- Currently only x86-64 support has been publicly released. I have aarch64 support working reasonably well internally and it allows me for instance run to rr in a Linux VM running on macOS ! I have yet to figure out my strategy for the aarch64 release so watch https://github.com/sidkshatriya/rr.soft for any updates.

- Currently can run only on a few recent Linux distributions (e.g. Fedora 40/41, Debian Unstable, Ubuntu 24.10) because it relies on robust debuginfod support that is not widespread yet. See https://github.com/sidkshatriya/rr.soft/wiki#how-does-softwa... for why debuginfod is required. The debuginfod requirement may be relaxed in the future with more work

Regardless of the tradeoffs this allows rr to be used in many more situations i.e. wherever HW Performance counter access is not possible/not reliable/broken.

I would love it if more people tried this out and let me know if things worked out well for them (or not) with their programs.

What happened to Swift ? I used to hear so much about it -- now not so much. Maybe I'm in my echo chamber ?

_Software Counters mode_ `rr` is a reverse debugger built upon the awesome `rr` debugger.

It relaxes one major limitation with the `rr` debugger - it does NOT require access to CPU HW performance counters.

This allows the debugger to run in containers, cloud VMs etc where access to CPU performance counters is usually disabled or not available.

In case you're wondering: reverse debugging is a powerful approach to locating bugs in your program using record/replay. Please see the blog post (https://github.com/sidkshatriya/me/blob/master/008-rr-everyw...) for a gentle exposition of the concept behind record/replay.

To understand a complex system, sometimes it better to understand a (simpler) model system. Sometimes an older version of the same system is that good model system. This is not true always but a good rule of thumb.

In the many years waiting for the 3.0.0 release have other OSS painting programs matured that have more features overall ? Is this obsolete on arrival ?

You can choose unsafe rust which has many more optimizations and is much faster than safe rust. Both are legitimate dialects of the language. Should you not feel confident with a library that is too “unsafe” you can use another crate. The rust ecosystem is quite big by now.

Personally I would still use unsafe safe rust than raw C which has more edge cases. Also when I’m not on the critical path I can always use safe rust.

Thanks for your answer ! So in the 32k global layer, every token attends to each of the other 32k tokens ?

[Edit: You answered the question when you said that individual attention layers are always dense.]