Deep Bug
marginalia.nu
marginalia.nu
Turned out it was a java jvm bug which was triggered when the jvm decided to recompile that code part since it was used more frequently.
Try running your code with the -server flag and see if that makes a difference.
Another debugging technique we use for heisenbugs is to see if `rr` [1] can reproduce it. If it can then that's great as it allows you to go back in time to debug what may have caused the bug. But `rr` is often not great for concurrency bugs since it emulates a single-core machine. Though debugging a VM is generally a nightmare. What we desperately need is a debugger that can debug both the VM and the language running on top of it. Usually it's one or the other.
> In general I’d argue you haven’t fixed a bug unless you understand why it happened and why your fix worked, which makes this frustrating, since every indication is that the bug exists within proprietary code that is out of my reach.
Were you using Oracle GraalVM? GraalVM community edition is open source, so maybe it's worth checking if it is reproducible in that.
That was almost 15 years ago. But to this day, every now and then I feel a little guilty about getting that degree because I haven’t truly solved that bug.
We (mainly a collegue of mine) eventually traced it down to a smaller part of the code until we noticed a subtle bug.
Multi reader + multi writer and a atomic operation that miscounted the number of readers.
Took us a couple of weeks to track down if I remember right.
(also, not entirely sure what you meant. There isn't a canonical C compiler, or toolset, and the language itself really doesn't do anything to prevent you from shooting yourself in the foot)
Well, that or JNI/FFI in some C code I guess.
Also, Kotlin has unsigned integers. So you could use that for the parts of your code that benefit from it.
Pointer arithmetic, I imagine. Elementary math after all.
With unsigned integers, type aliases and pointer types this would have been a lot easier to keep track of. The limited range of primitive integers also means static analysis is basically impossible. Assing a pointer to an integer or vice versa and that's a code smell. Assign an integer to an integer and the compiler is none the wiser.
You say that the lack of unsigned types in Java is in part responsible for some of your wasted time. But C has notoriously bad integer types and promotion rules, especially for interactions between signed and unsigned! If you had found this same bug in C code (a compiler bug), your first instinct as a C programmer might have been "it's probably some signed/unsigned interaction issue". And you'd also have wasted time trying to find a non-existent issue in your code.
What I'm building is what Lucene does (i.e. document indexing) and then the rest of the search engine as well including crawling and serving traffic.
I previously worked at a search engine which did its index building and querying in C, with its higher-level stuff (web-apps, scheduling, tooling, etc.) in Java.
Later when I built my own version, I started with C for the low-level and Haskell for the high-level. I made a few iterations in C, but eventually rewrote it in Rust, and I was pretty happy with that choice.
I was more familiar with C, and it was a really good fit for what I was writing. Terms and Documents just become termIds and docIds (numbers) sitting in indexes (arrays). Memory-mapping is a really comfortable way to do things: files are just arrays; let the OS sort out when to page things in and out of disk.
But where C fell down for me was in the changing the code. The meaning of the data&code were lost in nested for-loops and void-function-pointers, etc. Rust gave me a better shot at both writing and rewriting the code.
Java for the low-level was a non-starter for me for a few reasons, but the biggest two were startup time and difficulty of the mmap api (31-bits of address-space for a file? c'mon!).
Neither of these are issues anymore though. Especially not with graal's native images.
The program had a large number of hour without incident, was thoroughly tested with unit test, and had 0 valgrind warning. I'd been intermittently tracking the problem for months, and the fact that it was an hardware issue was a huge relief, as I really started to loose footing on the necessary minimum of confidence in predictability you need to write software.
https://github.com/MarginaliaSearch/MarginaliaSearch/commit/...
Directory: https://github.com/MarginaliaSearch/MarginaliaSearch/tree/ma...
Looking at the code, it seems it is copying val to counts. It does analogous arithmetic on val an on counts, and since they both should contain the same data, it's surprising that the result of the arithmetic does not match.
Can you reorder the code to see if the error is reproducible? Do it only for a diagnostic run, and don't change the actual code. For example, change the first loop into two loops:
for (int i = 0; i < length; i++) {
counts[i] = val[i];
}
long offset = 0;
for (int i = 0; i < length; i++) {
offset += val[i];
}
Another variant to try is to add another running counter that mimics the size arithmetic, but in the first loop, and compare the three values to see if you get 2 out of three to match:
long offset = 0;
long size_debug = 0;
for (int i = 0; i < length; i++) {
counts[i] = val[i];
offset += val[i];
size_debug += counts[i];
}
If I had to place a long bet, my guess is that somewhere in the first loop, after assigning counts[i] = val[i]; reading from val[i]; does not return the same value. You said it's in the same thread, but we need to admin this is a deep bug, so all bets are off.
This guy is extremely self-aware
It's possible that a write to the counts array is misdirected so it doesn't occur.
Is the sum of counts always less than the sum of vals?
On second thought, if counts is mmapped, GC might not be in play. But a similar bug might occur due to state being lost during an On-stack-replacement of one of the loops.
I'm not familiar with deep jvm stuff, but is there a way to ask the jvm for a compilation log to see if JIT OSR is happening?
https://lobste.rs/s/qjrdss/computers_can_be_understood#c_hqu...
For any fellow sufferers, the needle in the haystack is "skb rides the rocket." Same bug was also hit by Brendan Gregg around the same time.
The systems employ concurrency at multiple levels, where the system design lacks a unified paradigm for establishing its correctness.
So concurrency "errors" should be expected.
I always start wondering why such bugs aren't more common. But then I realize a dilemma.
For concurrency errors, you can know an error occurs but have no way to work backwards to the specific conditions of its cause (incorrect design). All you can do is perturb the system into not exhibiting the error.
Lacking a formal way to establish correctness, we are left to an engineering of attrition, in which the author is engaged.
If the failure becomes common, system parameters will be adjusted to perturb behavior back into obscurity with a black art called "debugging" that approximates correctness.
The contravening property of the system is the underlying logic runs so fast with respect to human attention that failure modes are pressed into high likelihood of being observed and therefore "corrected" (approximately). Bugs which are common enough to merit attention are perturbed out of "existence" by "version charges."
This seems to imply that such failures should be both expected as unavoidable and rare according to limits of human attention according to engineering by attrition.
Welcome to the author's world. We are all facing this sunk cost.
In the bug picture (haha) we simply abide systems that "work" according to a distribution of our tolerance for the nuisance of their inevitable failure in modes that are too rare to be "corrected."
As the author explains he's working with newer versions of a machine. Huzzah!
Meanwhile, the ubiquity of deployments is scaling towards infinity, implying that a small clique of individual humans can expect to be driven mad by faults that appear demonic while the hoard of humanity lumbers on enduring the costs of "good enough" design.
Except maybe for the contingency that the strategic nuclear deterrent is be placed under control of an AI.
Luckily for the individual human there is death.
Unluckily for humanity, someone is likely trying to place the strategic nuclear deterrent under control of an AI
https://www.youtube.com/watch?v=47lcjbyqF_k&t=4066s
1. Measuring Deep Metastability and Its Effect on Synchronizer Performance -
2. He Who Hesitates is Lost: Decisions and free will in men and machines - http://async.org.uk/David.Kinniment/DJKinniment-He-Who-Hesit...
"Buridan's ass is an illustration of a paradox in philosophy in the conception of free will. It refers to a hypothetical situation wherein an ass (donkey) that is equally hungry and thirsty is placed precisely midway between a stack of hay and a pail of water. Since the paradox assumes the donkey will always go to whichever is closer, it dies of both hunger and thirst since it cannot make any rational decision between the hay and water.
A common variant of the paradox substitutes the hay and water for two identical piles of hay; the ass, unable to choose between the two, dies of hunger.
The paradox is named after the 14th-century French philosopher Jean Buridan, whose philosophy of moral determinism it satirizes.
Although the illustration is named after Buridan, philosophers have discussed the concept before him, notably Aristotle, who put forward the example of a man equally hungry and thirsty, and Al-Ghazali, who used a man faced with the choice of equally good dates.
A version of this situation appears as metastability in digital electronics, when a circuit must decide between two states based on an input that is in itself undefined (neither zero nor one).
Metastability becomes a problem if the circuit spends more time than it should in this "undecided" state, which is usually set by the speed of the clock the system is using. Interesting"
I dislike the figure of speech "A lot to unpack," but the Catt dialog on "the glitch" and electric current has a lot going on and I found it well worth the listen.
In interview 2/2 on electric current the "Demystifying Science" pair strangely fall into the orthodoxy of wanting to block and control dialog in order to manage the apparent controversy illuminated by Catt's perspective. They fight back even as they clearly express curiosity about, and sympathy with\ Catt's views and laments. I came away noting there's a profound natural block in the discourse of science towards a commonsense which is obviously insufficient to accommodate the world as we now find it.
It so happens that N. Chomsky gives a lucid presentation on the history of science that pertains directly to Ivor Catt's laments on blocking of science by an engineering orthodoxy.
Chomsky - Mind, Language and the Limits of Thought https://www.youtube.com/watch?v=2NffP06zqkw
The talk begins with a proposal for 3 problems — "Plato's, Orwell's and Descartes'" — supported by a review of modern scientific thought, then segues into a supporting illustration of modern effects of these problems in the orthodoxy of NATO policy in Serbia. It's the tightest package of criticism of contemporary thought I've come across and will not be a waste of time for viewers at any level of interest and familiarity with the history of thought.
There are two versions of this presentation on Youtube made to different audiences on different dates. I prefer the clarity of this one.