Spending 3 months investigating a 7-year old bug and fixing it in 1 line of code
lemmy.world
lemmy.world
> Knowing very little about USB audio processing, but having cut my teeth in college on 8-bit 8051 processors, I knew what kind of functions tended to be slow. I did a Ctrl+F for “%” and found a 16-bit modulo right in the audio processing code.
That feeling of saving days of work because you remember a clue from previous experience is so good.
"You did awesome, but don't let it go to your head."
"Research shows" (I read long ago) that the happiest men are those take their wives' advice — that's certainly been true for my own N=1, for coming up on four decades now. I'd imagine we could replace "wives" with "spouses" and get comparable results.
Haven’t you just risked an infinite loop in your code?
But.. I met someone at the gym who's been struggling with an esoteric problem on an ancient piece of software for over a decade, and they approached me to ask if I could solve it. I said "maybe", sat on it for a few days, and then replicated the issue on my machine and solved their problem in about an hour. I asked for $50 and they gave me double, which was wildly more rewarding than being paid $100k to write react all year, not that that salary is on the table any longer.
Is the job market so bad right now? I‘m in a privileged position (and not in the US), so have no clue of the state of things.
But in business, you need to charge an honest / fair amount. (sure, sometimes that 1 hour bug fix had $100K of 'value', but we could argue about honest / fair).
You mention being an employee at $100K a year. Double that, gives you a contractor rate of $100 an hour. That is the floor of what you should be asking floor; as in the absolute lowest. Another $50 or $100 an hour is still fair and honest in todays economy.
Salary is driven by market conditions and nothing else. It is not an approximation of merit or even delivered value.
Proving it’s not a one time thing is what pushes you in the salary and seniority ranking.
And this is what saves the day.
Code is a liability.
Maximising the fraction of the product built by people who don't know what they're doing would however explain the emergent properties of modern software.
A good junior will write a hundred lines of code in a day. A good senior will delete a hundred because they realize the user dictated a solution instead of detailing their problem, asked them, and figured out that they can solve that problem by changing a config variable somewhere.
The most devastating value destruction (aside from the rare intern deleting the prod db) that I’ve seen consistently is with senior/rockstars who introduce new tech, takes credit, moves on. There’s a reason for the term resume driven development. Think about what a negative force multiplier can do for an org.
Maybe it varies per company.
But that’s the entire point you’re missing. The pay is not proportional to contribution or technical skill. It’s proportional to market forces and negotiation skill.
Negative force multipliers are easily remedied: just make sure you have an even number of them.
We had a guy who would argue about everything that knew the CTO so we had to tolerate him.
Then we hired a second one and they just argued with each other all the time and the rest of the team could finally make progress.
It was awesome.
In the former case, we basically had to demand management raise his salary to the low end of his market value, over the cause of six months, until they finally gave in. It was just so disgusting to us we couldn't let it go.
The reason comes down to a skill bias - it's a different skill set to navigate other people into getting yourself a good salary, versus navigating the ins and outs of coding. The skills don't overlap, so time spent on one detracts from another.
In the end he finally got the message we kept ramming in to his head, applied to work at a brand-name tech company, and instantly more than doubled his salary. He could've done so years earlier.
This stuff is the norm. I've been a manager having eyes on salaries while also having eyes on people's performance (although unfortunately not much of a lever on the former), and rest assured it is often a very jarring experience. Like "that person should be let go immediately / that person should job hop immediately".
So it's not true, then?
The GP claims this is universally true. All I need to do is post a counterexample, and I did. Yes, there are shitty companies that try to keep salaries as low as possible, not realizing that that will lose them their best people. Don't work for those!
This particular case sounds like someone got incorrectly hired as a junior. Maybe they didn't have enough "real world" corporate experience and that is why they weren't offered a senior position?
Junior/senior isn't necessarily about skill level; I'm sure many can find a senior with 1YOE ten times over. It's about trust both in technical and sociopolitical navigation through the job. That's only really gained with time and experience (and yes, isn't perfect. Hence, the aforementioned 1x10 senior. Still "trusted" more than a 1 year junior).
Understanding the business processes of your industry, how to solicit feedback from and interact with end users, how to explain things to management/sell on ideas.
If you put a junior dev in front of a panel of executives and ask them to explain requirements for a project odds are quite high they will info dump tech mumbo-jumbo. A senior should be able to explain risks, benefits, timelines, and impacted areas of the business in a manner that non technical people can easily grok.
I'm mid-level engineer. Honestly, several staff+ engineers may not be spitting tech mumbo-jumbo, but they do dump all other kind of BS. Political BS, "tactical tornados"[1]. May not necessarily mean they were good at engineering, but just good with people skills. Obviously, not everyone is like that, but I would say many are.
[1] https://news.ycombinator.com/item?id=33394287#:~:text=The%20....
Eg, for positive integers, x % 16 == x & 15. That should be trivially cheap.
Higher level change-the-algorithm aspirations haven't really been met by sufficiently smart compilers yet, with the possible exception of scalar evolution turning loops into direct calculation of the result. E.g. I don't know of any that would turn bubble sort into a more reasonable sort routine.
And more generally, with older architectures, integer division was much slower than integer multiplication, so compilers would generally transform this into a multiplication plus some shifts [0]. For context in that timeframe, MUL on Sandy Bridge introduces 3-4 cycles worth of latency (depending on the exact variant), compared to DIV introducing 20+ (per Agner Fog's excellent instruction tables [1]). So even computing x - y * (x / y) with the clever math to replace x/y would be much faster than just x%y. (It's somewhat closer today, but integer division is still fairly slow.)
[0] https://news.ycombinator.com/item?id=1131177 (the linked article 404s now, but it's archived: https://web.archive.org/web/20110222015211/https://ridiculou...)
[1] page 220 of https://www.agner.org/optimize/instruction_tables.pdf
That only works when y is constant. Otherwise, you need to work out what to replace x/y with… which ultimately takes longer than just using the DIV instruction.
Excellent point! That said, that was the case in this particular example.
> This 16-bit modulo was just a final check that the correct number of bytes or bits were being sent (expecting remainder zero), so the denominator was going to be the same every time.
(Libraries like libdivide allow you to memoize the magic numbers for frequently-used denominators, and if on x86 you have floating point operations with more precision than you need for integer division, you can potentially use the FPU instead of the ALU: https://lemire.me/blog/2017/11/16/fast-exact-integer-divisio...)
Did I win? Of course not, it’s hard for non-technical people to fully appreciate these things and any sort of larger infrastructure work, esp for developer productivity because it goes back to well how you going to measure that ROI.
Anyways, this was fun to read and brought back good engineering memories. I’d also like to say, as it brought back a bug I chased forever, fuck you channelfactory in c#.
When people bash gRPC today, they don't know of the horrors of the past.
That experience is actually what led me to switch over to Product among other things, I get it when people joke (half joke) about considering retirement rather than going through that again.
Instead, the actual source of problems was K8S and the huge amount of institutional knowledge it required that wasn't there. I still don't think K8S is that good, it's useful but it and containerized environments in general to this day have a lot of rough edges and poorly implemented design aspects - involved runtimes like .NET CLR and OpenJDK end up having to do special handling for them because reporting of core count and available memory is still scuffed while the storage is likely to be a network drive. The latter is not an issue in C# where pretty much all I/O code is non-blocking so there is no impact on application responsiveness, but it still violates many expectations. Aspects of easy horizontal scaling and focus on lean deployments are primarily more useful for worse languages with weaker runtimes that cannot scale as well within a single process.
I suppose, a silver lining to your situation on the other hand is that developers get to have a PO/PM with strong technical background which makes so many communication issues go away.
It's strange that we havent figured out how to trust technical leadership to assess these things for management.
In many ways, the answer is obvious: give technical leaders economic incentives for team productivity. They will then use their expertise to actually assess relevant teams.
I've rerun test locally -- no fail. I've changed the seed to one that was used in failing run -- nothing.
I add a loop to the code to repeat text hundred times -- still nothing. I run the test in bash loop hundred time -- 3 fails. So this already hints on some internal problems. I fixed every possible source of randomness and verified that all the inputs are identical between the runs -- still fails only once in a while. I started building MWE, but the function involved in reproduction is fairly complicated. I'm left with a hundreds lines of Jax code which fails in couple of percent of cases.
I look at the output of the compiler, and it is identical between failing and successful runs. So the problem is in the compiled code. The compiled code is ~1000 lines of HLO (not much better than assembly). Unfortunately HLO tooling is both unfamiliar to me and not well fit to this case (or at least I couldn't figure it out). So I start manually bisecting the code. I'm finally left with ~30 lines of HLO. It fails even less often (1% maybe), but at least it runs fast. It also seems to fail in exactly the same way (i.e., there is single incorrect output that I've between 3 fails). Now that's something maintainers can be hoped to look at.
It turned out that matrices with same content but different layout were deduplicated, leading to, in my case, transposed matrix being replaced by non-transposed one. The hash used for storage did take layout into account so the bug appeared only if two entries ended up in the same bucket (~3% of times). The fix was an obvious one liner [1].
[1] https://github.com/openxla/xla/commit/76e7353599d914546f9b30...
I even wrote an 8051 assembler in C, but found a good tiny-C compiler for it before it went into production.
You are not a programmer unless you’ve written key-debounce code :)
(OTOH, some of the worst programmers I’ve ever had the displeasure of working with were amazing low-level code hackers. In olden times, it seems like you were either good at that level of abstraction, or you were good at a much different [“higher”] level, seldom both.)
I've had to do that once, and I still consider it a blessing from Satan above that I both a) figured it out b) it worked every time (plus bonus C: explained the logic to better students in my class)
This product was so old in fact that nobody knew how to compile the source code. "
I think you mean "Management was so bad, nobody knew how to compile the source code".
There are plenty of systems out there that can and and plenty that cannot be reproduced from source. The biggest difference is the card taken to do so, not the age.
Find dusty Perl script forgotten for years. Still works
Not the first time that I hear that
(IIRC UI scrolled twice for every mouse movement + you couldn't select items in server browser with mouse wheel as it would skip every other one)
Ex: VSAB -> "I am attacking the enemy base!" (Voice, Self, Attacking, Base)
"I will let you know as soon as I have the fix"
If yamux's keepalive fails/times out, and you're calling Read on a demuxed stream, it blocks forever.
We spent over three months on it before finding a root cause. It was over two months before we could even understand how to measure it - we were seeing parts of the automated overnight test suite run taking longer, but every night it would be different tests that were slow. A key finding was that almost everything was slow on some boots of the device and fast on other boots of the device, and there was a reboot before each test was run. Doing some manual testing showed it being close to a 50% chance of a boot leading to slowness. Now what?
I eventually got frustrated and took the brute force / mindless approach... binary search over commits. Unfortunately, that wasn't easy because our build was 45-60 minutes, and then there was a heavily manual installation process that took 10-20 minutes, followed by several reboots to see if anything was slow. And there were several thousand commits since the last known good build (the previously shipped version of the device). The build/install/testing process was not easily automated, and we were not on git, otherwise using git-bisect would have been nice. Instead, I spent weeks doing the binary search manually.
That yielded the offending commit. The problem was that it was a massive commit (tens of thousands of lines of code) from a group in another part of the company. It was a snapshot of all of their development over the course of a couple of years. The commit message, and the authors, stated that the commit was a no-op with everything behind a disabled feature flag.
So now it was onto code level binary search. Keep deleting about half of the code in the commit, in this case by chunks that are intended to be inactive. After eventually deleting all the inactive code, there were still a few dozen lines of changes in a Linux subsystem that did window compositing. Those lines of code were all quite interdependent, so it was hard to delete much and keep things functional, so now on to walking through code. At least I could use my brain again!
Using the clue that the problem was happening about half the time and given that this code was in C, I started looking for uninitialized booleans. Sure enough, there was one called something like `enable_transparency`. Disabled code was setting it to `true`, but nothing was setting it to `false` when their system was disabled. Before their commit, there was no variable - `false` was being passed into the initializer call directly. Adding `= false` to the declaration was the fix.
So, well over a year of engineering hours spent to figure out the issue. The upside is that some people on the team didn't know how to proceed, so they spent their time speeding up random things that were slow. So the device ended up being noticeably faster when we were done. But it was pretty stressful as we were closing in on our launch date with little visibility into whether we'd figure it out or not.
This probably wasn't an option back then with your toolchain, but it's so reassuring to know modern compilers / ASAN are amazing at catching this class of bugs today.
These types of problems(undefined behavior and or uninitialized) are often hard(time consuming) to diagnose and fairly common.
Lots of places overlook simple static analysis or built in compile features.
Love to see it. That place needs more organic growth.
"Almost every bug turns out to be a 1 that should be 0, or a 0 that should be 1"
Keeping this in mind often keeps one focused on the detail of the underlying binary values and how they are being manipulated.
Anybody know what's the exact transformation here? I searched around and found this answer, but it doesn't work:
[0] https://stackoverflow.com/questions/2566010/fastest-way-to-c...
> I can still recall the cacophony of what amounted to an elephant on cocaine slamming on a keyboard for hours on end.