Last year we collected dev horror stories on the CircleCI blog for Halloween https://circleci.com/blog/dev-horror-stories/ and it was so much fun we want to put the call out again.
What's your worst? The scariest might end up on our blog roundup this year, feel free to make a note if you'd rather your story not be shared :)
Discovered race condition causing use of fallback crypto key. Taught tech support to decrypt with fallback key.
Looked through compiled machine code. Discovered race condition triggered by particular two-byte opcode sequence that happened to be readable ASCII and unique.
Taught tech support to search and replace that opcode sequence with Notepad++ in the DLL, changing a comma to a tilde, thus solving the problem once and for all.
The other horror story was at my current job. There was a data transformation that needed to happen to take data from one system and put it in another. Both databases happened to be on the same server so the person who designed this decided to use several SQL stored procedures to accomplish this task. This is where the fun began. First, if there was already a column named something, such as "BirthDate" and this individual wanted to use a column with the same name, instead of re-using the column that already exists and populating it with the correct data, he would create a column named "BirtthDate". Now that example is made up, but he would frequently purposely misspell words to make unique column names to avoid conflicts with columns in the same table that had the same name. He also would do calculations in his stored proc but then would never use the result, or he would set the variable holding the result to a static value lower in the SQL stored proc. It made things very interesting.
(Note the following is a criticism of the previous developer - not of Marmelade)
It was the most rinky-dink setup I’ve ever seen - unrepeatable build system that relied on binary files all over the file system (in .config folders, temporary directories, hard coded paths that only existed on devs laptop)
The texture compressor was able to fit all of the sprites into a sprite sheet with literally kb’s of texture memory left over, we couldn’t change a sprite because it would cause the very exact pattern of all of the other sprites to no longer fit together by the packer
Random crashes of the app we’re common, massive libraries would be included only to use a tiny math function within - to do a trivial calculation causing 100,000x slowdowns
Deployment involved manually building a bunch of object files, then copy + pasting them to other folders, then building more and moving them all around, then delete assetX from here, delete assetY from here, run packer manually, copy files here, run linker - none of this was documented
The code was a mess - major over abstractions 10 layers deep that could have been 2 or 3, each intermediate was instantiated seemingly randomly through the class hierarchy, all over the place, no consistency
There were lots of anti patterns throughout the code base - some god objects holding state they shouldn’t have, which led to lots of unreproducable race conditions. Memory leaking common, so steady memory growth until crash.
The dev was a total nut, huge rants over nothing
I laugh looking back at it now, but at the time it was painful
They pushed new code straight to the production server/DB. No safeguards.
They didn't implement any source control. All new code is emailed to the project manager and engineers filled in a form explaining changes.
Project management is done on a checklist. In Excel. Schedules are in spreadsheet tables. It was all managed in emails. People would have dozens of copies of the schedules and often pulled the wrong one in a rush.
Dsspite all this email traffic, the company emails had very low size limits. A week of work and the inboxes would be full. So people would regularly drop stuff.
Staff were not native English speakers. They regularly typoed things and were not consistent. On some days, the API would be "product" then "prodduk" then "products" (actual variable name changed for privacy). So, app developers had to change their code to work around this API.
I was in V&V at the time, not yet a developer (professionally). We had a maintenance task, an old system that was built some 15+ years before any of us [0] had started working for this company. Fine, it's normal. Pull up the various documents (requirements, specs, design, test plan). Test plan? Hah! It was only 50 or so pages. It was nowhere near comprehensive enough to verify the system behaved correctly. So we started expanding it. We set up our test stations and started flipping switches and developing new plans. Crap. Within a week we found a problem. Some background: These are safety systems. We made (for this particular project) the sensors that would be positioned in various places (think smoke or heat detectors) and the box that aggregated the results. We then sent the aggregate results to a control panel.
So I said we took input from multiple sensors. The test plan had each sensor tested individually. Trigger a smoke alarm on A, see it gets reported. Trigger a smoke alarm on B, see it gets reported. We decided to trigger multiple simultaneous alarms. I mean, it wasn't an explicit requirement but it was certainly a reasonable derived requirement.
Odd. Only the first alarm shows up. Trigger A, see A. Trigger B while A is still on, still only see A. Alright, try the other way. Shit. Same behavior, but you only see B.
Ok, fine. What happens if we clear the alarm on A and then trigger B. Oh god. B never gets reported.
This is the primary responsibility for these systems. If they aren't doing this, they have zero value. Essentially, you would only be able to deal with one emergency. And until a system reset, you would never be able to see any others.
Another guy and I raised quite a stink but we were both too young to want to risk our careers by escalating this too high. We left it about 3 levels above us (the systems engineer). Within a year he and I were both gone, that had left a bad taste in my mouth and he had been needing a change for a while anyways.
Being fairly vague because the quality of new software was actually pretty high. And from my former colleagues it's only improved since I left. But those old systems...
[0] 100% turn over the year before I got there, I did not know that going in.