Worst Data Engineering Mistake you've seen? | Hacker News Reader

Worst Data Engineering Mistake you've seen?

old.reddit.com

28 pointshermitcrab24 comments

Easily my favourite in the thread:

>...i once had to maintain a codebase that dealt with a variety of data in 2d arrays. the previous author (over a decade prior) wasn't aware that you could simply myArray[x][y] to do lookups, instead they wrote nested loops for all indexes in each array, and would break the loop when the index matched the one they were looking for. they also didn't use helper functions, so this happened inline throughout the app

To be fair, there may be very good data locality reasons to prefer myArray[w*y+x] over myArray[x][y] if you're trying to construct a rectangular 2d array. The two forms are typically not equivalent. One is an array of arrays (i.e. double indirection), and the other is not.

The loops I don't have an explanation for though.

Good point! The loops, however, don't inspire the sort of confidence that would lead me to personally believe this person was concerned about data locality. I might be wrong though hehe

I think that this is a good reminder, however, that there are few "absolutes" when it comes to programming and that there are always edge cases where some things that look like "code smells" or even look plain wrong have their place.

If you design your abstractions correctly, you can get the best of both worlds.

You can create a simple Matrix type which stores single array, with a subscript operator that gives a 2D facade. It can let you call it with x/y pairs, but just lower that to an index into your single contiguous buffer.

Yes, depending on the language.

Someone found out they could write a stored procedure that produces XML.

Then they found that you could nest them in one larger query and each could independently return its own node in the XML, cool right?

Then they found out you could correlate the queries, making it oh so easy to extend and build out!

Then they built an entire insurance business on the gigabytes of XML they crapped out every day, which based on their business processes needed to get sent out once a day.

The process took more time, but it was ok, they had plenty of hours to spare in the day.

People kept adding though, feature requests kept coming in, and tech debt came piling in.

A few years later, they called me because their process had started failing, it was taking over 24 hours on the biggest EC2 instance they could afford, and of course this was a single instance commercial database.

I took a look and it was the worst pile of nested SQL I had ever seen, over 300k LOC of pure XML/SQL hell all nested together, the query plan broke the client - over 100MB.

I told them there was nothing I could do, they should _start over_.

Did they go bust before they managed to rewrite it?

They fired the CTO and managed to rewrite it, still going strong as far as I know.

I wasn't there there, but I remember when someone at gitlab deleted the gitlab.com production database.

The team at the time setup a public Google doc with all the info and a public video stream on YouTube with the devs taking about what to do/what they are doing.

I've never seen anything like that before or after, although I bet lots of prod db get deleted every day:p

https://www.youtube.com/watch?v=tLdRBsuvVKc

This video goes through the timeline of the main points

> I've never seen anything like that before or after, although I bet lots of prod db get deleted every day:p

Absolutely. I have at least three stories from friends who witnessed these sort of events first hand, and so many of posts in the genre online affirm that its a regular occurence.

I've yet to hear of a case where there weren't backups for prod, but I'm sure those exist to.

This happened in the early days of GitHub:

https://github.blog/2010-11-15-today-s-outage/

These sloppy setups are common even today, where it's fairly easy to allow for a simple config error or a wrong environment variable (env=PROD) to wipe out production from a test suite. These safeguards need to be in place in the first days of your codebase.

can you an example of those kinds of safeguards? asking for a friend :)

Not letting devs access production :)

I think one thing is to make sure that when your test suite executes - you should force environment to "TEST" programmatically, before any bootstrapping code even runs.

So even if you allow for connection to production (sometimes that's allowed for debugging hard-to-reproduce issues), you want to remove any room for human error.

And your code should obviously NEVER be able to connect to anything other than the local test DB when your environment is set to "TEST".

Oh I was under the impression that it was a forgotten reverse ssh tunnel, that proxied localhost:some_port to the production setup.

Or maybe that was another story hehe.

The common thread seems to be unexpected massive bills from running jobs on third party servers.

Yeah I think I'd rather deal with a server or two of my own getting overloaded than having some kind of huge unexpected bill.

But everyone wants to work on geodiversified autoscaling superclusters. It starts with watching a presentation on AWS or Azure on YouTube, or even innocent Medium article.

Indeed. But a half way decent PC running halfway efficient software can generally easily process datasets with millions of rows. It is only when you get into billions of rows that you really need to start worrying about scaling.

Realistically, it starts as a byproduct of the turn toward Resume-Driven Development (RDD).

The bills aren't the only externalities from this design decision. Worked in an office a few years ago, brand new state of the art everything. Even the light switches were so damn modern and 201Xs that the lights were controlled by a server somewhere.

Funny thing happened when you pressed both the on and off button at the same time[1], based on empirical observatgions, both the turn-on-lamp and turn-off-lamp processes would spawn, and they'd both race against each other, almost turning the light on, almost turning the light off. So we got flickering disco lighting until someone called a technician to reboot the server. Good times.

[1] of course on and off they were discrete buttons, rocker swiches are so outdated.

We had to come up with some tasks for interns to attempt over three week period. We had some spaghetti data preprocessing written in java and parameterized by hand modified xml, orchestrated on a map reduce job. Every year the job needed to be extended to handle some new fields and it was an inevitable clusterfuck that required a full month to figure out. I personally stayed well away, it was clearly somoene's vacation work. But it was suggested that these turnaround times were not good, the individual jobs took hours to run and nobody was setup to debug locally. I estimated that given the data size (100k rows LOL) joined to dozens of rows (LOL!) should take no more than few minutes in python (both to write the code and run it (POWER OF PYTHON!) and there wasn't a really good reason why we should need hundred nodes and complicated data orchestration. It was obviously decided to rewrite into spark jobs because this was "BIG DATA" and we needed scale (100K rows LOL never going to increase). Somehow the rewrite never finished because the jobs were running out of memory (8GB LOL!). At the end I asked why didn't they just write the join in python? I was told it wouldn't scale and that this was BIG DATA. I personally have come to the conclusion that vast segments of IT are working on kickbacks and nepotism and maybe 9/10 of workers should be fired, but thats just the shitty places I've worked so probably that philosophy doesn't SCALE!!!!!

At a prior company, there's a 3 digit year floating around when it shouldn't be a 3 digit year. no one knows why and or who owns it.

Reply on news.ycombinator.com