Hacker News Reader

Comment by dmatthewson | Hacker News Reader

From the article: "Data engineers are the janitors who keep your data clean and flowing."

Hm, I wonder why he's having problems hiring janitors.

Bizarrely, I remember a recent HN discussion where a poster was arguing that any software developer who is not working in machine learning is like a plumber.

I guess this means that the entire profession consists of janitors and plumbers.

In the minds of middle management, I think this is precisely correct.

and as pointed out so much, is entirely why nobody wants to work for them. Respect these very bright people znd you have a starting negotiation position.

I recently had a plumber do some work on a >100-year-old apartment. I was lucky: he's a very good plumber.

The job didn't involve too many "pipelines" but the knowledge and creativity required to make them work was well above what I see from most software developers.

"Plumber" is not the put-down that poster thought it was.

Considering that plumbers and janitors have likely, in the entire history of human civilization, done more for health and longevity than doctors and scientists...I'm kind of ok with this analogy.

Doctors, maybe, but it was the scientists who told them about the germ theory of disease, for instance.

I've read, but not confirmed for myself, that in the US the biggest gains in health came in the post-Civil War period, when "plumbers and janitors" made the difference. Of course, that's really starting with, after the science, the civil engineers who designed the public works systems that supplied clean water and took away sewage, and let's not forget that politicians and like who found it worthwhile to buy votes that way (now, they take our infrastructure for granted and buy votes more directly...).

Confirmation: https://ello.co/dredmorbius/post/MsQfdPAn_0XUdZUoReBfbg

Thanks! There was a delay after the Civil War as you'd expect from all the chaos and disruption that caused (e.g. MIT got its charter before the outbreak, but wasn't able to start up until after), but it's pretty clear, and gets really dramatic the further you go forward.

I've been studying the period (mostly the Industrial Revolution and onward, though the accelleration of the late 19th / early 20th century is staggering), and it's pretty phenomenal.

There was a lot going on. Germ theory, of course, was part of it. But public health measures, especially sewerage systems, clean drinking water, and municipal waste removal, were all massive contributors. Note that the decline in mortality occurs well in advance of antibiotics and even most vaccinations.

For all the recent debate on vaccinations, it's interesting to note that the peak period of their impace (roughly 1930 - 1960) saw relatively little reduction in mortality, though there was a large decrease in disease incidence. It turns out that with septic control, antibiotics, food quality, and nutrition, many viral diseases weren't killers, but did present quality-of-life issues. And yes, often quite severe -- polio was no joke, and I know people who've suffered lameness from it myself. Measles and smallpox are similarly scarring and have long-term impacts.

But the major impacts of virtually all medicine are front-loaded to the period before 1950, with much the gains since attributable to either greater access (especially for the disadvantaged) and removal of environmental agonists (lead, tobacco, alcohol, asbestos, miscellaneous poisons, safety hazards).

Sure, that's true of recent (< 200 yr ago) history, but plumbing's contribution to health and longevity goes all the way back to ancient Rome. (Somewhat ironically, plumbing, from the Latin word for lead, "plumbus", may have also contributed indirectly to Rome's eventual decline.)

Janitors? They are certainly more than janitors! More like plumbers... getting your data safely from point a to point b without plugging things up while passing through [process] boundary's. How much does a plumber cost? $140 / hr? Sounds about right.

Data engineers are the janitors who keep your data clean and flowing.

In a boldface font, no less. The cockiness behind that language is really quite astounding.

It's really true, though. It's brutal, ugly work with no hope of an end.

Edit: Favorite paper on the topic: http://research.google.com/pubs/pub43146.html

So is the work that doctors, lawyers, and other highly-skilled people do, by and large. Everyone knows that day-to-day aspects of these jobs are hardly glamorous (or even cerebral), the vast majority of the time. Yet somehow we manage to accord these people with their due degree of respect, and wouldn't think of referring to them as "janitors".

I don't see what's so bad about janitors, though. They do very thankless jobs for not much money, whereas doctors and lawyers and other high-skilled individuals are often well renumerated or offered certain social prestige that your post shows is quite lacking when a humble janitor is considered.

A perfectly valid point. But getting back to the original article -- it pretty much takes a SV alpha-nerd (or aspiring CEO seeking to cater to them) to come up with language like that.

There's nothing wrong with janitors, but the work they do can be done by any dumb monkey with almost no training. That's why the pay is so low for the job: anyone can do it, as long as they can lift trash cans and push a vacuum cleaner.

Plumbers are entirely different. They have to get their hands dirty working on some awful systems, but they actually have to know what they're doing, get specialized training, etc. Soldering a proper joint with copper pipes isn't that easy, and if you screw it up, it'll leak later and cause a lot of property damage. Knowing which pipes and fittings to use where is specialized knowledge. It's not something you can just grab someone off the street and train them to do in 30 minutes. Of course, plumbers also cost a lot too, and the ones who are self-employed (rather than their assistants) generally do pretty well financially.

Hey I'm the author of this blog post and the CEO of the company that did the benchmark report. That was a very poor choice of words on my part, and I appreciate you flagging it. I reworked the paragraph to remove the janitor comment and (hopefully) make it clearer.

You should also not use "janitor" as a disparaging term. That would be another good takeaway from all of this.

I agree that it's a bad idea to use "janitor" as a disparaging term, and that was very far away from my intention. If that was what you took away from reading it, then that's more evidence that I didn't do a great job with writing the original draft.

Here's the original paragraph for reference:

Data engineers are the janitors who keep your data clean and flowing. Insights are great, and you need them. But to deliver insights at scale, you need data infrastructure. That’s delivered by data engineering. It’s not as fun to talk about as D3 visualizations and business intelligence dashboards, but it’s every bit as important.

Reply on news.ycombinator.com