Hacker News Reader

Comment by kafkaesq | Hacker News Reader

There is nothing more soul sucking than writing, maintaining, modifying, and supporting ETL to produce data that you yourself never get to use or consume.

I'm not sure I get why writing ETL code for data you'll never consume is any more soul-sucking than, say, refactoring JS code for a website you couldn't begin to care about (and which will never be properly re-designed anyway); or even doing "thinker"-level work but for an industry you couldn't begin to care about (advertising), etc.

In other words, what most developers of whatever technical stripe do for a living.

View on HN

And I also fundamentally disagree with the notion that moving a large amount of realtime data reliably and with accuracy, monitored and consistent with relatively little failure is not an interesting engineering challenge in itself. I find that for all the talk about data-driven organizations, most don't use a tenth of what is available but that when the tenth is needed, it's hugely satisfying to be able to provide it.

> And I also fundamentally disagree with the notion that [ETL work] is not an interesting engineering challenge in itself.

A lot of people think that certain DBA/ETL/BI/similar work is boring and simlpy don't want to do it and so don't learn to do it well. Which is fine by me: it means those of us who can do it well can get paid good money when someone needs it.

The only problem with this theory in practise is that many also think such work is easy and free of complications; so they baulk at paying for people genuinely can do it well, get people less experienced who say they can do it well but do it badly, and judge the rest of us by that standard and assume database people are thick and can't do easy jobs properly...

I'm with you. I hear people reflexively dissing ETL (and other aspects of front-line data engineering) all the time, but I've come to suspect they don't really know these systems are actually about.

Reply on news.ycombinator.com