Hacker News Reader

Comment by Lyngbakr | Hacker News Reader

I've worked in data for a while now as an Data Engineer, Data Scientist, and Director and working with experienced software engineers has highlighted to me that most of the data stack is fluff. All the layers/tools that are heaped upon one another just lead to complexity and dependence on paid services. I'm not advocating for reinventing the wheel, but rather that a bespoke solutions seem to be worth the cost. In short, I'd prefer to spend the budget on experienced developers who can build maintainable systems than on a plethora of MDS tools. YMMV, though.

View on HN

There is no "the" data stack.

Some organizations have complicated orchestration requirements, some don't. Some organizations have large amounts of heterogeneous data, some don't. Some organizations need to run large distributed machine learning tasks, some don't.

If you adopt tools that don't actually solve real problems that your team/org faces, then of course you'll end up with a bunch of extraneous fluff.

Your comment has the same flavor as the frequent HN meme of "I just use Vi and Tmux and a BSD server for my business and anything else is a sign of organizational failure and individual weakness".

All of these tools exist for a reason. Where you get into trouble is losing focus on solving your real practical problems.

> Where you get into trouble is losing focus on solving your real practical problems.

Absolutely agree. As I say YMMV, but my experience is that often people have adopted tools because "that's what data teams use" rather than because it's the best tool for the job. Often, we could — and should — build an in-house solution that fits perfectly rather than buying (every month) an imperfect solution.

> Some organizations need to run large distributed machine learning tasks, some don't.

I think almost no organizations need to run "large distributed machine learning tasks". Many just think they do, because it's the buzzword of the moment.

Also true, but I think maybe that's a great example of what I and OP are saying here. Do you really need it? Or do you just think you might need it one day because people are talking about it on Medium, or because you had a nice chat with some sales rep? Or do you just want it on your resume?

Facebook's ad prediction models weren't distributed until maybe 2014 or 2015. Just buying bigger boxes can get you a loooooinnnng way.

> All the layers/tools that are heaped upon one another just lead to complexity and dependence on paid services. I'm not advocating for reinventing the wheel, but rather that a bespoke solutions seem to be worth the cost.

Which third party products do you use in your stack, then?

Do you only use basic cloud building blocks (VMs, EBS, S3), or F/OSS components like Spark and Trino, or are higher level things like Tableau/Looker also part of the solution?

If you wouldn’t mind: in your experience how often does the problem seem to stem from “we might need that later…” or “we don’t have a concrete set of questions we want to answer?”

I have considerably less experience but so far I have experienced a complicated data science stack being used as a way to handle a ton of data without clear questions and desires on how to act on possible answers. As if they’re searching a haystack for… something?

IME, it has usually stemmed from "that's how everyone else is doing it, so we should too" without considering specific use/business cases. Senior management were simply not willing to deviate from the norm.

And also: I need a job. My job/workplace has these tools. They are usable for my job and if I were to question the tools that would come on top of my workload, so I use them even if I know that cheaper tools were used at my last job.

Do I care if they pay my salary and the tools are usable?

That's perfectly reasonable, but it sounds like you're saying the tools work for your use case. So long as the executive/management level is evaluating the cost-benefit of other tools, and is receptive to voices from ICs, then there's no problem to solve.

It's just not what the GP is describing. In some orgs you end up with these expensive, standard-ish "suites" that are way too generalized, don't solve business needs as efficiently as other tools, are painful for ICs to use to meet requirements, and it seems like no one can really explain why these tools are being used over something else other than rumors (the previous CTO came from this vendor, we got a sweetheart deal 10 billing cycles ago, etc.).

So really this an issue of ownership; someone needs to be accountable for infrastructure decisions and outcomes, and it's way healthier for an org if ICs are aligned with management on the same. It sounds like you're not in that sort of org, so good for you.

But do you care if they pay your salary and the tools are barely usable?

You are describing exactly how the MDS works: buy what you want and build what want.

Reply on news.ycombinator.com