Comment by staticautomatic | Hacker News Reader

Comment by staticautomatic | Hacker News Reader

staticautomatic

I'm a newly minted head of analytics who transitioned from a different domain, so I never had to muck my way through the MDS but attentively watched others from the sidelines over the last few years. Best I can tell, "the modern data stack" is just a marketing phrase invented by a cadre of vampire vendors. The lessons I learned watching others translated into a few simple requirements for our nascent "stack" that most importantly include transparent pricing I can reason about and divvy up, as many integrations as possible so I can minimize rolling my own, and a straightforward framework for ETL code. These three requirements plainly disqualify most of the MDS universe.

With the benefit of starting basically from scratch and not having to mess around with real-time analytics, it's pretty easy to ignore the MDS vendors. So far I've landed on BigQuery, AirByte, GitHub, BI Engine, Looker Studio, and Pandas 2.x or DuckDB for local stuff. I send as many things as possible straight to BQ, lock junior analysts out of gigantic tables, archive periodically to partitioned parquet files in cold storage, use mostly turnkey integrations, and ruthlessly prioritize custom ETL jobs. Putting GitHub in the mix isn't super ergonomic and we may be in the market for new tools once we cross the "big data" frontier, but that'll be a while from now. I'll probably never know or care what the MDS vendors think I'm missing.

You're on the right track with pricing. What I find ubiquitous about MDS people is a lack of old-school ideas like benchmarking and price-performance. Cloud salespeople talk endlessly about your company becoming data-driven until you bring up cost data.

Fellow head of data here. For a while, my team ran on MySQL, a shared file server we mapped as a local drive, a team server in the back closet for tasks too large for our laptops, a MySQL database, an SVN server, and a Wikimedia server for documentation. We're upgrading a few of those items, but it's amazing how cheaply you can run a machine learning team if you only have a few TB of structured data to analyze.

AirByte was definitely one of the MDS vendors.

They literally got a huge investment because their deck bragged about having a few thousand Github stars and a few thousand folks in a Slack channel.

I can't tell if this is envy or scorn

I don't think it's either.

Modern Data Stack is a set of characteristics - cloud based, SaaS, consumption based pricing, ELT bias, open source bias, component based etc.

AirByte is all of them. It's just a fact rather than a criticism.

I think this is fair and I don’t doubt that AirByte jumped on the MDS marketing bandwagon, but the fact that their pricing isn’t inscrutable sets them apart in my view.

Neither, just an interesting factoid that highlights just how zany the VC landscape was in the MDS space just a few years ago.

https://airbyte.com/blog/the-deck-we-used-to-raise-our-150m-...

Github stars used to be a signal of quality, and velocity of those stars indicated a winning product. But like all rating systems, they are for sale to the nearest Mechanical Turk or GenAI.

Airbyte is definitely one of the MDS vendors. Plus they have a ton of bugs because their only focus is having the most connectors on the market. A lot of them are broken or badly implemented.

I don't doubt the bugs. In fact, I expect them because so many of the connectors are community contributed and so I try to think of it as more like GitHub than a curated set of expertly-built connectors. That said, for the time being it's more important that I not break my other rules. The promise of a perfectly working connector (if such a thing exists) in exchange for unknowable pricing or a SDK I'm gonna have a hard time training people on is a tradeoff I feel I can't reasonably make right now, but I'm very open to the idea that my calculus may change.

Reply on news.ycombinator.com