Comment by mritchie712 | Hacker News Reader

Comment by mritchie712 | Hacker News Reader

(disclaimer: I'm a founder in this space)

> the project is either so incomplete or so lacking in a central design that the best thing to do is replace the old system

I put a lot of blame here on "the modern data stack". Hundreds[0] of point-solution data tools and very few of them achieve "business outcomes" on their own. You need to stitch together 5 of them to get dashboards, 7 of them to get real time analytics, etc.

We're going to see more products that achieve an outcome end-to-end. A lot of companies just want a few dashboards that give a 360 degree view of their data. They want all their data in one spot, an easy way to access it and don't want to spend fortune on it. That's what we're focused on at Definite[1].

We're built on the best open source data projects (e.g. DuckDB, Iceberge, Cube, etc.). If you decide to self host, you can use the same components, but it's generally cheaper to use us than manage all this stuff yourself.

0 - https://mattturck.com/landscape/mad2024.pdf

1 - https://www.definite.app/

2 - https://youtu.be/7FAJLc3k2Fo

The "you need 5-7 different tools glued together to solve anything" is the CORE problem of the "modern data stack". It also ties very closely with Resume Driven Development.

It leads to a lot of anti-patterns.

For example, the 5-7 different tools are constantly changing, so after hiring some proclaimed expert.. they end up re-inventing the wheel by choosing a new combination of tools than they've used in the past, hitting various unexpected issues as they go.

VERY rarely in this space do you see someone come in and go "I used these 5 tools in previous roles, they work great, and I'm going to build the best solution because I have done it before."

These guys always think they need to reinvent the wheel, and then end up wrecking the car with some combination of v0.1 untested FOSS, up&coming SaaS, and their own in-house DSL.

OMG I'm that guy -- 3 straight Vertica roles with $B annual revenue.

I did learn a lot from watching MDS people try to beat it (in the end I'm also looking for what should come next), but mostly it was confirming the article and RDD. What they didn't know about data warehousing they also didn't know about performance or price-performance or selecting tools or managing projects or vendors, so costs exploded.

These folks were hilarious because they kept insisting that Vertica is not "modern", while it beat the pants off them with basic columnstore stuff.

In my opinion there are many tools and products in this space and they all seem somewhat confused in their target audience (is it marketed to management, analysts producing reports or developers supporting the analysts?) The boundaries between these projects is often fuzzy and they are often complicated (does it include a scripting language?) When you are starting from close to scratch with an application and it's backing database and being asked to produce timely reports, I think these tools aren't the best place to start.

My process has been to talk to stakeholders and sketch out reports they find useful, preferably getting a set of data together that many people find useful. Running these reports against a read-only replica of production data is typically not a big lift. If dashboards are required, write this data out to another database to back the dashboard, probably on some set schedule. It hasn't been long but now we have the bones of an ETL process that is already returning value.

At that point I think these tools start to look more compelling. Now we have a handle on the source data, what it looks like and any places where we need to do something tricky to connect the dots to get our data out. We know what the reports and dashboards look like.

In short, we know what we need these tools to do and where they can help us.

One thing I often see in data people (particularly consultants) is that they are very focused on building reports and dashboards in my experience (manufacturing plant) Analytics should be the focus - if you get the analytical part right - make the data simple/easy to interrogate then the reporting can evolve naturally as a consequence.

Unfortunately I think the tools for a lot of this (exploratory data analysis) are somewhat lacking. I think we are starting to see new tools emerge - especially around Timeseries data that are promising but I don't think things are at the level where a non expert user can quickly glean insights from data in a frictionless way.

I disagree. Reports and dashboards provide decision makers with information that they require to make decisions. This is always step one.

Randomly exploring data for "insights" is why so many companies are turning against "Data Science"; it rarely bares fruit. This work should be focused on dialing in business processes.

I don't think we disagree (Report and dashboards are useful decision making tools), but in practice hiring external people to come in and build them causes more problems then it solves. In my experience processes on the ground can and will change and all of these pre-written reports and dashboards quickly become ossified and outdated.

If you take steps to make sure the data is properly stored and queryable stakeholders can maintain and develop their own reporting. Make the data understandable and accessible and reporting will follow. Human friendly schemas and such help a lot.

Agreed. A lot of my background is in banking / fintech. So much scrambling on "adhoc analytics" can be avoided if you have really well constructed, standard reports and dashboards.

The department I started my career at didn't have this and every exec request was a bespoke, adhoc request, tailored to answer that one question. I spent many late nights writing SQL and building excel files / powerpoints to answer a single question.

When I started at a fintech as the first data person, we built a few primary dashboards where you could drill down for detail. Those answered over 80% of the questions people asked.

Having lots of options isn't a bad thing. The tools you happen to have chosen for your product are not objectively "the best". The entire business model of SaaS is "it's cheaper for us to manage this than you".

I am also in this space. Personally, I like piecing together my tools, rather than using some black box. YMMV.

Definite - seems interesting.

But what is the Definite warehouse? Is it built on open standards?

Yes, our warehouse is built on DuckDB and Iceberg (https://iceberg.apache.org/). DuckDB is used as the query engine and storage or smaller or static data and Iceberg is used to store larger / more frequently updated data (e.g. CDC from Postgres).

I'm not a founder in this space like yourself, but I do a solid mix of consulting and building on the modern data stack for AI/ML and always appreciate a well constructed data stack.

Reply on news.ycombinator.com