Hacker News Reader

Comment by 1a527dd5 | Hacker News Reader

Blimey, that is a lot of moving parts.

Our data team currently has something similar and its costs are astronomical.

On the other hand our internal platform metrics are fired at BigQuery [1] and then we used scheduled queries that run daily (looking at the -24 hours) that aggregate/export to parquet. And it's cheap as chips. From there it's just a flat file that is stored on GCS that can be pulled for analysis.

Do you have more thoughts on Preset/Superset? We looked at both (slightly leaning towards cloud hosted as we want to move away from on-prem) - but ended up going with Metabase.

[1] https://cloud.google.com/bigquery/docs/write-api

View on HN

I’m the author, but posting as a private individual here, these being just my options and all that… but I can shed some more light on why I did move us to Superset.

Preset is great, as are most of these tools’ hosted versions! Lots of great folks working on these.

But, tbh, as an infrastructure company this is somewhat the core business of ngrok - hosting another DB + K8s service is something that we have great tooling for and lots of expertise in the infra space. And using ngrok makes it even easier.

The whole dogfooding aspect is important too - if I don’t run an app in production with ngrok I have a hard time empathizing with customers who want to do the same. My previous job encouraged that too and I’ve always liked that.

Also, yes, lots of moving parts - but most of them are very reusable and they share a lot of code, infra, and logic/operations playbooks etc. Costs are manageable - Athena charges $5/TB scanned iirc, which tends to be the biggest factor.

Appreciate you taking the time to reply :)

I guess the underlying tone of cynicism in my tone speaks to the question that I didn't directly ask - how often do each of the components/moving parts fail and require manual intervention/fixing?

I often get pulled into complex distributed systems and the team responsible for that flow (data or not) often have no idea where to begin.

Edit* On the point of Athena, I desperate wanted to use it but provide BigQuery to be much better in every way you could think of. It's the black sheep in the company, as every other cloud thing we have is AWS. But honestly, nothing I've found in AWS circle comes close to BigQuery.

BigQuery + Metabase is such a powerful combination. Easy, affordable, effective.

I appreciate the time you took to write this all out (both the article and your response here). In particular, this line from the article resonated with my own experience over the last couple of decades:

> This particular setup—viewing DE as a very technical, general-purpose distributed system SWE discipline and making the people who know best what real-world scenarios they want to model—makes our setup work in practice.

The common analyst-to-DE path has some benefits for sure with respect to business-centric data modeling, but without the deep technical infrastructure investments and related support, the stack becomes a beast to deal with at scale (or just ends up being a massive cost on the balance sheet from outside vendor sourcing). You really need both verticals in order to be optimal IMO.

Of course if internally an org doesn't already have the platform/infrastructure to dogfood in the first place, this admittedly makes the proposition a bit more of a gamble.

blimey indeed. This is a lot of work to set up.

I think you'll see more platforms that offer this set up as a service:

* cheap storage / datalake

* pipelines to get data to storage

* BI / dashboards on top of the storage

We're doing this at Definite (https://www.definite.app/) with Iceberg (same as in this post) + DuckDB as a query engine.

We are waiting for Metabase to support DuckDB on their cloud version. It's pretty neat.

Reply on news.ycombinator.com