maxisaurus

thanks for the feedback!

- "automates root cause analysis" -> it means (1) showing which rows have affected the metrics and (2) provide some automated context (is it an update? a delete? a dimension that changed? etc). But it is still very early for 2.

- Metrics are defined by users in their usual "data" repository (using dbt for example). The metric computation is not defined on Datadrift, we only go "read it".

- No, it's really for batch processing in a data warehouse (like hourly / daily computations)

- That's not something we had in mind (I know some dbt package can help you do this)

Hey HN,

Sammy and Lucas here. We are building an open-source framework that monitors your metrics, sends alerts when anomalies are detected and automates root cause analysis. Think of Datadrift as a simple & open-source Monte Carlo for the semantic layer era. The repo is at https://github.com/data-drift/data-drift

Datadrift started as an internal tool built at our former company, a large European B2B Fintech. We had data reliability challenges impacting key metrics used for financial and regulatory reporting.

However, when we tried existing data quality tools we where always frustrated. They provide row-level static testing (eg. uniqueness or nullness) which does not address time-varying metrics like revenues. And commercial observability solutions costs $manyK a month and brings compliance and security overhead.

We designed Datadrift to solve these problems. Datadrift works by simply adding a monitor where your metric is computed. It then understands how your metric is computed and on which upstream tables it depends. When an issue occurs, it pinpoints exactly which rows have been updated and introducing the change.

You can also set up alerting and customise it. For example, you can decide to open and assign an Github issue to the analyst owning the revenue metric when a +10% change is detected. We tried to make it easy to customise and developer friendly.

We are thinking of adding features around root cause analysis automation/issues pattern analysis to help data teams improve metrics quality overtime. We’d love to hear your feature requests.

Datadrift is built with Python and Go, and licensed under GPL. Our docs are here: https://github.com/data-drift/data-drift?tab=readme-ov-file#...

Dev set up and demo : https://app.claap.io/sammyt/drift-db-demo-a18-c-ApwBh9kt4p-0...

We’re very eager to get your feedback!

Work a lead data in a fintech company based in EU. I built a simple observability tool for key data assets in a data warehouse. It's a python monitor you add to a given table, it checks that table daily and tells you when there is an issue & which rows introduced that issue. We used static testing framework like great expectations but that was not enough. We did not have the budget for the big data observability players like Monte Carlo, so we kept it simple.

Repo if interested: https://github.com/data-drift/data-drift

(Disclaimer: I am focusing full time on this project to see if it's an interesting business opportunity. It's 100% open-source -- feedback welcome!)

Congrats for this - Love the bitemporal aspect. It was a real struggle for me in past analytics experiences where we spent a lot of time recomputing key metrics 'as of' certain dates for reporting / auditing.

Been following this https://news.ycombinator.com/item?id=38108044 as well, might interest you!

It's more about engineering management but 'Accelerate: Building and Scaling High Performing Technology Organizations' by Nicole Forsgren (Github VP), Jez Humble and Gene Kim is a must-read.

Currently doing it for an open-source metrics observability and troubleshooting tool (15 PoC in production, no revenues yet). Committed about 30% of the amount so far, but it's tough and expectations seems ever increasing (revenues, community traction etc). Curious to hear others experience as well!

Have you considered mentoring with platforms like codementor or superprof? Not sure it fits the "reasonable amounts" but it sure is a nice exp.

One that comes top of mind is Swedish "startup" H2 Green Steel (https://www.h2greensteel.com/). They're building a steel plant powered by a giga-scale electrolyser to produce hydrogen (rather than using coal).

Aligned with the humble way. Have you tried the user research angle like "hey I'm building XXX, thought it might be useful for you because YYY. Would you be open to try it and give us your feedback"? I've been doing this for a dev tool for data analysts and works pretty well. Anyway keep trying and good luck, been there and it's not easy.

Never tried it myself but check OnlyDust (https://www.onlydust.xyz/) PS: not affiliated with them, just know on dev there

Congrats - love this, especially joining across old and current data (or any point in time if i understood well?)

I recently stumbled upon the concept of "bitemporal modeling" (ie. rewinding data "as of") - thought it described well this use case.

I guess there's plenty of technical solutions to track changes, like scd or snapshots or audit tables - what I find interesting about git approach it's that it's much more user-friendly vs. SQL archeology to understand what changed?

Same situation as you. Pushing hard on content (quick demo video has been really helpful for us). That and targeted emails to people having their own community (substack NL most often)

Been looking for an open-source survey alternative tool for a long-time! Congrats on the launch.

(sharing this great article btw => https://staltz.com/time-till-open-source-alternative.html)

Hiya! I'm a former VP Data in a large Fintech. We were using Progression for career path and "level up" assessment, thought it could be helpful for you.

It was also used by the engineering team and they have templates from FAANG companies to help you as well => https://progression.co/library/?tag=Engineering

Made me think of Excalidraw when I saw it - cool stuff! Will try my 750 words on it.

(love what you do at Lago btw!)

Same for Chargebee lol. Was an analytics lead at a fintech company. We computed our revenues in our data warehouse based on Chargebee data. It was a nightmare because:

#1 - raw data is indeed very messy.

#2 - MRR can be theoretically be computed on either the invoice or the subscription table. Using invoice: sum the subscriptions fees by month. Using subscription: sum the subscription fees for the period where the subscription was active . Well, does not really work that way. In practice, invoice table holds a lot of noisy information not useful for MRR and get the active period for a subscription is not straightforward (involves snapshots and scd).

#3 — Discounts and offers. Chargebee's coupons and discount pretty much are a blackbox with some pecking order.

Long story short: we never had the same number as Chargebee UI.

Been that person for a long time at work and I had a hard time learn how to say no. I started pushing myself for "1 no / week" (with explanations ofc), started to get used to it and saw cool improvements afterward (most people requests are not that prio).

Been using Brevo (https://www.brevo.com/pricing/) in former companies. Contact are unlimited and it's around $30 per month for 40K emails. Provide loads of other features but never tried them

Was the employee #1 at a B2B SaaS providing supplier invoice payments. 2 things got us really started in the early days: - design partnerships (great read of A16Z here -> https://a16z.com/a-framework-for-finding-a-design-partner/) - indirect distribution (for us it was accountants)

Hope this helps, good luck!

How I spend my time on the internet is not something I want to see lol. How about privacy issues?

Co-founded a startup this summer, growth slower than what we want for now (isn't it always the case?). Every users feedback and business angel calls is an energy-giver. Every VC call has been a drainer.

Hey! Thanks for showing :) I work on a full remote team and we've been using Gather for our virtual office (https://www.gather.town/). We love customizing our space, tables, chairs etc but they lack your "smart" blur part (if that really works?) and health tracking part.

Shoe Dog by Phil Knight (business-not-so-business book, loved it)

Would 100% recommend Posthog, it's open-source and a great product. "Classic" events (like pageviews) are tracked and out-of-box dashboards with cohorts, DAU/MAU charts ready to be used & shared.

Used it in my former company and been using it ever since (not affiliated in anyway with them btw)

Works like a charm on my mac (intel)

Been in analytics in fintech scale-ups for the past 5 years. Been there, done that for financial or regulators reporting with manual SQL archeology across snapshots