Comment by malisper | Hacker News Reader

Comment by malisper | Hacker News Reader

For a post detailing the modern data infrastructure I'm surprised they intentionally leave out SaaS analytics tools. I find this especially surprising given a16z has invested >$65M into Mixpanel.

Based on my experience working at an analytics company and running one myself, what this post misses out is that an increasing number of people working with data today are not engineers. These people can range from product managers who are trying to figure out what features the company should focus on building, marketers to figure out how to drive more traffic to their website, or even the CEO trying to understand how their business as a whole is doing.

For that reason, you'll still see many companies pay for full stack analytics tools (Mixpanel, Amplitude, Heap) in addition to building out their own data stack internally. It's becoming more and more important that the data is accessible to everyone at your company including the non-technical users. If you try to get everyone to use your own in-house built system, that's not going to happen.

I don’t think Mixpanel fits here. Mixpanel it’s just one end-to-end suite, mostly behavioral data that is captured from user sessions or user derived events/sub-events. Basically web analytics.

The whole point of data infrastructure is that sometimes you’re collecting data from the most random places. Many of that data is not necessarily user behavior. Sometimes it’s things like temperatures, latencies, CPU usage or instrument tallies. Sometimes it’s a stream of minute to minute weather data or timings or anything, really. Besides many companies have been collecting data for decades but it all live in silos where it can’t be used for anything.

Mixpanel can’t capture all that data, or query it, or analyze it. Mixpanel is just capturing a super small subset of web event data and it happens to provide an analysis suite on top that data they collect.

That’s why Segment shows up in this list instead. They help to move a lot of siloed data into a common systems. Mixpanel is just another source of data. You need something like Snowflake to put everything together and be able to do queries across multiple datasets.

That's a great point. On similar vein, marketing teams too are increasingly data driven and would tools like Braze, CustomerIO etc to run personalized data driven campaigns. Support teams are using tools like GainSight

All these tools need to be fed data about user behavior - from apps, server backends, other tools etc. It's a messy data connection problem, not just one way from SaaS to warehouse. Mobile App->SaaS; SaaS->SaaS; Warehouse->SaaS; SaaS->Warehouse and so on.

Don't forget the challenges around identity resolution and privacy compliance to properly join it all up effectively and accurately.

Indeed. Even when you have the full identity graph in warehouse and just want to assign a cannonical-ID (by doing a transitive closure), it is not easy in SQL. We wrote a blog on it (sorry for the shameless plug) https://rudderstack.com/blog/identity-graph-and-identity-res...

Creating the ID graph is a next level problem altogether!!. How do you know a record in Salesforce is the same as the anonymous visitor on your website. Requires joining across at-least 3 (possibly more IDs) - anonymousID, userID email (if the user signs up) and Salesforce record email.

Should the data pipe do this automatically? If not, what API abstraction should be exposed to the user?

Reply on news.ycombinator.com