Comment by kevinsundar | Hacker News Reader

Comment by kevinsundar | Hacker News Reader

Wow this is an oversimplification. I've had years of experience working in a data lake within a FAANG handling > 5 PBs of data per day ingest. There's so many things this misses:

1. What if the domain teams don't actually care to maintain data quality or even care about sharing data in the first place? This model requires every data producer to maintain a relationship with every data consumer. That's not gonna happen in a large company.

2. Who pays for query compute and data storage when you're dealing with petabytes and petabytes of data from different domains? If you (the data platform team) bill the domain teams then see above, they'll just stop sending data.

3. Just figuring out what data exists in the data mart (which this essentially is describing) is a hassle and slows down business use cases, especially when you have 1000s of datasets. You need a team to act as sort of a "reference librarian" to help those querying data. You can't easily decentralize this.

4. How do you get domain teams to produce data in a form that is easy to query? Like what if they write lots of small files that are computationally difficult to query, whose gonna advise them? Data production is very related to data query performance at TBs scale. The domain team is not gonna become experts or care.

5. What do you do when a domain team has a lot of important data but no engineering resources? Do you just say "oh well, we're just a self-service data platform so no one gets to access the data"?

At some point one has to ask, what are you guys doing ingesting 5pb of data per day?!

Unless this is google, that doesn’t make any sense. That’s an average of 7.5mb per human on the planet, every day.

Why the concept "data lake" emerged in the first place:

Thinking about what to store (and what to log) is not trivial and takes careful consideration. Plus, there's always the argument: "But what if we need something that we forgot to store or log?".

Answering that takes time and risk-acceptance that most developers in most projects don't get.

"Datalake" was just a pseudo-solution to gain peace-of-mind: We throw everything into a big bucket and figure it out later.

Worked in a similar environment. Events and logs. When a single page view does 200+ database queries, triggers another hundred requests, ML services, analytics and other tracking, and a transaction kicks off a chain of a hundred events in your stream, it's pretty easy to reach those insane numbers. That one page view can easily add 1MB of data.

Just like performance optimization, it's cheaper to buy more hardware than to pay for humans to think about it and coordinate.

I did say FAANG :)

Believe it or not this is just for security data. But that fact combined with SOA leads to lots of logs. (5Pbs is the uncompressed amount, but we decompress incoming data, then ETL).

Would love to know some of the answers to the questions you posed

I believe the data mesh claim is that as well as the operational API exposed and supported by some given domain team, the domain team also gets a new goal of exposing and supporting an analytical API to deliver a data product to potential consumers. For this to happen, an organisation would need to value this new objective -- perhaps comparably to operational objectives -- and fund and resource it adequately. Arguably throwing enough budget and appropriately skilled headcount at it might address points 1, 2, 4, 5.

But, it's not obvious it actually makes sense for an organisation to value analytical concerns at a similar level to operational concerns, with similar resourcing. The value to the organisation of building and maintaining analytical APIs to serve data products is likely to be considerably less -- perhaps by one or two orders of magnitude -- than the value produced by actually performing and maintaining the core operational function.

If the stars align, maybe in future the analytical data could be used as input to an optimisation project that improves some key metric of the operational function by 10% or 1%. How much is it worth paying to have the possibility of an outcome like that in future? Not obvious, really comes down to how valuable a 10% or 1% lift would be, and how much it would cost, needs some kind of business case. Not obvious that resourcing analytical data APIs owned by arbitrary domain teams everywhere is a sound investment for the business.

Just on point 1, maybe in the context of said FAANG data quality was by choice if at all. In other industries, e.g. Finance, it can be regulated and audited, so operational teams care in terms of not being able to continue operations. That addresses point 2 partially, it is a complex topic, though.

In my experience data quality in finance is much worse than FAANG. It's common to have just the raw data feed from markets/trades/network dumped into a OLAP DB and whoever is using it has to sort through it whereas FAANG have data engineers to clean stuff up.

I am in Finance and while I don’t question that may be the case for some, the reality is that with regulatory requirements getting more sophisticated, anyone who does not focus on DQ end to end is making a very costly decision. When the quality of data pipelines ends up impacting the capital requirements, the cost of bad data quality is hitting the PL straight away.

On point 1: "That's not gonna happen in a large company."

This doesn't happen in small or mid companies, either. Or if it does happen, it happens begrudgingly. SWE's have too much to do.

Reply on news.ycombinator.com