Comment by dejobaan | Hacker News Reader

Comment by dejobaan | Hacker News Reader

Not a data lake expert here, but I think there's a parallel with telemetry/logging for video games. The author talks about mass "ingestion without purpose," which is what I've sometimes flubbed in the past—during playtesting, we'd collect piles of data about where the players won/lost/moved/whatever, and there was so darned much of it that we'd just toss our hands up and not do anything with it.

The article suggests (sensibly!) that the right thing to do is to actually go in and design what's being logged with purpose (i.e., everything should answer a known question). But I don't always know what I'm looking for when I'm testing a thing. At the risk of using LLMs as a hammer that hammers everything, I wonder if they'd help with that problem?

I've pointed Codex at Posthog logs, for instance, and it's been able to say, "yeah, well, look at this pile of residential proxies hitting your site." I wonder if it'd similarly be able to say, "hey, I notice a pattern where everyone who picks the Cleric class stops playing after level 3"?

I'm building a product and exploring these questions in that domain. I'm thinking about it a little differently. I have a known set of hardware sensors that I'm collecting all the data from. Per sensor the data has a strong schema with a timestamp, what is interesting is the insights that can be gleaned across sensors. So what sensors do you have that you want to record, have a strong schema against that sensor, now have something you can use to tie across sensors, now you can use LLMs to query the data and draw interesting insights

You could even ask the LLM what other sensors fit here and how would it expand my knowledge in collaboration with the other sensors and data feeds.

It used to be normal for video game playtesting departments to simply film the player playing the game. Things that are important are usually on screen if your game doesn't suck. It may be tempting to record everything that happens in the game world, but if you can't work it out from what's on screen, neither can your prospective players.

> look at this pile of residential proxies hitting your site

How does it know that?

Yeah, my impression is that even the best LLMs are pretty bad at analysing big data directly.

It's not even the hallucinations that are the biggest problem, it's more that they're so bad at managing their context windows that they end up ignoring huge chunks of the data without realising that they're doing it. You end up with a result that looks plausible but is often extremely misleading.

A better approach seems to be getting the LLM to write code to use more traditional analysis techniques (eg. iterative k-means, or whatever). That way you can at least be confident that you've looked at all the data rather than just a small slice of it.

Reply on news.ycombinator.com