Hacker News Reader

Comment by jhoechtl | Hacker News Reader

I disagree. Big data came associated with a new swap of algorithms. To big to handle? Use new algorithms, maybe not 100% accurate but can handle the load. And streaming data as opposed to static data.

The are a lot of approaches like Change Data Capture CDC or HyperLogLog - but the norm? Far from.

I think the marketing BS fell out of fashion when every database designer became a data scientist, but that's another issue.

View on HN

Big Data was about being able to store and query data beyond the limits of a single machine or existing database. The point being to store as much data as you possibly can and then extract value out of it. That grew out of Hadoop/MapReduce which let you cheaply store and access data that doesn't fit into one machine. Streaming was not part of the initial marketing pitch.

That said if you want to do streaming nowadays then you just integrate with Segment. If you want to track your database then you can dump data using Fivetran. If you want to track client events in excruciating detail then you can use Fullstory/Heap to do so in real time. That's all now table stakes for any company and outsourced to those platforms.

I'm non-technical on business/strategy side of things at a tech company, I have no idea what any of those things but I interact with my company's datalake/warehouse. I don't even know the right term for it but its the source of truth of all reports, dashboards and presentations. I don't know what they used before this but I imagine it was a bit more painful to use

Those algorithms and improvements in large data processing got bundled away into a platform/infra layer a developer or user interfaces with unaware of what's going on in the background to produce the results they want.

Reply on news.ycombinator.com