Hacker News Reader

Comment by misterman0 | Hacker News Reader

This is interesting. Without fully understanding the design of this system, I wonder how far it scales. Can you process big data with LoCal?

View on HN

Is it web scale? :-) Sorry couldn’t resist... https://m.youtube.com/watch?v=b2F-DItXtZs

I actually thought that by now, in 2019, this forum would have realized that big data is real. It's a thing. Some of us are forced to deal with it. It's a hard problem that is well-suited for having discussions around on forums that are populated by intelligent people, but apparently it's not a suitable topic on this particular forum. I will certainly keep that in mind from hereon out.

Big data is mainly a marketing term to me. I've been involved in a Big Data project that wasn't because someone wanted to massage their CV with fancy tech. With current multicore machines and SSDs or cheap spinny disks you can hold and process incredible amounts of data.

So, serious question, how do you define big data? By some size threshold, or "it can't possibly be done on a single box", or something else?

And If I may, what is the BD work that you do (feel free to decline that one).

TIA

I actually thought that by now, in 2019, that people doing big data work understood that big data covered a vast number of processing use cases and saying "can it process big data" has about as much semantic meaning as "can it process blue". I will certainly keep that in mind from hereon out.

Well, if you have a big document collection or something, the faster you can process each individual document the more you can scale, right?

I guess I'm not sure what scalability would mean here other than speeding up that individual, sequential processing job on one node (e.g. the map step in a MapReduce).

Reply on news.ycombinator.com