Hacker News Reader

Comment by MichaelGG | Hacker News Reader

I think it's astounding how "easy" we can get away with stuff these days. But how does this scale to larger datasets? Say, capturing all signalling traffic for a telco. It's not hard to have 100s of GB a day - that's only a couple 10s of Mbps. Suddenly 20GB/sec means 5-10s queries just for that day.

They can also leverage the bulk speed to reduce the size of posting lists. Just index each term to a chunk of, say, 1-5GB. And don't even index full terms, just hash them - a 64-bit hash gives you enough room for around 10 billion unique terms without causing much problems with collisions. New data in the last hour can be brute forced while the index is being generated.

OTOH they seem successful so I suppose it's working quite fine.

View on HN

If I read correctly, this was staged out on a single core user machine, then pushed out to an Amazon 16 core machine. Not the most powerful either, and no mention of any tuning. They mentioned looking to hit max scale out by simply adding more machines. Which is why they mention going from 20MG -> 100 -> 1000 not being an issue. In theory, wire speed is their limitation before they have to look at other methods. At 1000MB/s you could start chunking your data into result sets in advance, taking the time it takes a human to read the results visually as always being the bottleneck, and using that time to your advantage, to make the user think the app returns answers instantly all the time. I can't imagine 1000MB/s ever being too slow when you factor in their current dashboard and what it is returning. More complex dashboards and things certainly will need reworking.

I do something similar with the stuff for http://lifecode.co

Couple of terrabytes processing easy but yeah limited to only a few queries per day.

A lot of the effort goes into caching so disk reads and swap file use is minimised. Really that's where all the overhead is.

All the public stuff for that is at

http://theborgmatrix.com

Having worked at a telco where our tools would take a minute or more for results to come back, 5-10 second queries that we only do a few (or even a few dozen) times a day is _fine_.

I was thinking more along the lines of SIP tracing for customers, which happens much more frequently for tech support roles, at least in my experience. With a quad core and a single 1TB drive I was able to service 100s of GB a day of SIP with ~1s response times. Use a mix of brute force (for temporary log) then index in background.

The interactivity shifts quite a bit when you get close to "instant". Though I know some companies that are content with even 30s query times, but that's gotta limit flexibility.

I guess its an interesting difference right - what say a developer would consider "instant" (lets say <10ms) and what the end user is satisfied with as instant (maybe <1min)...it's interesting the tradeoffs you can make when you loosen your latency requirements.

Thinking like this leads to products like Rational Doors, a requirements database where searching is dog slow and you need to know where to look if you want to find anything.

Really? I've no idea what that product is but I hardly think it's a "thinking" problem, more a managing expectation or shit requirements gathering problem. If it was necessary to search in a certian way over a certain size in a certain time, you could make the appropriate trade off. Sounds like you just wanted to bitch out about a particular product pal.

I admit I like bitching about slow software. But if you don't put speed as one of your top requirements, over time your software will become slower and slower. Having one operation that takes 100ms instead of 10ms is fine until two years later when somebody else who doesn't bother to read your code uses it to implement something that now takes one second instead of 100ms, which is incidentally used a hundred times to complete some query that now takes minutes instead of seconds. Now imagine you're a company like IBM or SAP that works on the same code base for decades and the cruft keeps layering and layering until even finding out which part of the code needs to be tuned becomes too expensive to explain to your customers.

There is http://accidentallyquadratic.tumblr.com with a nice collection of "it was fast enough at the time" code.

Why would it be difficult to profile and find the bottleneck after a few decades? It should actually be even easier with better profiling tools and practices.

Uniformly shitty code is hard to profile. See for example what Chandler has to say about writing good code

https://www.youtube.com/watch?v=fHNmRkzxHWs

There's an explanatory post there that in particular is very much worth reading: http://accidentallyquadratic.tumblr.com/post/113840433022/wh...

>what the end user is satisfied with as instant (maybe <1min).

There's a number of thresholds you can hit:

5 minutes: this is taking a while, better grab a coffee

1 minute: this is taking a while, I can let my thoughts drift a bit

10 seconds: it will be done soon, I'll just stare at the screen till it's done

1 second: it's not slow, but sluggish and interrupting my flow

200 milliseconds: that result came instantly when I pressed the button

50 milliseconds: the results update instantly when I use this slider

Hitting another of these milestones transforms the user experience and allows the dataset to be experienced in a new way, allowing more spontaneous exploration of the data

Reply on news.ycombinator.com