Hacker News Reader

Hacker News

Comment by nikhilsimha | Hacker News Reader

Full thread

nikhilsimha

Impressive, but I don't get why folks don't normalize it to events per second instead.

It is 463k eps for anyone wondering.

View on HN

Isn't it 10x that?

Sanity check: say 10^5 seconds/day. Then it would be

400 * 10^9 / 10^5 = 4 * 10^6

https://www.google.com/search?q=400+*+10%5E9+%2F+%2824+*+60+...

You are right. 4.63 million eps. My bad.

And the nameplate capacity - up to 1GB per second - is easily within the nominal abilities of one single machine.

I love reading these threads and thinking this sort of stuff to myself... How much of twitter's infrastructure is self-perpetuated or otherwise exists just to serve other infrastructure? You obviously cant fit everything their business does on a single machine, but you could probably get dangerously close with enough determination. Especially, if you just focused on the core 280 character tweet abstraction.

An options order consists of more data than an average tweet, and we can certainly process them at a higher rate than twitter would need to go in practice. Many financial exchanges operate on a single thread. 1-100 million transactions per second with jitter measured in tens of microseconds. I don't see why other software products & services can't leverage similar concepts.

Yep, probably a single good mainframe would handle all of it with a smile. And provide the uptime that most "cloud" vendors can dream of.

But that wouldn't be nowhere near as cool as building "cloud infra", right?

There’s certainly lots of room for improvement in this kind of system, but I think it’s a bit reductionist to compare this kind of work to HFT, which hits one hot path that is ultra optimized and doesn’t have to deal with complexities like “this data needs to be replicated and kept consistent”. Your question is a bit like “planes go fast, why can’t my car use some of the same tricks to go fast as well?”: yes, your car can probably go faster, and it could probably steal some aerodynamics and material science from your plane, but reducing the problem to something simple that they should “just” do is absurd.

> doesn’t have to deal with complexities like “this data needs to be replicated and kept consistent”

I'm sure there are other excuses available, but this is not a good one. Non-repudiation is a critical system requirement of any financial exchange. The figures stated include persistence to durable media, and the whole point of running everything on a single thread is to ensure serialized processing of all activities (i.e. consistency).

I would argue the need for replicating tweets is less urgent than ensuring 7-8 figure financial transactions don't go unaccounted for. We could probably make some compromises for the twitter use case to make this even faster.

1 GB/s is on the order of how fast a computer can parse requests. Throw in processing on top of that and you’ll slow down considerably.

Reply on news.ycombinator.com