Hacker News Reader

Comment by nice2meetu | Hacker News Reader

This reminds me of my previous job where we had an optimized single-server solution, whereas the competition scaled horizontally. They couldn't out-scale us at what we did until they got to 20 servers or so, which only the biggest customers needed anyway, and even then they would be really slow. You can achieve some impressive speedups when you really focus on the problem.

Still, it wasn't that helpful from a business perspective, because apparently it is easier to justify mark up on solutions with lots of servers.

View on HN

I’m currently working on a project that sounds similar to this. A single server processes and aggregates steaming data with about 60Gbps of data. Customers want horizontal scalability and lots of buzz words are being suggested. A active-active configuration would be best (in my opinion) as that solves for redundancy and failover. Overall fun and challenging project. Somewhat frustrating at times though because of all the opinions of which technology or architecture would be “better”.

What happens when your single server solution is disconnected from the network for whatever reason or you exceed the current configuration? Bad times i bet. Cost of components isn’t the only factor here.

This may seem trite, but if you can get 20 servers worth of performance out of one you can afford to run two active-active and still reap a 10x capex/opex savings. The technology to have simple but reliable systems has been around for decades. You also can't assume that the cloud is going to never fail, so you always have to defend against failure whether it be running two servers or two availability zones.

Also a lot of the time you're not trying to achieve (and can't anyway) an uninterruptible uptime - you just need a rapid recovery from infrequent outages.

Apples to oranges - you’re not gonna get 10x saving by running a/a in a strong consistency mode

I don’t see how this follows. If you have one single-threaded server doing the job of 20 similarly specified servers running the distributed system, you could run every job twice, on two completely independent servers on two completely independent networks, and still be 10x as efficient unless both failed catastrophically during the same job. Or you could run three completely independent copies of the same job and still be at nearly 7x efficiency, etc. There is no need for any sort of “consistency mode” here. This is just brute force, without any synchronisation between servers or resumption of aborted jobs at all.

That only works if the servers don't rely on external dependencies or if every single opertaion is idempotent.

Weren’t we comparing parallelised and single-threaded data processing jobs, though? If you’re introducing external communications in real time as a factor, wouldn’t that be a fundamentally different situation?

TP specifically refers to many servers. Conceptually it isn’t actually that different - the cores and memory are on the “network” of sorts.

That’s only if there is no contingency plan in place and a backup server. Usually these things are thought out beforehand. And it’s not like you’re excused from this type of problems when you’re running your solution in the cloud. From cloud outages to configurations nightmares, data inconsistencies, not knowing what is happening because pinpointing on a complex infrastructure takes more time and so on. Sometimes the cloud way is the way to go but other times it is not justified.

We had a replication HA solution, but it was "warm", and customers didn't really want to run another server doing nothing. I think we also charged a lot for it.

The area we were in, customers could tolerate some outage. Restoring from a backup didn't take that long. I believe we were known for good support helping people get back up.

But yeah, what you are saying is all the arguments that kept coming to us. People were used to having a few servers for the database, and different front ends, and layering things like that.

Isn't this a solved problem? Just use a failover, right?

If your consistency model allows you to use their asynch replication and potentially lose data in surprise switchover - sure. Otherwise you’re looking at similarly worse performance and also worse reliability

In our case async replication was acceptable, so we had a failover solution available.

Reply on news.ycombinator.com