Hacker News Reader

Comment by hinkley | Hacker News Reader

The biggest thing we're going to regret looking back on the early Cloud era is the foolish notion that you need 100x as many servers to do 100x as much work.

Servers don't scale linearly. It's more likely you'll need, at a minimum, 107-110x as many servers. Between 100 + log(100) and 100 + sqrt(100). So making your code 100x faster saves you more than 100 servers.

I think the arguments never end because it's all fuzzy math. Get too attached to running the whole thing on one server, and your architecture starts making irreversible decisions about where the source of truth is, and you get locked into a single server.

Give up too early, and you spend a lot of energy herding cattle instead of building features. And speed is something you outsource to the guys who write the checks. That's fine if you lock in your vertical, but I've worked on projects that lost out to a more responsive or financially efficient competitor. It really, really sucks.

View on HN

> So making your code 100x faster saves you more than 100 servers.

Yeah, but "making your code 100x faster" might cost significantly more than just adding 100x servers.

As with most things, the only absolute rule is "it depends".

adding 100 servers damages the environment 100x more and costs jobs. There should be no question that optimization is the responsible thing to do, just buying servers is wasteful, greedy and irresponsible.

We need a commitment from all the (supposedly) environmentally conscious SV companies to pick optimization if given a choice between making the software faster and just throwing more computers at it. That would be relevant and meaningful, not just the usual "our office AC isn't set as cold as possible, so we are green"

I mean, as a general expectation, that’s ridiculous. Businesses will always optimise for profitability, expecting them to eat their margin voluntarily for environmental concerns is unlikely to be a solution.

As a policy, though, it makes sense - tax the environmental externalities appropriately to reflect the actual shared cost. If it’s still economic to add the servers once those costs are factored in, then you wouldn’t have any grounds to complain.

Problem is, taxing electricity appropriately shuts down your whole economy. Prices would need to be sky-high before SV would feel the squeeze, till then mom and pop wont be able to run their electric hob and your neighbourhood steel smelter will move to china even faster. Taxes may seem like an easy solution, but if you look closely it is far more complex.

There's nothing saying you have to tax the first watt like the millionth watt.

Just design the tax regime so that it is high usage users who are paying the high costs, and exclude any usage levels that would impact homes, small businesses, and "normal" acceptable uses.

I agree that tax policy is more complex than just "tax the shit out of it", but everytime a tax is proposed, there comes a group of people who argue that taxation is so complex that we can't possibly grok it. That is clearly not the case.

Everytime a tax is proposed, people like me suggest to look at the German Energiewende. Small businesses and private homes are taxed to hell and back on electricity, to keep prices for large businesses internationally competitive, because those bussineses can just pack up and go elsewhere. Or cause a revolution when going bancrupt. Taxing large consumers more also doesn't work. Taxes are never the single answer, you always need to combine them with import duties, ending trade agreements, leaving the WTO, maybe even leaving the EU to be able to levy duties on other European countries with a lesser energy tax. Because that is what it takes to create a fair tax regime on energy.

I guess only the very radical greens would want to stomach that backlash...

I'm not sure why you are being downvoted, that sounds pretty reasonable to me. The idea that we'll have to make (at least) this level of compromise to survive as a species seems to still not be taken seriously in the tech world.

I can sympathize with downvoters, demanding change is always easier when the demand is aimed at others, not yourself or your peergroup. And I'm not sure if this level of compromise is really necessary, or if optimizing computer power consumption is just negligible on a global scale and the resources spent to optimize here (it will cost money after all, even if the initial investment pays later on) would be better spent elsewhere.

My comment was aimed more at the hypocrisy of claiming to be green and then skipping the obvious savings in optimizations.

No, it definitely doesn't have that direct impact on environmemt. If you're using cloud servers, what it costs to you is not always connected to how many physical servers are running. On top of that, AWS, for example, claims to use renewable sources for over 50% servers.

Moreover, there are other considerations. If you're spending too much time on optimization, the company may not survive... leading all the work done to potentially go to waste. What about the environmental impact of waste there?

> Yeah, but "making your code 100x faster" might cost significantly more than just adding 100x servers.

Cost? No. But it certainly requires more talent.

Unfortunately, programming talent doesn't scale with cost at all. In that it's like a creative endeavor: you can hire 10000 "professional writers" if you want, but that won't get you another Shakespeare.

Which is why I find the "software engineer" moniker silly. Software isn't at all like engineering.

The term “software engineer” was invented to describe the work being done by Margaret Hamilton and her team, when developing the guidance software for the Apollo moon landings. It definitely is a form of engineering.

From Wikipedia:

> When I first came up with the term, no one had heard of it before, at least in our world. It was an ongoing joke for a long time. They liked to kid me about my radical ideas. It was a memorable day when one of the most respected hardware gurus explained to everyone in a meeting that he agreed with me that the process of building software should also be considered an engineering discipline, just like with hardware. Not because of his acceptance of the new 'term' per se, but because we had earned his and the acceptance of the others in the room as being in an engineering field in its own right.

https://en.wikipedia.org/wiki/Margaret_Hamilton_(software_en...

> The term “software engineer” was invented to describe the work being done by Margaret Hamilton and her team, when developing the guidance software for the Apollo moon landings.

There's a considerable debate about when and where this term was "invented", apparently with some oral histories placing it even into (late?) 1950s.

As for grandparent's comment, I think it was referring to the fact that the nature of the job is completely different. For example a large portion of a traditional engineer's job deals with fighting the natural world. A bolt may rust away and stop doing what it was doing until then. A source code line in the form of

  a=a+7;

won't do any such thing. As long as it exists it will keep doing the same thing. Hence software engineers have to deal with a completely different set of issues, to the extent that calling it "engineering" is to a certain extent metaphorical.

Code rots/rusts faster than most physical objects. Requirements change, libraries update at a breakneck speed. I've had to deal with some (functional) languages that don't allow you to change the the variable where some would allow that sort of modification just fine for your simple example. Change is everywhere.

> Code rots/rusts faster than most physical objects. Requirements change, libraries update at a breakneck speed.

That's a forced metaphor as well, not actually an identical process. If "requirements change", that's not the same thing as a bolt rusting away. Surely the requirements for the rusting bolt didn't change, or the requirements for its "dependencies", i.e., the things it's attaching together.

> I've had to deal with some (functional) languages that don't allow you to change the the variable where some would allow that sort of modification just fine for your simple example.

I completely fail the relevance of that. Or maybe it is somewhat relevant, but in the completely opposite way, i.e., that this phenomenon does not even have an analogy in the physical world, thus underlining the vast differences between software engineering and classical engineering disciplines. (Likewise the dichotomy between a computational procedure and the process generated by said procedure, the latter being constrained in the way you describe, doesn't exist in traditional engineering either.)

That has nothing to do with talent IMO. Every software engineer worth their salt should be able to inspect a full stack, measure and evaluate the performance. Instead you have 80% people stuck in whatever the current fashion is (js, I look at you) and that cannot tell a png from a .so .

So in the end, it is more about attitude, not talent.

Agreed. When we focus on the part of the spectrum of optimizations that do require talent, we're thinking of the exceptional, interesting, makes-a-viral-post-on-HN stuff.

The bulk of the work, in my experience, is simpler than the programming part. It's just a culture/priority thing. Like how most of the industry used to not use source control. Now we do, we didn't need to add "talent" to get that done either.

I agree. One heuristic is the specs of our computers 20 years ago had.

Those computers were an order of magnitude slower. Yet many equivalent apps today run at about the same speed.

Performance is not being prioritized. And I suppose back then, devs had no choice given the resource constraints they had.

I've seen many times devs paying for larger databases instance without first even taking a few hours to try and optimize their queries.

The depressing part of all of this is performance optimization is really fun! I love sitting down with flamegraphs and other profiler output and making things faster.

> software engineer worth their salt...

...is a scotsman that doesn't exist.

>Cost? No. But it certainly requires more talent.

Talented people usually cost a lot to optimize your code to be faster. And the more time they spend optimizing existing code, the less time they spend writing new code.

The more time they spend potentially unlocking the ability to write new code, though.

And faster code in existing places can unlock new business opportunities which can unlock more money.

Fwiw engineering talent doesn't scale with cost either. What you've described is actually one of the areas software and engineering are similar, not divergent.

I’m not sure what you’re talking about. I’m not going to ask my developer - who I pay $2000 a day - to spend 10 days optimising my codebase to avoid an extra $5000 in server costs.

It doesn’t matter if they’re a 10xer or a 1xer, their time always has a price tag.

That doesn't fit in my accountant's brain at all. $20k one off cost vs $5k recurring cost could mean anything if you omit the time period! $5k a year depends on your cost of capital, $5k per month is a no brainer

The way I read the comment is $20k cost for a dev to optimize vs. $5k / yr cost in server, making it 4 years until you break even.

It is still not clear if that is a good or a bad thing. It depends on your circumstances

>who I pay $2000 a day

American salaries sound great.

That’s not a typical American salary. $250 per hour is specialist pay, consultant pay, or overpaid.

That's $730 000/year assuming work every day, and $500k assuming 5 days a week plus $30k for holidays.

Engineering is using science to build technology, using engineering principles that are common to all engineering disciplines. Software Engineering is using engineering engineering principles to build software.

Project management methodologies like scrum are implementation level details, together with platform (user interface, programming language, database e.t.c). Software Engineering starts at the feasibility study, followed by detailed design, then comes the implementation phase. Most Software Engineering job posts only list implementation phase requirements, which I reckon has led to SE being misunderstood.

I don't think 100x more structural engineers can make a bridge with a hundred times longer span either.

code has a bigger upfront cost but after the initial investment it's all profit.

Servers can also go down, but the 100x faster code will remain the same.

Most likely, squeezing 100x performance out of your code will significantly complicate it. For example, you're probably going to be dealing with cache invalidation a lot, and you will pay for that complexity forever, as you try to build new features.

It depends on the baseline, I suppose. I could see code written (poorly) in a dynamic language to be 100x slower than necessary. Maybe it was fast enough when written, so nobody looked too hard. Inefficient DB queries, excessive IO.

But I agree with the sentiment. In a code base where performance matters, that level of optimization would often be impractical.

It depends. In many benchmarks Java is 50X faster than Ruby. So having code 100X as fast can just mean choose a fast language and framework for your project

That's under the assumption that code doesn't have to be maintained, updated with new features, and that writing it on any programming language has the same cost.

Otherwise we would all be writing assembler.

Humans are way, way more expensive than servers in almost every case. Exceptions might be small pieces of code ran by millons of devices, but that's far from being the common case.

Assembly is expensive because you have you re-write your code every time your architecture changes.

Architecture (x86_64 or arm on mobile) hasnt changed in decades. The way to optimize for it changes in years but you get a slightly less optimized version. Im sure software written in 2007 in assemble for linux pc works now, and fast. I think assembly is not cheap because the language is tiny, offers no abstraction and youre on your own for everything, not because of arch turnover.

If the code has been optimized to the point of being illegible/difficult for others to maintain then, assuming you’d like to update that code at some point, you’ll need to pay for that too.

> The biggest thing we're going to regret looking back on the early Cloud era is the foolish notion that you need 100x as many servers to do 100x as much work.

> Between 100 + log(100) and 100 + sqrt(100).

I bet it doesn't even make the top ten. You have so many other problems by the time you're scaling to hundreds and thousands of servers and beyond. A few percent difference in server count? Whatever. That's still almost perfect scaling, and not going to be your bottleneck or your huge cost center.

Servers can and do scale effectively linearly depending on the workload.

The broader point stands though. The more servers you have, the more money a 1% cpu optimization saves you in raw dollar amounts. At a certain scale it always makes sense to optimize your code.

The moment the servers need any consensus at all, you've opened the door to Amdahl's law. That could be sharing caches, running Raft, or talking to a transactional store. Even if it's just to keep my UI in sync between the desktop and mobile site, you have costs.

But since so many sites have social aspects now, you need to be able to communicate between accounts, which means interprocess communication.

The Amdahl's law fraction is at least logarithmic to the number of users, but more likely to be proportional to the 'surface area' - sqrt(n). Every new server gets you a little bit less oomph than the one before.

There is basically no overlap with Amdahl's law here. Amdahl's law just says that there are diminishing returns on the whole when increasing parallelization in a program with significant serial parts. It is a fairly obvious observation, but people seem to distort it and apply it to general scalability.

First, everything needs to be synchronized but that doesn't mean synchronization is expensive or blocks computation from happening. Lots of synchronization isn't even dealt with explicitly by users, it happens in queues in the kernel, in networking hardware, or even in the memory controller when talking about reading from memory.

The closest thing is the database becoming a bottleneck, but that is not because of Amdahl's law or even synchronization, it just about reads, and those are extremely parallel, so it likely comes down to total CPU power when it is a problem. That's why query caching can be so effective - it isn't about synchronization and definitely not about serial parts of a program, it is just about usage.

For connected social media sites, not everything has to be done on a page request. Anything done ahead of time is also done in parallel.

And remember, even if we aren't sending more and more data between our users, we're collecting more and more telemetry data from our servers all the time. Coalescing logs and gathering stats requires funneling data from everywhere down to one machine (the one that built the graph)

I actually wasn't thinking of social sites when I said "certain workloads".

It's true that every server extra has a bit more overhead because of stuff like caching / load balancing / whatnot, but the amount extra can be tiny even at very large scale. For certain workloads that are parellelizable.

(It'd be fun to talk in details from personal experience instead of in general terms, but you know those pesky NDAs...)

Unfortunately the probability of failure also scales up.

Let's say you bought good hardware and the probability it will fail is x.

Let's say you buy a hundred of those. The probability that one of them fails is now 100x. You just increased the probability of failure a hundredfold. You will need a load balancer who can detect failure and not forward to that machine anymore. The load balancer can fail, too. Now it's 101 fold. You also need a separate database if you share sate. 102 fold. Make it redundant. 103 fold. Add router / switches. 105 fold. Make those redundant. 107 fold.

I have never used more than one crappy PC for any project of mine, and some of those have had/have pretty substantial load on them. When I say one server, I mean one PC but with two power supplies and RAID storage. I'm not trying to tempt fate here.

But I have no devops team, no admins juggling failing machines, I don't have racks. I have one slot in a rack somewhere. The machines have been running for years. Every few years, as I approach MTBF of the hardware, I move to a new server. That entails a small downtime.

Result: People use my servers in their "is the internet up" scripts. Because my failure rate is a fraction of what distributed systems have.

In fact, practically all of my downtimes have been software bugs in my code, or me fat-fingering input somewhere. I can't even remember the last time the hardware failed. Not even sure I ever had an actual hardware failure that caused downtime. I once had a raid hdd throwing errors and it had to be replaced. I think that's it.

It’s usually more about specific bottlenecks than specific code in my experience. It’s hard to use too broad of a brush on these things.

From my experience its almost always the database. On any sufficiently complex web app with a ORM its easy to trigger literally 100 queries by loading a single page if you don't make a specific effort to avoid it.

And that is the first universal truth in this whole thread.

Add tests to check that you have not inadvertently added a massive innefficiency with a simple change.

I'm not sure where this argument is coming from?

If you already have a load balancer, you should be able to scale on the server part from 1-100 exactly linearly. Where are your extra 7-10 coming from?

And if you already have a database, you usually just upgrade its specs linearly too. And if god forbid you need to shard... that's perfectly linear too.

For pretty much all practical planning purposes, servers do scale linearly. The equations you're describing seem to imply a deeper hierarchy of server connections, but that's not usually how it works -- servers (whether load balancers, web servers, or databases) just come in bigger pools, not trees.

Where are you getting your log() or sqrt() terms from? Is it some kind of application-specific caching layer or something you've used in the past?

> If you already have a load balancer, you should be able to scale on the server part from 1-100 exactly linearly.

I think this only counts in the simplest of cases? Such as serving HTML straight from memory.

In other cases there are various micro or macro services to call to, state to communicate on the enterprise service bus or something equivalent, separate database servers to run queries on, etcetera. A single request might hold up several other requests due to badly-thought-out dependencies between services and physical servers. More servers wouldn't scale exactly, and it might be difficult to find out where the next bottle neck is.

Thanks... but all the things you're describing are still all linear though in terms of servers, no? All the services you're describing will scale linearly.

And if some external bottleneck exists... that's a separate issue and adding more servers isn't going to solve any problems at all.

I guess I still just don't see it... I know servers don't scale "exactly" in the sense that one service will require another server before another service does, but that doesn't make it nonlinear. It just gives it variance (fluctuating both upwards and downwards) around a line. It doesn't turn it into a curve.

And all of that is on the assumption that 9 women can make a baby in 1 month.

> So making your code 100x faster saves you more than 100 servers.

The first thing that came in my mind is the following article from back in 2013: https://blog.iron.io/how-we-went-from-30-servers-to-2-go/

> Yeah, but "making your code 100x faster" might cost significantly more than just adding 100x servers.

And moreover, it takes time to make code faster.

Very realistic scenario is that you have 2 months of runway in the bank and investors have committed to the next round but are holding back the money and still having 3-hour coffees and 2-hour lunches with you about how you can improve your go-to-market strategy instead of letting you get back to work on actually scaling your tech.

Frist spent some time and try to optimize. If that's too hard or time consuming for proper reasons, then scale up. If you see many long bars in htop on the biggest machine your provider has to offer, then scale out.

Of course, you might want to scale out earlier for other reasons e.g. fault tolerance, if you want to allow different kinds of failures.

I think you’re describing fanout, but I’m not sure. What do your math expressions represent?

That’s still linear scaling.

It's asymptotically linear as your funding goes to infinity, i.e. not linear in the real world.

Reply on news.ycombinator.com