Best practices can slow your application down

I don’t want to badmouth automerge - they’re working on it and have made big improvements this year. But that ratio - 5min:5ms - gives you a sense of the mountain of inefficiency we live with on modern computers with a lot of modern software.

Modern CPUs and ram are obscenely, incomprehensibly fast. With the exception of a few very specific tasks, you should never be waiting for a modern computer to do anything. We sent men to the moon in fewer cycles than it takes for modern windows to boot, by several orders of magnitude.

Complaining about the global chip shortage is a bit like a hoarder complaining about a room shortage in their overflowing house.

(With carve outs in my scorn for video editing, AAA game engines, AI, etc)

This looks crazy to me at first glance. 4k requests per second, for any reasonable sized request, should be easily handled by a single C thread. With plenty of room to grow.

On the other hand, I've seen technical blogposts and whitepapers on how to optimize a "big data ingestion system" that requires clustering and tweaking and whatnot for a fraction of what we're able to do with a single machine. People should try harder to scale up before starting a cluster and scaling out.

1.) Rush to build. Backpressure and deadlines lead to "lets just spin it up thje easier way"

2.) We said "premature optimisation is the root of all evil". This ended up in devs not thinking about performance at all because "hurr durr" premature optimisation. I've witnessed this mentality causing huge problems in performance due to simplest decisions that would be normal in other circumstances, but snowball into a huge perf issue. Exactly as the title says, best practices can slow your application down, especially when used out of context. Some include "I'll serialise this into json and deserialise it again 5 lines later", "I'll use a regex to parse this simple string", "I'll nest 3 map calls here on a list", "I'll find it later in the list again".

3.) People forgot/never learned what the stack is. Honestly, when people say software was harder to write back in the day, I'm never sure if I agree with it or not. One one hand, yes, we had less tools and materials at our disposal. But at least the thinking through it was easier, since with the small set of tools you have, you could go up and down the stack in your head and know that if you do X, Y and Z, that's what would happen. Nowadays, we have so many tools and layers that it takes a huge amount of knowledge, time and mental energy to reason about your whole stack. We have people specialised in tiny subsets of the stack, so getting performance squeezed out at overall stack level requires people who understand more of the stack, which is getting rarer and rarer by day.

4.) We have lost touch with the power of hardware.

Consider that most of senior devs are from an age where hardware was way less powerful and in the last 20 years, the power and availability of hardware has increased so much that it is hard for us humans to grasp - especially since the machines we use on a day-to-day basis still work nearly the same. Yet, we hear all these stories about hundreds of lambdas serving thousands of requests, so most of devs don't even have an idea of how much a single machine can serve but picked up "oh it's cloud scale" mentality. Combine this with "fear of low resources" and the ease of spinning up a new machine when resources are low and the answer is obvious - spin it up.

5.) We still think linearly.

People don't think exponentially most of the time. We don't assume that line of code will net us 30x speed improvements. We don't assume this loop will go quadratic. We tend to ignore tiny accumulations until it's too late. And when met with the performance wall, we discard optimisation because its hard to believe that we can easily 10x or 20x the speed since in our heads "it's just unnatural". You can't change tires on a car and have it go 2000mph suddenly. But with code you can, that's the problem.

Throwing $350K/yr at AWS sounds like peanuts to me, to be honest. How many extra engineers would you hire if you could completely avoid $350K/yr in expenses? One?

There are other, totally valid, reasons to understand your computing environment, but a mid-six figure bill isn’t one of them for anyone with millions in engineer payroll.

Software and SaaS is a very high margin business compared to storing widgets in a warehouse and selling them in a retail store. So, you're right: who cares about efficiency?

(Well, it is important, but it isn't always the top priority.)

But I’m in a similar boat, and when I think about our ratio of cores to requests per second, it’s scandalous.

When a problem is big and the patterns are set, there’s not much you can do, and the people I flag as potential recruits have a habit of quitting instead of diving onto this grenade. I have learned all new ways that some known (to me) anti patterns can screw you over.

And it grates that Tony Hoare said this 40 years ago:

    I conclude that there are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies and the other way is to make it so complicated that there are no obvious deficiencies.

This is practically worth its own thread, so I will.

I think it’s the principle of the excluded middle biting us on the ass again. Vertical first risks having people assume shared state across events. Everything collects in memory, because look at all of the “unused” memory that nobody will complain about me using for my dumb feature because nobody is looking yet (when the moments are still teachable).

To your hardware point, many people don’t understand how that “free” memory is used by the OS to reduce mean response time. So they overload the machine and don’t bother trying to do anything until they hit 80%, when the penalties have started racking up at 60%.

So we shall force the issue by giving them tons of tiny machines, as if going to the other extreme ever fixed a goddamned thing.

You don’t want tall. You don’t want wide. You want square.

I saved us almost 10% on hardware by swapping in larger VMs and fewer of them. We don’t have a homogenous workload, but a good fraction is, so larger machines can share a bit of OS cache and statistical clusters of slow events can be load balanced better, improving p95. But at some point the performance improvements we’ve made will take us in the other direction, because then we’ll have too high a fraction of traffic on a single VM. That’s not a problem by itself, but could be half of a production incident.

What is the cause of the failure rate for plaid? Is it their tech stack, or the fact they are fighting with bank's anti-fraud systems constantly to make their fundamentally fragile web scraping architecture work?

This is a curious point, considering that removing the constraint of having to make a single instance work well could lead to relaxed (read: lazy) development practices and therefore suboptimal performance that needs horizontal scaling to remain viable.

That said, i also enjoy sleeping during the nights and not getting paged, so running a single instance monolith makes little sense if that's a goal of mine. Horizontal scaling gives people the ability to deal with short term problems that would otherwise be escalated by just bumping up the resources that are available to the deployment and the failure of individual instances is no longer a showstopper.

In my eyes, the best approach would be to do both - expect everything to work with just a single instance (with proper integration and load testing, possibly chaos engineering just to be sure), but run multiple instances in practice (with the exact same tests as before, plus additional ones for byzantine failures and unreliable networks being written in addition).

Why can't we take best of both worlds - a focus on the quality of engineering, but also as much scalability as needed. Sure, you might disagree, saying that personnel expenses dwarf those needed for just running sub-optimal code on multiple nodes, but my salary is such that you could hire a team of people like me for the cost of 1 developer in the US.

And yet, ignorance about the performance and how optimized code is in many "business centric" domains (e.g. web development vs OS development) seems universal. No one wants to pay for nearly perfect code in 5 years, but rather good enough code in 2 years.

By far the biggest contributors to latency in user present operations was either latency in the bank's responses or "stuff" required to integrate with the long tail of banks. In fact we found that latency below ~5 or so seconds was _too_ fast for users to trust, they didn't feel confident that it was working if it returned too quickly. As time wore on, latency grew but it wasn't due to technical issues, it was bank caused.

> ...not to mention the priceless value of simplicity, a system that can fit in the mind of a single experienced developer...

I've got to say that the bank integrations were _amazing_ to work on. The tooling was incredible and it was super easy to work with. If there was a problem with "a bank", you could spin up the entire bank integration stack very simply, took maybe 10 seconds, less if you kept it up to date, and plunk away.

Your point stands for much of the rest of the stack, pretty complicated.

> ...agility (very slow deployment times)

I think it was on the order of single minutes to reach each environment, I don't recall deployment times being onerous.

> ...reliability (Plaid is notorious for having a ~20% fail rate for requests)

Business decision and the difficulty of building on ever shifting sands (banks change things), the problems are not technical.

There are some more details on this page, including the typical CPU usage on those servers.

Did they aggressively optimize their setup because of hardware costs?

My reasoning is that they should only be broken if you understand the reason for them, how it applies to your situation and the implications (or if nothing's at stake, which can be a great learning experience as to why those recommendations are made).

Fewer, but larger DB servers would typically mean much less maintenance and dev ops headaches.

As others have said, skipping best practices is typically reasoned out really well before proceeding. Not everyone needs every best practice.

I can quite believe that caching SQL results saved time (hardly not a "best practice"). The waters become murkier when you say you can't afford basic abstractions to make things testable.

They provide no statistics on how much time their "optimisations" actually saved. Likely, their optimisations wouldn't have made a difference, as they would have been JIT'd away anyway.

I am very much opinion that if you're writing code without some sort of test, you're writing legacy code.

Engineering teams should aspire to advocate "We run this entire thing on these few resources vs our application runs on 26k nodes" The latter just means we've lost the plot and throwing money to solve a problem. Forgive my sarcasm here, but well done k8s&&containers!

Don't see any reason why FB can't run on the same comparative hardware as SO. I.e. replace the CCSS (commercial closed source software) bits with the FOSS/COSS ones. And when we take it in as a whole, this is much more greener/sustainable to work with.

Also, there is a legend out there that I heard of many moons ago :) ... THERE is a billion dollar company running on a single sql-server and single iis web-server

I think often best practices are based on experience, but they stem from a lost programming zeitgeist that the new practices have replaced.

Before Don't Repeat Yourself became popular, there was a serious problem with code duplication. Now there is a serious phobia of code duplication instead (to the point where it's a problem). DRY was good advice in a world where DRY was an obscure idea, is bad advice in a world where DRY is a popular idea.

If you look at something like Clean Code today, it seems like a bunch of pretty bad advice, but in the context of the late 2000s Enterprise Java, it seemed like a really fresh breeze. We've already adopted the good ideas, and discovered that pushing farther in that direction was a bad idea.

These books and ideas were lauded for a reason, but often that reason isn't there anymore.

A large number of best practices come from Big Corp engineers or organizations. Big corps have thousands of engineers that need to be able to read and contribute to code from many other teams. They have systems that need to be able to handle millions or billions of users.

In situations of massive scale strictly adhering to best practices is worth the investment, since it will make code easier to contribute to (readability/architecture standards), make sure it does what it's supposed to (unit tests), or keeps things performant. But the RoI of following best practices decreases as you move down the spectrum of scale

Complaining about best practice, however, is itself a best practice.

The wise programmer is told about Tao and follows it. The average programmer is told about Tao and searches for it. The foolish programmer is told about Tao and laughs at it.

If it were not for laughter, there would be no Tao.

The highest sounds are hardest to hear. Going forward is a way to retreat. Great talent shows itself late in life. Even a perfect program still has bugs.

> Too many best practices in software engineering are followed to improve developers experience rather than the quality of the end result.

Thus spake the Master Programmer: "Though a program be but three lines long, someday it will have to be maintained."

> Too many best practices in software engineering have no empirical basis.

Thus spake the Master Programmer: "When you have learned to snatch the error code from the trap frame, it will be time for you to leave."

Depends in how tight a loop you do it. If you do a lot of stuff and once in a blue moon you do this - fine.

If you create and destroy objects every time you do basic operation that most of your code consists of - you'll be several times slower.

In my current job we are required to do this:

    Optional.of(user).ifPresent(this::doSomethingWithUser);

    if (user != null) {
        doSomethingWithUser(user);
    }

Most of what we do is these null checks disguised as opionals.

And then people complain performance got worse compared to mainframe assembler.

You see the same thing in Go. Libraries that are zero-allocating are significantly faster than the competition.

Well, they might have tried an unreasonable inversion of control framework (like Spring) and observed that Spring uses reflection on top of reflection and does slow down your code significantly. The solution isn't to abandon IOC (or, god help us, just give up and declare all your variables static/global), it's to abandon IOC frameworks. They're useless and they don't provide any advantage. IOC is a coding style that you can follow without the help of some idiotic "framework", just like ORMs.

and you'll have a few blog posts full of avoiding allocating stuff and how significant difference it made over years

Sometimes I get the feeling that the TDD cargo cult members haven't read the original material.

Yes, it raised quite a few eyebrows when they announced this decision.

I like SO approach, maximum utilization of the hardware and don't try to be fancy, if you don't have to. Less hardware to manage, it's easier to find people who can quickly be on-boarded to the team, easier to debug.

"Programmers waste enormous amounts of time thinking about, or worrying about, the speed of noncritical parts of their programs, and these attempts at efficiency actually have a strong negative impact when debugging and maintenance are considered. We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. Yet we should not pass up our opportunities in that critical 3%."

From some readings of Knuth, I have a feeling that he always think about efficient code.

If you have lots of users hitting your servers, it's worth investing more in efficiency, but if you barely have any, it's much more worth it to write in a convention focusing on maintainability.

Also, don’t forget that we are trying to balance at the top of the often non-reducable complexity mountains, never ever seen before. Compared to some of the programs out there, going to the moon is just some trivial computations. The former may very well not be remotely possible without some structure for our limited human brains.

Of course, if your entire software only consists of trivial glue code, then don't bother with unit tests, I guess.

End-to-end tests are useful, but they take a long time to set up, maintain and run. They have to have a live database to connect to and that database itself has to conform to particular preconditions that can easily be violated even if the code itself is just fine. What I've seen, over and over, is that the end-to-end test fail because the test database instance is under load, or somebody wrote a test that inserted some data but forgot to remove it, or it doesn't work right if two particular tests run at the same time, to the point where they report false negatives so often, everybody just starts ignoring them.

I read that and I feel a surge of sympathy for the new guy they hired to work on this thing. How many MONTHS did it take him to even compile it? How many hours of frustration did he have to spend thinking, "I can't even get this thing to build, they're gonna fire me, I need this job or my kids are gonna starve..."

Sure it's sometimes necessary, but I wish people would stop with the expostfacto justification. It doesn't make your code better if you add it just to test your code.

I had the same thought - if SO says they're fast because they don't use DI, then ok, maybe SO is fast because they don't use DI AND they have an elite group of amazing full-stack programmers who write OS kernels and prove that P != NP over their lunch break for fun. Most programmers fuck things up on a regular basis and don't even realize it until it's broken production for a couple of years for a handful of users who go down a non-standard path every once in a while, and they do it in a way that would have been caught if the code had been run even one time, but they can't run the code, because somebody else insisted on using a hundred static (i.e. global) variables.

[1] How data gets to the reporting database is implementation specific. Simplest case is run a nightly job to copy the data over. More complex cases use things like data lakes and/or event buses where all interested parties subscribe to data they want. I'm sure there are many others.

But it didn't make sense to me either. I don't know anything about C# but I assume there's some way to use compile time flags to assign static values. Seems like they should be able to mock that way.

Also, consider that a lot of the lag you experience in the web today is actually caused client-side by excessive use of JS.