The Servers Are Burning

> think of the piece of software's entire input domain, carve it up into a set of equivalence classes, and then determine what the expected behavior should be for a piece of input from each of those classes

This is an extremely good way of putting it. Doing some of this at the requirements stage if possible can be very helpful too, because you can detect nonsensical or conflicting requirements before you even write any code.

These are great techniques and should be done more. But I'll add a caveat: they don't cover all the ways software can fail and we probably shouldn't pretend they do.

The environment in which the software runs is also an input, and you often don't get to pick that. Consider new browser releases, browser addons, and browser bugs.

Performance is a global property of a system that crosses most abstraction boundaries. Outside hard real-time systems, we get few performance guarantees.

Some domains are simple on one level and very complicated on another. The set of all <=280 character strings may seem simple as far as the data goes, but setting policy for Twitter is not.

Still, being able to decompose problems and treat bugs as either functionality or performance (and not a complicated mixture of both) is pretty helpful.

> In order to write effective tests, a programmer has to think of the piece of software's entire input domain, carve it up into a set of equivalence classes, and then determine what the expected behavior should be for a piece of input from each of those classes.

OK, so let's revise the statement to "in order to write effective tests, you must know how your software should behave in the face of all kinds of inputs".

In my mind, that's an equivalent, or nearly equivalent* proposition. It's still above and beyond the capabilities or available resources of people developing very complex software. That's not excusing anyone. Just saying that your correction isn't that different from the original statement.

* Sure, "someone unplugged the computer while it was running my code" is a failure mode that's not a function of (what most people would consider to be) the input domain. Genuinely external failures like that, while damaging, are rare. What you usually end up defending against are the downstream results of those failures (i.e. "a network partition meant that half of my input dataset was null pointers").

> has to think of the piece of software's entire input domain

If you're a startup this is not clear and changes frequently. Practicing TDD in this environment has very real downsides that exist less in mature companies with clearer understanding of business requirements.

My philosophy is a little different. It grew out of writing a thesis on software quality in a post grad course. We studied lots and lots of ways to improve software quality. I can't count the number of papers I read on the subject. The first paper you read is all gung-ho about their particular method of course so you think you are done - you've found the holly grail. Then you find the second paper is equally gung-ho about an entirely different method. Eventually you become jaded by what is really opinions on the effectiveness of the technique their paper explores (dare I say delivered with the almost religious zeal of a true believer) disguised by blizzard of equations, tables and lemma's that take hours to decipher.

Being a post grad thing I had been writing software for many years before reading these bloody things. I did learn a whole pile of new techniques (my thesis got a high distinction with the comment "you sure know a lot about improving software quality"), but for most of it I was none the wiser on what I should apply to my everyday work, and that was the point of doing the post grad in the first place. So I changed tack, and started reading papers that reports the results in productivity. It's really hard to measure of course, and these papers were very thin on the ground and their figures involved a lot of hand waving. Even so I was pretty confident in the end the _only_ technique that had a significant effect on productivity was unit testing.

Note that considering quality and productivity together is very different from looking at quality alone. The most effective way of reducing defects per line is the method is probably the one developed by IBM when developing code for NASA's early space program. It relies on heavily on formal inspection. In that system programmers weren't allowed to compile their own code - and compile errors where counted in their defect rates. It was spectacularly effective at developing software with low defect rates. Programmers don't usually think of their profession of being shining example of the quality engineers of all types can achieve when they put there minds to it - but this code was just that. 100's of thousands of lines of real time code, not a single failure in flight. Contrast that to the efforts from other engineering disciplines like the O rings, tiles falling off, yada, yada.

But most of us don't develop code that must work on first deployment, with catastrophic life and death billion dollar consequences if it doesn't. In the trenches where most of us spend out working lives unit testing delivers more bang per buck then any other sort of quality measure. The reason is simple - it automates testing that would otherwise have to be done manually. Automates it so effectively that it's almost free. If you are wondering how that could have effect net productivity just read the story - the entire team stops work for days to rescue a burning site, people too scared to deploy code. But that's probably dwarfed by the lost sales while the site was down.

Still, writing tests take time. Writing too few doesn't get you the quality boost. Writing too many slams productivity. How many tests to write is a puzzle. Your post about all the tests you could write to test everything is certainly pertinent - but if you go down that rabbit hole too far you will spend all your time writing tests.

My way of deciding that point isn't like yours. In the end I decided the most important thing an easy way to add test for every line of code. The only proof I can believe is a coverage tool reporting 100% code coverage (or better yet 100% branch coverage if your tool supports it). Handing waving to explain the rest could be right, but to verify it requires a 2nd person to read code _every time it is modified_. Remember we are on about maximising productivity here, so creating manual work like this for every release is really hard to justify. Thus the first step is to write tests for your code until you have 100% code coverage.

But it goes further - it must be really easy to find what test covers a particular line of code. This usually means the same line will be tested multiple times - once by the test that targets it, and many other times by tests that happen to use it. Someone programmers view this is inefficient, and it is in a way, but you will see later there is a reason to spend this extra effort. Sadly it is time consuming to verify this has been done in every case as you would have to check every line of the code being tested, but it is amenable to auditing: if you select 10 random lines and they are all good the odds another random line doesn't have an easy to find test is 0.1024%.

As anyone who has written a lot of tests knows, this insistence on 100% test coverage has flow on effects to the code being tested. Arranging code so it is easy to spot how you get to the line you want to test can be the most tedious step. It's not unusual for the "writing tests" step to force the code being tested to be refactored for visibility. It is this "refactor your code so it is easy for others that code later to add more tests" that is real goal of the 100% test coverage condition.

Once they've hit 100% then, and only then, can the programmer take into consideration the sorts of things you describe. It's difficult to check if they have done that of course - at doing such checks really requires 2 people to think up the tests required, which doubles the work. But perhaps it doesn't matter as at lot will happen anyway if your programmers are diligent in developing test for every subsequent bug you hit. Here diligent means thinking about all the other places this bug may arise, and testing it there too. Relying on this is effectively leveraging off the all the systems testing that follows to produce the sort of tests you describe automagically.

Adding these new tests is easy of course because you already have 100% test coverage, and its obvious what code tests each line. You can ensure adding these new tests happens by insisting every programmer report every bug they find in their own library or someone else's into a bug database, and later record the test.

Doing it this way does not guarantee your code will have a 0% defect rate. But then nothing, not even what NASA did can do that. What it does guarantee is over time your defect rate will drop, and it will do that with minimal extra work. I find the "extra work" bit is important, because when you are in the "creating the new shiny" mode finding the energy and money to do the work isn't so hard. Hopes should be high is the next big thing that hits the jackpot. But once you are in maintenance mode reality has hit home. Every new addition has to justify every cent spent on it. I suspect the that is because existing KPI's are in place, everybody has a reasonable feel for the impact a change might make - and most changes are small increments. Yet over the years most development happens via the cumulative weight of these small increments. Ensuring the cost of maintaining 100% test coverage during these small incremental changes is minimal is only hope you have of ensuring it's done.

Yup, classic.

"...no time/resources to do it right in the first place, but plenty of time/resources to fix it when the customers complain..."

I always thought it better to find the bugs in-house before shipping, but so many others don't see it...

This is a smart way to think of testing and will discover bugs when your understanding of the code is the most lucid.

But it does require some nuance. Unfortunately, some people think writing tests or test harnesses are wasteful because they think of shops that insist on 90%+ coverage and have test suites that are easily 5-10x the size of the application's codebase.... so they decide to forgo them completely. When in reality, the answer is probably something like: you have at least 25% of your codebase that would greatly benefit from testing, even if they're just "smoke tests"

Don’t you take this bit about out of context? If the premise the article builds around faulty tests is correct, then stuff like unit-testing is mostly a waste of time since you’ll still need to fix things when the servers are burning.

We’ve done TDD, projects with coverage only vital parts and no software-tests where I work and there is no difference in production on smaller projects. Maybe we’re terrible at writing tests, but if so, then it’s still something we have to deal with.

So I think it’s hard to outline the correct amount of testing for every case.

I don’t think I would want to work on a major system with multiple contributors if there weren’t automatic tests, so it’s not like I’m against testing either, I just don’t think it’s always a holy grail.

Do testing frameworks normally catch memory leaks?

Skipping tests generally doesn't allow for creating a working MVP any faster. It's just an illusion.

Easy to say, but unfortunately rarely seen in the wild, depending on the experience of the project manager. Once that product is live and climbing, no owner wants to hear talks about slowing down for reasons of better test coverage or refactoring.

Yes to the last! Whenever bughunting my first step is to reproduce the issue in a failing test. This has the benefit of me being sure I know what is failing and how, but also crystallizes then knowledge and experience into the repo for future engineers (or future me) to know the most common/practical ways things break in the real world. It is also marginally useful for catching regressions, if the test is clever/general enough.

The big problem I've noticed when reading unit tests in various projects is that they frequently test the implementation rather than the outcome:

https://softwareengineering.stackexchange.com/a/304910/24932

I'm not a great programmer, I can't usually write code that does exactly what I expect the first time, so I use unit tests with a debugger in my IDE to run small portions of my new code until it works the way I envisioned it before I started writing it.

This is a lot faster than my old method of writing code that doesn't quite work, and running the whole program over and over with small changes and print statements each time.

When you have a well thought-out external specification, it really gives you lots of good material from which to derive all the right tests while still treating your implementation as a black box.

Unless you are implementing an established RFC or some other well-defined system, such a tight specification rarely exists in a software setting, and is very difficult to effectively design a priori. For this reason, techniques like test-driven development have arisen to try to make the programmer repeatedly and consistently assert their understanding of the specification via unit tests, while evolving those assumptions as needs and understandings change.

When I was doing QA at work, I never used the official tools to generate the data files required for testing the application, but wrote my own. This brought to light a ton of issues and unwritten assumptions (both in my code, and in the application).

Without automated deploys, how do you know that what you deploy is actually what you tested?

I think just SSHing to box and doing deploy is not function of users as you noticed but function of how many devs are working on project. For 1-2 devs manual deploy might be ok, when 3rd comes in, it is time to get rid of any magic that might happen when deploying by hand. Because you know someone will do something special and not document that, when he updates automated deploy he does not have to document or talk with other guys, it is documented in deploy tool.

All I know is that if enough people rely on the same dependency for long enough, the chance of encountering large bugs becomes smaller and smaller. Especially if the dependency has a stable interface.

Good software gets better the more it gets used and abused, so I tend to stay away from small dependencies, they're usually not worth the time.

While I generally agree that functional programming does help immensely to make secure software, in this case, the function wasn't pure. It was meant to return data that didn't exist from a database, but the error was silenced. Also, the reason why the servers were "burning" was because this somehow resulted in a memory leak that consumed all the memory available to the system, which is why they also had trouble pushing fixes. Even in a pure function in Haskell, memory leaks are possible and the same scenario could've resulted.

While I can understand some of the benefits of functional programming within a single sub-system, or within academic research, I struggle to understand how pure functional programming can address the need for the large amounts of state information and abstraction layers required for a large scale system. I work with systems where (large/numerous) sub components require very specialized domain knowledge to understand internally, so multiple levels of black-box abstraction are required to integrate/interact with the sub components. Additionally, both the client systems and the sub components may have huge amounts of persistent state information.

Could you suggest any resources that might help me understand how this can be accomplished in a pure functional paradigm? Something that assumes only a casual understanding of functional programming would be especially helpful.

A slight change in performance characteristics can also lead to cascading failures. Pure functions don't help with that. That aside, even functional programs need to have side effects (other than heating up the CPU). Referential transparency doesn't help when the state of your monad is a little bit weird and that causes some corner of your code to kill the DB.

Anecdoctally this happened to me the other day in my Haskell backend: had a recursive function (in the IO monad) that every ten second would take a connection of my connection pool (and since it was recursive) would not give it back to the pool. Of course it was a stupid bug, but it was hard to figure out because it would take a while before it got to that point and once it was there every call to the backend would block then timeout. I like functional programming, and yes: it was in the IO monad so not pure, but I think most applications ends up in some stateful monad even if your stack is built for functional stuff... This was 'not at scale' but it as a Haskell project it is fairly big. Point being: yes, pure apps will not havr this predicament, but even with a stack built for pureness I can see it happen quite often. Running software is messy...

> If a function is "referentially transparent" or "pure" it can't do anything unexpected.

Sure it can; at a minimum all functions consume compute resources. A tiny change in a "pure" JavaScript function could lead to cascading deoptimizations that destroys performance and compromises service availability. For example, changing 0.0 to 0 could do it.

By definition, pure functions can't read from or write to disk, can't read from or write to the database, and can't read from or write to a socket.

If you aren't able to do any of those things, there is no "at scale" - you're trapped in a single process on a single computer, spinning away but not able to communicate with anyone else.

Pure functional programming helps within an individual component by letting you quarantine the statefulness at defined areas (e.g., in Haskell, at the program's entry point), but it doesn't let you make it disappear. Sooner or later, you're gonna have to deal with it.

At scale, you've got tens or hundreds or thousands of different processes, and each of them has its own statefulness that it ultimately has to deal with. When you're looking at the big picture like that, I'm not sure it makes sense to worry about whether it's been quarantined, Haskell-style, or not. It's still there, and, in the big picture, it's still distributed more-or-less homogeneously throughout the system. Even a process that doesn't use disk or talk to the DB or anything like that is still not referentially transparent, because it's still subject to side effects by virtue of the network being unreliable, the OOM killer being unpredictable, and all that fun stuff.

I was immediately struck by the same set of thoughts. I don't know whether to be appalled or just ... sad.

An outrageously unprofessional software process that created a product worth $50M!

https://techcrunch.com/2011/02/02/match-com-acquires-online-...

I washed out of their interview process when I was fresh out of high school for not knowing bit twiddling hacks[1] in the technical phone screen with one of their engineers.

After reading this, I can't help but feel like I dodged a bullet there.

[1] https://graphics.stanford.edu/~seander/bithacks.html

It's a free dating service. They have a responsibility to respect their users' data privacy, but the rest kind of doesn't matter. Eventual stability is more profitable than eventual release if you're not a Bank or making self driving cars.

I'm sure the OkCupid folk are nice people but it was:

1) founded by mathematicians and an academic, and 2) written in C++, with a custom templating language

Even giving them the benefit of the doubt that the core matching algorithm warranted C++, the whole site really shouldn't have been built that way.

I'm really not surprised there's poor technical decision-making there.

Outrageously unprofessional as it may be, they probably ended up laughing on their way to the bank when they got purchased by match.com.

> That's an outrageously unprofessional software process.

People who call me unprofessional forget that being a professional just means I'm paid to do it, it's not a pass time.

You have to judge it in context. If it's entirely a senior team then the value of code reviews is greatly diminished. You rarely catch a bug in a bug review when you're reviewing a Senior software engineer's code. those bugs aren't caught until you do testing.

it worked well enough until they could do something better. Never let perfect be the enemy of good.

In my experience this ‘outrageously unprofessional‘ process is very normal. Most companies are not software companies and they are not going to pay for whatever is not adding to the bottomline. And even in software companies that ofcourse happens. It is unfortunate but the shock and awe are a bit over the top; when I talk with partners (ctos is large corps; banks, insurance, retail), they generally glaze over and mumble about their dreams of doing tests and code reviews but unfortunately it is not possible ‘at the moment’.

> However, the moral of the story was essentially: "software is so complicated that it is bound it break, so you have to be good at fixing it". While that is certainly true, I think that developers have a responsibility to use whatever tools they can to write high-quality software.

I actually disagree that we have a responsibility to write high-quality software. As engineers our job is to create software that is good enough for the task at hand and that can be improved later.

Sometimes that means that you invest in 100% coverage plus integration and system-level testing because "good enough" means that it has to never fail. Sometimes that means you don't spend much time automating tests and instead do informal development testing before you check something in.

It's entirely up to your judgement based on what your code is going to be used for, and I don't think it was inappropriate for OkCupid's engineering staff to avoid testing in this case. After all, their business was really successful through the period where the error happened.

That moral seems related to a meme floating around that Mean Time To Recovery is more important than Mean Time Between Failures. I think it came from, or through, John Allspaw:

https://www.kitchensoap.com/2010/11/07/mttr-mtbf-for-most-ty...

It's popular with a lot of people i know, because it's one of those ideas that's counterintuitive but believable, and so makes you feel really smart for knowing it.

Programming would be trivial if things were so simple.

The reality is, that "either it creates revenue by being what users want", or it enables creating more of that later on, or prevents the speed of creating more of that from dropping, or prevents the whole thing from catastrophic errors (like in this article), ...

Those are the trade-offs engineers have to consider daily.

In a surprising turn of events, the user showed they did care.

> “@okcupid how am I supposed to get my daily dose of crushing rejection and emotional humiliation if your site is down????”

I just turned off CSS in Firefox (View menu -> Page Style -> No Style) and the fixed sidebar was no more and the odd scroll bar behavior was gone.