All of us test in production all the time (2019)

Glenford Myers in "The Art of Software Testing" said: "Testing is the process of executing a program with the intent of finding errors."

The preface to this is even more enlightening:

"When you test a program, you want to add some value to it. Adding value through testing means raising the quality or reliability of the program. Raising the reliability of the program means finding and removing errors.

Therefore, don’t test a program to show that it works; rather, you should start with the assumption that the program contains errors (a valid assumption for almost any program) and then test the program to find as many of the errors as possible."

> They are convinced that unit tests are unnecessary extra work for their project but they don’t realize they are losing time by having to manually test and research changes.

If people are finding tests unnecessary extra work, maybe they're right for their project.

Most jobs are just "ship feature asap, try not to break anything, repeat". Most managers don't care about code quality as much as speed of delivery. Most coders don't have much "skin in the game" and can always get another job when the project fails.

Writing tests is a sign that you know what you're building, you have a methodology for building it, and you have a commitment to keeping it working in the future. That's simply not the requirements for most programming work, and it can be a disservice to both the coders and the management to push for testing that won't help achieve the business goals. If a business needs to iterate fast and reliably, write tests. If a business needs to iterate fast and is ok with breaking things, skip the tests and have a rollback plan instead.

Self-documenting code is all code. The only thing that speaks to what a piece of code does is the code. Comments aren’t syntax, so they don’t help you there. I’ve never seen a codebase that does a good job of documenting the system architecture, expected usage patterns, and why the code is built the way it is. And I don’t know that having all that be inline to the code would be helpful anyway.

I find it far less error prone to just read the code and determine its purpose and allowable modifications from that. I am thinking about what a citation or footnote system might look like for code now though. That has the potential to provide context on-demand without needing to separate out all the pieces of the code into little silos that are hard to piece together.

Maybe "Once you deploy, you aren’t testing code anymore, you’re testing systems"? Trying to lift something out of the article instead of writing a new title.

Your self-description aligns very much with my own. Feel like we might get along well. Would love to chat. Shoot me a PM if you're willing.

Cheers, mate. There definitely is a place for people like us in the world.

Feel your pain <3 You can tell it's been a few years now but I have worked at Big Corp Inc. before.

I know what it's like to ask someone in charge of 8k people to personally approve my server permission escalation even though he's never met me, doesn't know anything about the IT systems, has never visited the site I work at or probably even been in the country I reside.

Anyway, a good thing on this front now is that (speaking from Azure exp.), you can dump logs directly to blob storage (s3, whatever google call their data storage). Then you only need that permission.

As for the non-UTC servers... -_- but if it's a problem, you can always append the tz info to the log date format.

As QA I think testing, test environments, test ressources should be first class concepts in software. Building whatever "modern" architecture twice, and simulating traffic is not easy but possible, at least to an extent. You need to get that extra license for any piece of software for testing during procurement, and provide means to create test ressources, like typically test users as needed. Sounds trivial, but is often so complicated (banks, insurances...) that it should have been considered early during design.

So, I wouldn't use the word "lazy", but we could do better.

To be useful? That seems to be an extraordinary low bar to hurdle for a QA environment. I imagine most clear that bar easily. I know mine does.

Can you explain that more? Aren’t most of the complex modern tools specifically designed to automate provisioning servers and deploying code?

Not sure where you are working but not a lot of people work at/on a huge code base.

Most codebases are build and operated by 1-3 teams and if you are not able to reproduce your prod env as a test env, your architecture is broken.

I have seen google and Facebook first hand. The practice is widespread, but the acceptance of "let's test it in prod" as a real stage in software development varies. Some people look at you like you are crazy person that should not be trusted, some people acknowledge and know it to be true.

Feature testing is one thing, but you can go a/b testing a whole binary between new and old version to catch interesting bugs too.

True but imho its somewhat of a fan-out or at the very least knock-on effect. The explosion in devops complexity is a necessary byproduct of the explosion of projects that adopt new languages, runtimes, datastores, etc. No one ever fully deletes and retires the old system, so the overall environmental complexity grows exponentially. The tools it takes to solve those exponentially larger problems (k8s) look ridiculous when you consider them in the context of "a rails app with some node stuff". But they make perfect sense when you say "every quarter or two, for a decade, a dev team has adopted a new component or two into the stack. less than three of which have ever been deleted."

Yes, but there at least is some genuine innovation going on. The backend frameworks are just churn for solving the same boring and lucrative business problems in new, fashionable ways.

He mentioned that he had a job doing that, but the company moved away from the framework without technical justification.

He could find another, but there would be no guarantee that the same thing wouldn't happen again.

My experience is that nobody actually does things The Rails Way, which is where most of the benefits come in. No, they’ll start out in Rails such that their underlying data model evolves into a huge mess (because Rails guards against most problems in the code layer, so the data model problems don’t really appear as bugs yet), and then some resume-driven developer will push for serverless or microservices to solve scalability problems (when the actual bottleneck is the database and the data model, not Rails), and now you’re left managing this Frankensteinish “rails” SPA beast that could be rewritten as a proper database schema and a 5K LOC Rails app, but there’s too much change control process to actually achieve that.

After it happened twice, I just gave in to the inevitable and decided to value jobs based on other factors.

> Sometimes what works now won't be competitive in the near future.

What do you mean by that? I can understand in terms of hiring devs, this is true, but any other reason I would be curious about. Because I work with/in many companies, I see a lot of different tech and competitive is usually just what the team knows best. So unless you are making a new team, I cannot phantom why you would switch.

Java with Servlets/Spring and C# with asp.net have been competitive for almost 2 decades; maybe some people can shave off a few minutes here and there but in the grand scheme of a serious company with a serious project (500k loc+) it is not going to matter. There are exceptions as for instance machine learning but those gaps close and this was about web frameworks.

Take a new site for example

Its just text and pictures on a web page. The new york times re-invented a database out of kafka to power theirs (mind bogglingly stupid)

The Guardian re-wrote their CMS at least 4 times, at a cost of well over 50 million, if not more. Thats before they got to re-doing the layout.

The FT spent 6(!) years rebuilding their entire stack, (150 tech staff for 6 years. a good percentage contractors. > £70million)

All of it is just fucking text on a fucking web page. All that effort to add fancy gizmos for hosting, optimising for framework x, which has a half life of 6 months, adding animation to drop downs. Just a spectacular waste. They are still re-platforming every 9 months, each of the 5 major teams.

All of that money could have been put into content, advertising for customers, collaborations & events.

Most news sites can get away with a static, highly cached system (I mean look at the Daily mail, the page is slow, static and looks like shit, yet its the biggest new paper site online)

So form a business perceptive, its spaffing money up the wall for no real gain.

My experience has been the opposite. The Rails ecosystem makes updates straight forward and is very well documented. Been running various Rails apps for almost a decade now.

This is the primary reason why I love the .NET stack so much. Not only is a security update almost universally a non-event, but so are major framework revisions.

Updating .NET applications for security concerns in many cases boils down to letting windows update do the thing it should already be doing automatically. In a much smaller portion of cases for .NET Core, a security update is a matter of rebuilding a self-contained deployment and pushing it out to production. Neither of these cases requires editing any code or configuration.

Only in the transition from .NET Framework to .NET Core did we find any appreciable difficulty, and even then it was still within the realm of reasonable. Most of the pain here boiled down to usages of System.Drawing and DirectoryServices, both of which have alternatives that are supported on all platforms. We opted to use the compatibility pack instead, but would be OK if that was removed at some point and had to replace with the other implementations.

We have lots of sourcecode that has withstood the test of time because of these technology choices. Many files are completely unchanged since .NET Framework 4.5 was released and now running flawlessly on top of .NET Core 3.1. This is the only reason we are still in business right now.

I ended my journey with NodeJS (and AngularJS on the same day) when I attempted to update all of my packages after just a month of inactivity. Seeing the resulting pile of nonsensical dependency graph trash in my console may have radicalized me into using more "enterprisey" solutions when writing software that I might get put on pager duty for.

I loathe NodeJS with the passion of a thousand suns. When Rails projects churn, at least you can rely on the stability of Ruby. With Node, the number of times I had to work around inconsistencies with the framework and even language were maddening. Babel is a horrible horrible thing, and Javascript was never meant to run on a server.

Maybe it's better now. But I won't ever do serious work again with the stack so I'll never find out.

I didn't say that any of this was simple or non-trivial. Again, it depends on your priorities and your values as well as your company culture. In fact, I specifically said that testing systems are hard and provided examples of how hard systems are to test. Do you think that Cassandra is a tightly controlled API with a small configuration space?

You seem to feel like close enough is good enough. And that's the cause of the problem I'm trying to address here. Does it really matter if you don't get a notification when someone messages you on Facebook? Or if you get two notifications? Is that particular problem worth testing every possible Kafka configuration? I think that you are saying is no, it doesn't matter.

But I'm arguing a different point. I'm not arguing about whether the testability of any individual feature is important. For obvious reasons: some features really just aren't that important. But not being able to do that, and actively choosing not to understand that system is a symptom of a far deeper problem. When a company makes the choice you have just described, the company has decided to accept that they can't, won't, and will never fully understand their own systems. It's often not a conscious decision, it's a decision made by habit, policy, and culture, which is what's so subversive about it. People don't make big-picture decisions to intentionally have a system that is unknowable/untestable. People make small decisions just like the ones you are talking about that make systems that way. And it's the practice of letting lots of disconnected people make the small decisions of what does and doesn't matter, what is and isn't worth it that destroys systems.

Systems are hard, and I agree with that, but systems are made even more so by bad process.

The being old analogy didn't seem to resonate with you, which is fine. But let me ask you a question about a system.

You have a database. It gets backed up every night. Or maybe every hour. Your job is to take snapshots and store them because that's what you're supposed to do. Yeah, I know, that should be or can be automated. Whatever.

The big picture system and purpose is that you are supposed to be able to recover from a hardware failure/data loss. But that's not your problem. Your problem is that you have to back up the database manually every day. The data team only tests restoring backups from dev to dev instead of prod to dev. Because reasons. Because it's hard.

That type of backup system checks all the boxes you're supposed to check when you get audited. Or at least enough to get through it. But when you really need to understand the system, it fails for all kinds of reasons and people are sitting around looking at each other saying, "well I did what I was supposed to do."

Individuals sitting around making isolated, disconnected decisions like the ones you're talking about (i.e., it just isn't worth it; it's not feasible; it's hard) compound in organizations and create the kinds of systems you don't want to deal with. You're making your own hell here. You seemed to have missed that key point in my earlier comment.

Laziness is a good trait in an individual programmer. But laziness is the absolute death of an organization. Agile is really just distributed, organizational laziness. That's what creates horrible, unknowable systems.

Conflating test/experiment with what the original article claimed to be talking about (and then later walked back) is borderline disingenuous. No one is talking about A/B testing or intentional experiments.

The article is talking about rolling the dice in production deployments and claiming that's fine and something to be proud of. It isn't fine, and it's not something to be proud of. She's the CEO. She should fix her company instead of being proud of how bad it is.

A lot of what we're talking about here is a matter of perspective. And that is the problem I'm taking to task both with you and with the article.

Thats a weird point you are making.

Yes i would test basic / standard customizations a customer would do.

I would test the customization system itself.

If it is to complex, your cusotmiztaion system will bring you much more issues later on.

I would argue that using the right words is important here. “On call” typically refers to a specific thing, where if something breaks that person on call is, well, called (or paged, etc.) to fix it when it happens, even at 2am. This is how I took the meaning of that paragraph as well.

If the author meant that code ownership is important, or that the engineer(s) who wrote the code are responsible, that message could have been conveyed by saying, “the persons who wrote the code should test it once it’s deployed and are responsible for fixing if it breaks or is broken when deployed.” This is much clearer and doesn’t use terms that could be understood incorrectly.

I'm always happy to help some poor SRE in the middle of the night, and I once even drove to the office in a rainy Sunday, in the middle of my vacation, to access IP-restricted stuff because a support intern messaged me on Instagram.

...but with that said: I'm glad I only worked in countries where work is properly regulated and "on call" means "I'm getting fucking paid every cent for each hour I _must_ answer that goddamn phone". Which in practice means there's no PagerDuty.

The unpaid on-call culture is bullshit. The company can either pay me or go fuck itself.

There's more going on and worth exploring if engineers are so unattached to outcomes they pick technology in a vacuum. Pager duty is a heavy stick - it's important not to avoid root cause analysis. So if your engineers are making bad choices - what's really going on?

That assumes engineers are even empowered to make technology choices. At many companies they are not (whether by dint of organizational structure or the roadmap not allowing a major technology shift from whatever "shitpile" you and your team have inherited).

Having clear escalation strategies (and knowing when escalation to the original engineers behind a project is even appropriate) is often lacking. I wouldn't want to call engineers in at 3am for a problem that can be fixed by following a documented devops process. Plus - what happens when the engineer you need to reach is unavailable? They are sick, or don't wake up, or their phone died?

What happens when business pressure says "we're ok with calling engineers twice a week as long as the roadmap moves"?

"You built it you're on call" is a fragile way to handle problems in more ways than one.

Which isn't to say there shouldn't be shared responsibility. Of course there should. But responsibility without power is toxic. At the very least it increases flight risk - but in practice often has a far wider reaching deleterious effect than just that.

And then that one person gets hit by a bus and you go out of business. Very-interconnected large-scale systems rarely have failure modes that are as simple as something the dev did/didn't do.

Yes, I think the headline must reflect the content of the article, otherwise it confuses me and make me write false comments and wastes my time.

So much the worse.

> willfully misinterpreting

Often, when someone looks like they are “willfully misinterpreting” something, they are actually presenting uncomfortable implications or experiences of how a phrase is put into practice. Right or wrong, it is irritating to listen to.

Is it worth enduring your irritation in order to see if someone is meaningfully correct?

—————

> Don’t toss “untested” code.

“tested” is not a binary state. Code could always be better-tested —- otherwise why would you have hotfixes?

The problem is that this article is misusing the catchphrase and now people are discussing the misuse of the meme instead of the article's content.

> The opinion of the article is that the catchphrase has become an excuse to build poor observability and in-prod testing tools, and we should be better about that.

Is that a problem that actually exists though? I've certainly never encountered it, as the types of teams the are strict about good pre-release testing are also the types of teams that are big on observability.