There are other techniques which can give similar confidence, but tests are the easiest one.
However, many in the industry forget that this is the underlying reasoning, as seen just two days ago here on this site. Read through the top-rated comments on this post: https://news.ycombinator.com/item?id=13119138
They describe an emergency situation where a single "3" needed to be changed to "4" ASAP or people would lose their jobs, and everyone's applauding the gatekeepers who insisted on significant refactoring and the creation of additional tests before the change could be approved.
I agree with those who say those improvements should maybe have been demanded immediately after the fire was out, but those who would have delayed the firefighting out of blind allegiance to the rules seemed, to me, to have forgotten that the rules are there to serve the programmers (particularly, their ability to quickly ship working code), and not the other way around.
A rule that's failing to do that should be changed or ignored.
But if people's jobs were truly on the line, I'm inclined to agree with the "screw it, push it through" approach.
If you are hired as a programmer, then yes, just do whatever we ask of you.
But if you are hired as an engineer, everything the business asks of you comes with an implicit: "and make sure it's done in a proper way that won't break anything, or slow us down, or cost us too much, or limit our ability to gain a competitive edge."
You don't just change a 3 to a 4 because the CEO wants you to. You have to make sure the change doesn't come with unforseen impact that would put the company at risk, and you have to make the change in a similar way. That's what the CEO expects also. If you did the change, and it had caused impact to the business, that you had not pointed out, and for which the business believe is more harmful then having waited a few more days, you and only you are to blame, and you will be. You can't say, but CEO told me, you're the engineer, you're the person they hired to know this stuff and prevent these issues from happening, not the CEO.
But if Scotty delivers on time at the cost of overloading an expensive piece of equipment that, after the battle is won, requires a week in drydock to replace, that's probably a successful execution of exactly the kind of call a senior engineering manager is expected to make.
In your example, Scotty knew what he was doing though. He didn't say, wow, what Kirk wants me to do could kill twenty redshirts in the process, I'll just take the gamble since he seems to want me to. He knew exactly the impact, and made it knowing he would easily be able to contain it.
Which is often not the case in Software and in practice. You have to do something to know the impact, because most problem we solve is always new. Its not something we did many times before. If that variable was often changed, then it would be completely different, because he'd known, just like Scotty, that its something they can do. In that case you can make the choice to say, lets change it, and later handle the tech dept of the less maintainable code.
Also, in software, its almost never the case that people can't wait a few more days.
In the example with the line of code that took 6 days, there was no dramatic emergency in production that required cutting corners. If it had been an emergency, of course the code refactoring demanded in code review should have been postponed; those changes increased the impact of the change, and therefore the resting requirements.
And if it really is an emergence that requires people to drop what they're doing, then someone with sufficient authority should be directly involved in order to override all the usual procedures.
But you don't just drop all procedures just because somebody claims somebody said something. That would be dangerously irresponsible.
If the CEO lacks the understanding of the technical consequences of a change that may blow up the company, the engineer should make the decision. If the engineer lacks understanding of the business consequences of not making the change - like losing an important client, or suffering a wave of negative PR, or facing a lawsuit - then the CEO should make it. Ideally, both sides should be communicating these consequences so that both of them have all the relevant information and would ideally make the same decision. Then the decisions can get made at the lowest level that has all this information, and the CEO doesn't have to get involved.
In practice, there are many cases where the CEO can't communicate all of the relevant business realities, eg. if you're facing a lawsuit if you don't make a change, it's often better not to worry the rest of the staff or make them subject to depositions, and simply to ensure that the change gets made. That's why the CEO is the decider by default in organizations, and also why it's usually expected that employees will obey a direct order from the CEO or be fired.
This is a distinction that is a very thin line and most people with "engineer" in their title would not sign up for.
It all my years (15) of professional experience, I've only worked under one PE, an Electrical Engineer.
In some states you literally can not have "engineer" in your job title unless you have a PE certificate/accreditation/whatever.
In my opinion, computer science and engineering is not about just making the code you're told, it's about questioning whether that code needs to be done in the first place, and if so, how.
It still depends.
David: It's for Philip. It we don't do this right away, we'll have to have a layoff.
and
Judy: OK, then I'll fill out that section myself and put this on the fast track. ----- 2 days later. ----- David: What's the status of 129281?
"It we don't do this right away, we'll have to have a layoff" and "2 days later", making an impression of this is an "a day or two" task, not a "fire/emergency"
I don't think if this change finished in 2 days anyone will be unhappy. But if you took this to production, and somehow failed, everyone would blame QA/testing
In theory, could informed, intelligent, rational actors without ulterior motive do so, sure. I'd sleep on a couch and eat ramen for the chance to be part of such a team, but I haven't met them.
The engineer did their due diligence, wrote tests to make sure it had the desired behaviour, and got it done in minimal time. Clearing technical debt in old modules should be done, I agree, but not while trying to put out fires. It adds considerable risk to a change which should not have any impact except for the request.
"We can do that, but it will take us 6 days, otherwise we risk taking the plant down and aggravating the issue."
I wonder if the CEO would have just said ok thanks.
In my experience that's the case. The engineer in that link got himself in a bad spot, because he didn't know what was involved for the change when he communicated his estimate. And most of his back and forth that slowed him down would have been avoided had he known beforehand how to properly do it. Even with everyone's feedback it sounds like a 1 day code change. That seems to me like the reason the change was slow is more ramp up time for him working on a code base he doesn't normally work on.
Then the boss can make an informed decision.
The important thing here is to provide the risk-assessed alternative in writing. This covers the asses of both sides! If something blows up and management was not warned about the possibility, they're well within their rights to rake the engineers over the coals for it. But if engineers warn management of the potential consequences and management chooses to take the risk, if it blows up, you have your CYA right there - they were warned in writing.
If you can't trust your manager to take responsibility for the decision, then it's better to make a decision you're going to be held accountable for than to let your manager make the decision and then hold you accountable.
I've seen other engineers in the position where they give a risk analysis and warning in writing, and then shit hits the fan and they get fired. Maybe it doesn't happen immediately, maybe the reason they were fired isn't explicit, but the change in the manager's attitude toward the engineer traces back to when they did what the manager said.
There are also mangers who won't follow through on the tech debt part because they don't trust their engineers even if their engineers are trustworthy. When they discover that they can bypass testing by pulling the emergency lever, they'll start pulling it all the time because they see it as a way to get what those lazy engineers to do their jobs faster. And when tech debt catches up with you, bugs abound, and development slows to a crawl, the engineers get blamed.
Maybe you have a boss you can trust to take responsibility for their risky decisions. Maybe your boss trusts you when you say that paying down tech debt is necessary. But maybe your boss and their boss don't have the same relationship, and the shit rolls down hill.
Yes, I want to work in a trusting environment where my interests are aligned with doing what's best for my company, but an at-will employment capitalist economy doesn't always work that way. It's every man for himself at a fundamental level, and exceptions to that are too rare to make a blanket claim that people should just do what's best for their company.
I work 35 hours a week, make $125k/year, have good benefits, like most of my coworkers, and am doing relatively interesting work for a company that isn't completely evil. Sure, my boss can scapegoat me if I let him make technical decisions, but I don't. That's a tradeoff I'm okay with. I'm always on the lookout for something better but I'm happy where I am.
And besides, most people who think their bosses won't scapegoat them if something goes wrong are naive. If it's your job or theirs and they have the power, you're screwed. It's better to avoid the risk and not put yourself in that situation.
To be clear, I'm not saying don't take risks. I'm saying wroth risks on your own behalf.
What economy does work that way?
Trade offs that are business related can be delegated and should be delegated to the business to make. But saying something like: "if we do A, it could make us vulnerable to X,Y,Z technical problems, but would allow you to have what your business needs 7 days early" can be dangerous, because it assumes the business stakeholders truly understands the dangers of technical issues X,Y and Z, which they almost always do not.
As an engineer, you should advise the business towards the proper way to do things, and you are responsible to make sure your advice is clear and loud. It is sometimes appropriate to suggest the less ideal scenario, but if you do, be ready to assume full responsibility and ownership on it.
You should never have an advice that goes like: "Yes we can do it, but it'll cause other problems." and expect for these other problems to not be your responsibility to fix and mitigate as they appear. The implicit of all your suggestions is always: "I can make this work." So better be sure whatever you suggest you can actually have it work.
Plan C is what the Strategy guys always want to go for, because they can't recognize when the repucussions happen. They just tell themselves it's those good for nothing engineers screwing up again.
But too often there's been one guy ithe engineering group who values accolades over stability and will offer to be a hero, only to wander off without finishing anything. Later in life I've realized I should have suffered through more of the issues at companies where the team at least had solidarity. I knew I wanted that in a team, but didn't know that I needed it.
Plan C doesn't differ from Plan B in terms of technical approach. It differs in terms of the consequences of failure - not the technical consequences, but the political consequences.
My first really interesting project that led me down the sort of DevOps/Agile trail was in the mid-1990s, when I helped write a risk management and contingency planning process for the mid-sized company where I worked. One of the features of the process was that management could not reject a risk analysis on any grounds except incompleteness. They couldn't say "I don't want that written down!" They had to sign off on it, in writing.
This turned out to be very popular with both teams and management. Teams felt they were finally getting the opportunity to cover their asses, and management finally felt like they were getting straight answers from the teams about actual risks. In the earlier lack of process, teams were at least passively discouraged from honest, written risk analysis, as naysayers who were resisting business opportunities. Worse, if a warning was given and then the problem happened, the people who warned were blamed for not preventing the problem!
When we beta-tested the analysis process, the first project that tried it actually turned down business because it was too risky. That had never happened before. We probably saved the company many millions.
Later...never arrived for tomorrow is another Very Important Thing that must be done now!
Somehow management continues to forget the critical resource: time to do things right
Eventually the software engineering department has a code base that is riddled with tech debt, to a point where changing a 3 into a 4 ACTUALLY takes 6 days.
Then management asks "WTF? What did you do? Why does it take 6 days to turn a 3 into a 4?", next comes frustration and SWEs leave the company.
I've seen this happen at 3 different companies in 3 different industries, a huge company (hundreds of devs), a medium company (50 devs), and a small company (2 devs) [1].
Every time it happened, it was because a SWE team was constantly delivering under pressure by management that disregarded the cost of writing code "that works" without ever addressing tech-debt, no matter how much the SWEs warned management.
[1] To be fair the small company only fell into that anti-pattern because neither the engineers nor the managers knew any better.
By all means, if you have additional resources, invest them in refactoring and improvements and additional testing and continuous integration and all those things that make our days enjoyable and our product quality high. But your first priority has to be making sure that your product solves your customer's problem.
At some point organisations forget that the process is there to serve us, and not us to serve the process. Moving variables to parameter files, renaming legacy variables, these all seem like much more risky things than simply changing the value of the variable.
I think it depends on the type of product you are working on. Certain domains require very strict adherence to policy - for very good reason. Just because your shiny new aircraft's deadline is tomorrow, doesn't mean you can skimp on the required testing.
A fire is something like, "the site is down for XX% of customers!" (for a large value of XX) or the "the software is routing product to the wrong place!" No doubt a real fire would have been handled differently.
I certainly didn't applaud the gatekeeper because I believe they opened themselves up to a great risk. If you are doing an emergency patch to production you should minimize the amount of changes. All the refactoring of code was an unnecessary, reckless risk. The refactoring should move to the next scheduled release as the top priority since part of it is already in production.
I write tests to check if my code works. And tests that document how the code is supposed to work currently are usually enough to prevent code from breaking in the future.
Anything related to privacy or security should be fully tested. But for the typical startup, I'd posit that beyond that test coverage should be more closely related to the number of users and level of usage rather than to the amount of code.
How often do you write code that's not related to privacy or security? As soon as you connect something to the Internet it's related to privacy and security.
The only situation where privacy and security don't matter a whole lot is if your code runs on airgapped devices with very limited tasks.
Sounds hard to believe. What kind of code would that be?
> There's a lot of code that's not written by web devs.
There's also a lot more than the web that has some form of connectivity with the Internet (even if it's not directly connected it may still parse data that comes from untrusted sources).
There is a widespread belief among many that "security is important, but doesn't matter for me". The most extreme example is obviously IoT ("Who would want to hack my coffeemaker?"), but there's a lot more. The unfortunate truth is: There is hardly any code these days that is not security relevant.
You do realize there's a bajillion non-networked apps in existence? Word processors, excel, editors/IDE's, system tools (esp monitoring/backup), media players that don't download stuff, compression libraries, MATLAB-style tools for numeric analysis, and so on.
All of them parse potentially untrusted inputs. They don't have to be directly network connected to be a security risk.
Just pick the first example: A word processor. It is not a security risk only if you can guarantee that you'll only ever open documents that you created yourself. If you ever use it to open documents you got from someone else it needs to take security into consideration.
Still not writing or patching a networked app.
"Started new area of research: software safety." That was Bob Barton in Burroughs B5000, Dijkstra on THE, and Margaret Hamilton on Apollo code. Maybe they mean first dept at MIT or just making status of sub-field more official. Then TCAS II. I recall reading that long ago as an exemplary work in formal specification & safety analysis but project was too heavy for me. Article says them too haha. Props to her for it & others. Article shares my view on scattered groups & methods. At least seen STAMP referenced once but unfamiliar with it. They wrote against N-version programming being re-invented... which I proposed for subversion resistance. Hmmm. I'm sure my variant is the one that works this time. ;) Also did SpecTRM at their company that looks a lot like state machine and modeling schemes I saw elsewhere in high-assurance. Not claiming a copy rather than inspiration or independent invention + convergence of multiple parties. Usually means a good idea.
Very interesting person. Thanks for the tip. Your sister is going to learn some wise things for sure given they've got sane methods and got results before. I especially liked how the article jokes about writing what she knew on high-assurance development then gotten wiser or more confused. I know the feeling where I'm redoing the foundations now with what a decade taught me. More slowly this time given I have more doubt than certainty.
Note: Just got to the last part. Wait, she was the one who wrote the THERAC paper? I just assumed it was some guy (male-dominated field) named Levenson since that name was all I saw in references to the report. Never saw it again. So, she wrote up an investigation we've been citing about software safety for decades, helped spearhead efforts to legitimize it as a field, did huge projects, and I basically never hear about her. Unreal. I'm bookmarking her stuff to go through it later.
> Sounds hard to believe. What kind of code would that be?
Device drivers, compilers, and some embedded systems come to mind immediately, there are plenty of others out there. I've worked on a lots of software where the only inputs were physical and sensor based, and the only outputs were to the screen. Device didn't even physically have network equipment.
There are few things where security doesn't matter. But they are extremely rare. The situations where programmers think their code isn't security sensitive are probably vastly more common.
I hope you're joking but I suspect you're not. Device drivers have the highest level of privileged access in many operating systems and code quality for drivers is so uniformly lousy (certain large vendors whose names begin with "N", "A", and particularly "Q", I'm looking at you) that attempting to break the drivers would be among the first things I'd consider if I were trying to root a device.
I jump up 1000 levels of abstraction from time to time, and when I do, I agree that security is extremely important (FDA class III device and HIPAA compliance is mandatory.) I'm also a lead, so I have to know enough to call BS when I hear it from a team member.
A good example is that I care a lot about making sure our search endpoint doesn't return private user data. But beyond that, I'd rather just know that the endpoint returns a 200 and let someone tell us if it's broken rather than have an extra three hundred lines of code to see if it's returning the correct results. If we get a ton of users then that will probably change, but for now the cost of writing and maintaining those extra tests wouldn't be worth the benefit.
That used to be true for the software in cars, but it no longer is. The problems that result are not the fault of the original authors; that belongs to the people who decided to bridge the airgap without thinking through the consequences.
For me the most important word here is 'supposed'.
All the time I read documentation describe how code works step by step (what each 'if' does, but spelled out more verbose). And test that only test that a function does by mocking out everything else.
But I don't care reading what code does. I can see that by looking at the code. I want to know what the developer intended/expected the code to do, so I can validate it against what the code actually does. Most of the time assumptions are made with those expectations. And with those expectations you have a much better idea why a trivial refactor of a piece of logic could unearth a massive 'undocumented feature'.
Which works great as long as your changes are shallow. If your changes aren't shallow then you have to change your tests as well and that defeats the purpose.
Automated testing is good for freezing an interface and it's behavior; allowing you to change implementation while maintaining the same outputs. Lots of technology requires this: Networked API endpoints, libraries, etc.
But most change I encounter is from changes in requirements that necessarily requires reworking and re-arranging code that won't be compatible with the test suite.
I've had to completely redesign entire subsystems; break down components into different pieces; move code between different layers; etc.
Honestly as a manager nothing bothers me more than developers who try and patch complex changes into existing systems without re-thinking how it affects everything else. I have to re-factor a project that's a total mess because code was added but not removed or changed over the course of some very big requirements changes. I'm sure all the tests pass but it's impossible to follow now.
The problem with unit tests is it adds an extra layer of friction on making changes that benefit the product. You are actively discouraged from changing your design from your initial assumptions! This change friction can be seen as a benefit if you need all your interfaces to be stable (like with a library). But it's a trade off and it's not appropriate everywhere.
I'm not saying anything about bad architecture. You have a good architecture for today's requirements that is a bad architecture for next year's requirements. Tests lock you down to whatever your first architecture is.
I prefer automated tests of larger units or subsystems, and testing against requirements, protocols and specifications rater than implementation details would change. Of course anything will change over time, but I believe this kind of tests provides higher value and lives longer.
- for catching regressions in the future
- to verify that your code works. How else would you know that your backend service reacts properly on some edge condition
- as documentation
- as a kind of quality mark, for instance to be able to pass a code review or when writing code for an external party
Unfortunately the last reason is also the most useless and still it is the one that seems to be the main motivation for many big enterprise developers.
Thank you for including this - it's underappreciated. So many times recently I've needed to write code against some poorly documented API (not always due to lack of effort, some things are hard to document well in prose / JavaDocs, or just due to constant change), but I took one look at the unit tests and it all made sense - and I knew it was up to date for the latest work.
Reality, on the other hand, is usually suboptimal.
You should be testing input/output and results as opposed to testing how the internal gubbins works. That's the line we have to carefully tread when making a test. The test shouldn't force the item to behave in directly the way it expects; more that the I/O is correct.
it depends how brittle your tests are. You can write tests that make sure internal stuff works at a unit level without them being so brittle
to your point testing I/O or behavior is the way to accomplish this, but it can still be done at the level of internal functions/methods
Unit tests should be used for extremely small and isolated mission critical objects while functional tests should generally cover the entirety of the I/O chain. That's how I do it at least and it works extremely well for a fraction of the cost!
This is in no way a benefit unique to unit tests.
If your project breaks because of local changes I think regression tests with real data and bisecting is better and less work though.
1. Your unit tests fail basically every time anything changes. This is the scenario where your unit test is something like "the command line arguments are -abcd" and every time you add one you need to change the test. This makes the unit test worse than useless, but actually a source of extra work every time you change something.
2. Your unit test never fails. It just doesn't fail ever, at all, under any circumstance. It's so obvious that it should work, but someone wrote that test anyway. It's a waste to run it every time.
3. Your unit test fails when you refactor because it tested some internal functionality. You need to throw away your unit test every time you refactor. It's a waste to write one every time.
The only tests that ever show that a refactor broke something are integration tests. The 200+ unit tests in my project either NEVER fail. Except for that one that you have to keep changing every time.
> I get paid for [code and tests] not for tests.
> I get paid for [code that works and is maintainable], not [more tests than are strictly necessary to achieve that goal]
I don't know this guy, so I can't speak for him in particular. But, in general, I wouldn't be surprised if the opposite was true: too many so-called software developers give "shipping" too much importance, leaving none for any other aspect of the job. Shipping is a feature, not the feature.
This is not some random guy, he is like the "father" of TDD. And that is why the guy that wrote the article thought it noteworthy to mention his quote.
If it came from someone else it would not be that important to make such a fuss about it. But when it comes from Kent Beck then it is worthy at least some discussion.
Literally the second sentence in the article.
> Kent Beck, respected authority, creator of Extreme Programming, TDD and writter of several great reference books, mainly at the great Addison-Wesley edition
Back in those days, there was a backlash against "big design up front", and very little respect (in general) for testing as a practice. Unit testing in it's modern form was reasonably rare.
After this Agile/TDD stuff caught on, many people ended up over-testing things. This is a pretty typical thing to do when you're learning about how much testing is sufficient. I've definitely done a good amount of this myself.
It can take a good deal of experience to know where to draw the testing lines in particular contexts. I think this blog post points at this specifically - that we should write high-value tests, and just enough of them. We also use feedback over the long-term to have heuristics of where we tend to have recurrent issues, so we can test a bit more in those areas.
Far from being focussed on "only shipping", he's underlining the fact that "just enough well-written tests" support working software - and that should be our focus, instead of thinking our job is to "write more tests" (or focus on test coverage, etc).
Unfortunately, there isn't a really great language that enforces working like this, while being simple enough to push onto a big team :/ (if there is, please let me know)
I've done a lot less statically-typed code than dynamic (mostly Python) in the last few years, but I was playing with Unity3d recently and had a chance to write a good deal of C# with Visual Studio.
I hit a hairy problem some time along and had to do a big refactor to support a new feature. I deleted a single line of code, followed an error trail for 10 minutes, and suddenly everything was just done.
It was a pretty interesting moment for me. I realized that statically-typed languages can really have the potential to be as or more productive than dynamically typed languages paired with a good enough IDE. (And Visual Studio with C# is about the best pairing you can get).
I've had to reconsider my thoughts on these things a bit. Sure, there are things that statically typed languages can make much harder to test or work with. (Want to stub an external provider? Okay, you're going to need an extra interface, then you'll need to create a new stub version that implements that. Want to read/write pretty-arbitrary JSON? Good luck). But there are other places where you get huge wins by bugs just disappearing by the boatload.
I'm still not on board with heavy OO/inheritance, and love the pattern of simpler struct-style constructs with just functions in functional programming, but the static typing can give a lot of wins.
I think something that gives an inherent advantage to OO languages in IDEs is that SomeThing.<tab autocomplete> makes a lot of sense and is easy to compute! I can take an object and know what I can do with it at a glance. I haven't seen a functional language with enough structure to support that simple feature yet (though maybe I'm just not looking hard enough). This is really where statically typed languages can make the most of an IDE. For some reason, that's the big thing I think of when I'm thinking about the downsides of functional programs I'm working with. The editors just seem to help a lot less (though I haven't written any FP professionally-speaking, so have less experience in general with tooling).
F# using Records with member methods looks like it might be able to get that sort of benefit though, I'll need to try that. It looks like they're just pure functions declared on immutable structs, which I think is the perfect middle ground.
A lot of object-oriented languages have taken tips from functional languages lately (map/reduce/filter is the new hottie), but I think there's a lot of benefit still to get in the opposite direction.
I'd point out, though, that is still is only part of the picture. You're holding on to a value and you want to know what you can do with it:
widget.<tab>
widget.someData
widget.doSomething()
freeFunction(widget)
handler.handleWidget(widget)
hammer + widget
widget[part]
// returns a widget, does this one count?
widgetFactory.buildWidget()
It's hard to continue hewing to the tab as activation with these other structures, which may be why IDEs and REPLs don't really try.
I actually dislike tab complete in usual forms most times. Auto import is nice, but i feel that auto complete is a form of searching the code base. And, when I am coding seems a poor tube to be searching for the answer.
When debugging, however, jump to symbol and quickly listing alternative methods helps. And sometimes I am just searching. So, good feature. Just not something i want to rely on.
F# (as well as OCaml) offers something similar in that you'll use a lot of functions that are within modules with the same name as the type you're working with. So you can write "List." and get a list of functions (map, reduce, etc.). I'd prefer something like Idris which will disambiguate functions based on the relevant types, but at least it makes IDE support easier.
> A lot of object-oriented languages have taken tips from functional languages lately (map/reduce/filter is the new hottie), but I think there's a lot of benefit still to get in the opposite direction.
Something in particular I wish F# would add it general non-linearity of definitions. All files and definitions in F# must be strictly ordered (either type A can reference type B or vice versa, but not both) except for specific, contiguous blocks. It presents a challenge for type inference, but I think just punting it back to you for the tricky cases would be fine (and it often has to do this anyway).
If you do need to get around it though, you can have mutually referential types in F# if you use the "and" keyword (although the definitions of the types have to be right next to one another). And in the next update to F# you'll be able to have mutually referential types and modules within the same file which is often good enough for most other things you might need that sort of thing for.[1]
[1] https://blogs.msdn.microsoft.com/dotnet/2016/07/25/a-peek-in...
Parsing JSON in Java was one of my worst programming experiences, so I have to agree with you here.
But in Rust, using the `rustc-serialize` library (and Serde, but I haven't tried that yet), parsing and writing JSON is really pretty painless. The really nice part is that you can declare the structure of your JSON data as a completely normal Rust struct (just with a derive annotation that makes it Encodable and/or Decodable), and with a single function call turn an instance of that struct into a JSON string. And in reverse, you can just parse() a string and it will return either an instance of your struct or an informative error if the JSON is malformed or doesn't match your structure. Makes JSON really easy to work with.
In golang recently I had to take some json (that I only knew part of the structure of), and modify just that small subpart of it without touching the rest. It was a really painful thing to develop, and the code ended up very messy.
There were a few golang libs for reading arbitrary json, but none supported writing to it that I could find.
Indeed, and C# isn't even a particularly safe typed language. It still has pervasive null, for instance. When you get into F#/OCaml/Haskell/Rust-type languages, it's a real eye opener.
> Want to read/write pretty-arbitrary JSON? Good luck
Not sure I see the problem. Just deserialize JSON into a JsonValue which provides dictionary semantics like JavaScript.
The break-and-follow-the-errors approach is very powerful. It's usually not hard to find the exact break that will show you all the things you need to change, and then just work through them. My record is 5 days without buildable code, working in C++; once I'd worked through all of the errors, the program worked, and without any non-obvious problems.
I miss this a lot when working in a dynamically typed language.
(Thing by Jonathan Blow that touches on this: https://web.archive.org/web/20140929232443/http://lerp.org/n...)
I'm surprised this myth perdures.
Reading arbitrary anything is trivial in a statically typed language: use a hash map.
There. You're merely emulating what a dynamically typed language gives you, of course, but it's trivial. And at least, statically typed languages give you the choice: you can be dynamic or static. You don't have such a choice when you don't have types.
Another is that some code reviewer asks for more tests. A third reason to spend time on tests is that they're required to maintain the same standard as the shipped code, even though they're only run in the presence of the developers, and their breaking only affects the developers, not any customers.
People forget the ultimate reason for our work oh so often.
I write tests, many tests, when I'm working in a dynamically typed programming language. I write tests even when I'm working in a soundly typed language. The only difference is that in soundly typed languages the type system guarantees many properties for me so I don't test for those.
Personally I like to write tests first but I don't believe that gives me any productivity benefits. It's just the way I think.
> People forget the ultimate reason for our work oh so often.
Tests are important because reliable code is important. The customers are important but so is the business. It costs quite a lot of money to support error-prone, poorly designed software. Tests aren't a silver bullet but they are a tool to alleviate the problem.
For example when I changed some code for which tests didn't exist, so I tested what I changed and wrote some extra tests while I was at it, and it was blocked in code review because my extra tests didn't report failure in any detail. What I did said "x failed" if a test failed, no details. The reviewer said much the same as you did now to justify that additional reporting was absolutely required.
It's a fine sentiment when it actually applies, and I wish it weren't applied quite to often to justify YAGNI and other rubbish.
The advice you received with regards to defect locality sounds reasonable - tests that don't give much in the way of isolation can cost a lot of developer time to hone in on. It's hard to say, not knowing the exact details however.
I also find it hard to reconcile the idea that "tests are an important tool for writing good software" with "justifying YAGNI". How would an "openness for developer testing" justify an attitude of "not writing things you don't need"? Those two concepts sound almost entirely orthogonal.
Specifically, if a test passes right away, then its error reporting isn't important today. It probably comes in useful if the test ever breaks, but will the test ever break? Therefore, spending significant time on the error reporting today is YAGNI, even if minimal version of the test is useful.
My complain is that even if unit tests are useful to a degree, people trot out the reasons for usefulness primarily when those reasons do NOT apply.
Not to mention it will literally take you 2 minutes to add the better reporting.
So if it takes you significant time to get proper defect locality, I think you should see if there's a better way to approach the problem. This should be essentially the default for typical/modern unit tests. Perhaps you're writing tests that are more like system level tests?
I'd also say that if you're essentially certain a test will never break, then (other than for documentation purposes) why are you writing it? To paraphrase Kent Beck - we should only test things that could possibly break.
You might be overgeneralizing what "people" say about unit tests - I'm not sure what your specific scenario is, but there are a whole spectrum of opinions on the subject. Perhaps this is just an organizational code-smell of the place you're mentioning.
Dogmatism/cargo culting in general can be annoying however.
Yuck. Imagine if you got a bug report that just said "X failed" with no details.
Why do you think that is the wrong approach?
When something doesn't work as expected, I now check my design and not the code.
Or you're tired. Or you're just not as focused as you could be. Or you have a deadline. Or you're trying something new. Or you're not fluent in the language yet. Or you're fluent in the language but not the framework. Or you're fluent in the language and framework but not the design pattern (if you use those).
Or a whole host of other things.
I'm not saying spot-checking the design isn't a good idea, but saying that it's the design more often than the code just doesn't match up with my experience.
Based on a true story
Out of curiosity, did you use any debugging on paper for your code?
1. Write tests to prove your code works, which is sometimes referred to as "test-driven development"
2. Write tests to catch any side effects or regressions when altering code
Funny thing is that if you write good tests, then the results are pretty much the same regardless of your motivation.
1. Have an overall idea of what your software will do before writing the first line of code.
2. Challenge and change any touched assumption from #1 during development when you refine that idea.
3. Test that the program satisfies your refined idea after it's written.
4. Create some assurance you'll keep #3 correct while you write any further code later.
However you fulfill those needs, if you got them all, you are good.
Automated testing only shows its real value when you go back to change code that was working before. With the manual approach you'd have to retest everything to have any real confidence. With the automated approach you just run a command.
I'm a big fan of automated testing, but if I didn't expect to have to ever change code, I wouldn't bother with it.
> With the manual approach you'd have to retest everything to have any real confidence.
This kind of implies that software development before "~tdd" was a complete disorganized gong show of quality especially where refactoring, but in fact that was not the case. There are ways of coding that are more conducive to quality than others.
> With the automated approach you just run a command.
Once you've written all that code, yes.
> if I didn't expect to have to ever change code, I wouldn't bother with it.
Usually you don't, in which case the extra effort on testing is wasted, usually.
The typical claim is that code maintenance is at least 10x as long as the initial write.
The question isn't "do we need to test." Testing can just take the form of running the code manually and making sure the output makes sense, but you do need to test.
The question is "do we test automatically or manually." It's the same question regardless of how good your practices are. Note that this is totally distinct from the question of whether or not to use TDD.
> Once you've written all that code, yes.
I've found that writing tests often doesn't take much longer than testing manually, and rarely takes longer than testing manually twice. Sure, if you obsessively try to test every possible case and input, you'll waste time, but well targeted testing doesn't have to be slow.
> Usually you don't, in which case the extra effort on testing is wasted, usually.
For non-trivial projects I have an average number of revisions per line of code much closer to 2 than to 1. Sure, some of the code only gets written once and never touched, but other code gets revised multiple times. If you're good at writing tests, the tests will focus on those often revised lines of code.
And, again, you have to compare the effort against the effort of manual testing, not against the effort of writing code you've never run and shipping it.
end to end tests, integration tests, regression tests, etc, yeah.
Unit tests though...usually no. Often if you're making any kind of significant change, the entire code paths may get refactored away or change too much and the test will get nuked anyway.
And it's that kind of test that usually confuses people, so it's worth understanding.
A unit test's goals are many:
It proves at authoring time that the code works. It saves you the time right away of having to go through the UI or spinning up a server just to validate a function is working. It proves that you thought about specific edge cases (and if you do testing consistently, the lack of test is your evidence of unconsidered edge cases. It's documentation of all of the things you considered when writing the code. It is an example of how to use the code with all it's use case.
And when you nuke a piece of code away, the failing tests are now a guide of all the cases you have to make sure are truly no longer necessary.
If, in the future, you do a refactor of an implementation detail (so the existing tests are still valid), then that's bonus as you get green/red validation. But in practice, that is less common than all of the other reasons for testing. That's why the "I don't expect this to change" thing isn't really a reason for or against writing tests.
That pretty accurately describes my experience with unit tests. I've been part of several projects where we had pretty comprehensive unit tests (I'd say small to medium sized projects) and I never managed to get as much use out of unit tests as I liked. After one or two big refactors most of the tests needed to be, as you said, nuked anyway. While seeing all the pretty green lights is reassuring, they are rarely working when you most need them - during large refactors which blows away big sections of code.
I picture unit tests as a row of black boxes sitting on a table in certain positions. Unit tests are great when you don't move the black boxes but do change the mysterious processes are running inside them. But refactoring is rarely ever that isolated in programming since you tend to move some of the black boxes around, remove some entirely, add some new ones, change the contents. To then expect the unit tests to give you back useful information on whats broken is rarely possible.
I've had more success with e2e testing using things like Selenium, but it's still frustrating as a developer to read articles about how great unit testing is (like this root comment) and never able to actually get a decent working version of it in your projects (because of the reasons mentioned by this parent comment).
Building up small oases of dumber/more verbose code that has unit tests seems like the best way to wrangle legacy code into something that anyone else on the team can understand and not mess up 6 months from now when it's their turn to have to touch it for the first time. Of course no one wants to touch that 200-line, 10-levels-of-indentation monster method, but bringing just a little bit more and more of it under unit test over time will help a lot. Every other benefit of unit testing that you listed besides the time saving aspect (e.g. why launch a big e2e test if you can test the same thing in an xunit-like context? even if it's not strictly a "unit" test) pales in comparison of the benefit of making crap code nicer to work with. A corollary is that if your code is already nice to work with, and you have a system to keep it that way, unit tests won't be very valuable. (Though other tests, which may or may not be in an xunit-like context, may still be quite valuable.)
"Because you should write code that works in the first place!" -- your boss
But when changes can come from anywhere (lots of developers potentially making changes) and the need for changes can be urgent, there's a great deal of value in complete code coverage.
Not every developer thinks the same or has the same tendency for errors. I would caution people against thinking about tests as personal verification. With any luck, you'll be handing off that code base to someone else in time.
My tests and practices are, occasionally, like a five point harness in a race car. Without it anchoring the driver in front of the wheel, they could never ever drive that fast with any safety at all. This one comes out as a counter to the 'straightjacket' argument some people like when the tools won't let them just write shit code that the rest of us have to babysit while they run off somewhere else to write more shit code. Which we are supposed to be grateful for... I digress.
But most of the time my tests are more like smoke detectors. The ones that go off for no reason get replaced or removed. The ones that go off after I can see fire and smell smoke? Why am I paying for upkeep on those exactly? What's the value?
OTOH, projects with excellent end-to-end tests tend to have lower coverage but better regression management.
I think this is the point of the article?
I think that full test coverage is nor sufficient or necessary for the working software. What matters is the sufficient covering of the edge cases.
I think that an additional point to keep in mind is that most probably Kent Beck talked mostly from the point of statically typed languages.
Testing makes a lot of sense if you're doing data munging, but for front end or other kinds of code that are almost defined by their ability to create side effects, it's usually a waste of time. In my humble.
"We" write comments and use "plain" language documentation, or formatted but otherwise plain language for documentation generators to parse to document our code, and that includes "what the code does."
I'd argue that if your test code is what you're relying on to "document what the code does" then you are probably (almost certainly) over-testing, testing the wrong things, or some combination. Oh, and also using test code incorrectly (as documentation).
Do you allow your juniors to use their sense? No, you make them hit 100% coverage until they start to learn.
As Paul McCartney said: "Learn the rules like a pro, so you can break them like an artist."
If the input data is complex enough to drive a non-trivial branching logic in our code - yes we do. Well, at least I do. The bonus is that it provides the fixed point for further changes at the same time.
You don't want tests for that - you want a type system and a static analyzer.
So what methods do you use for that purpose?