Fear makes you a worse programmer (2014)
jvns.ca
jvns.ca
As those projects dragged on and I was unable to make "progress", whatever that meant, I felt shame and a mounting dread of returning to the console each day. Eventually, fortunately, I was able to roll off them (not having accomplished much in the preceding month or two) and got back to doing useful things.
These days I can usually recognize such projects in advance, but it's still not always possible to avoid them.
When I see people struggling with these, it's usually from a lack of information seeking/gathering, where they first sit down and code rather than spend the first few weeks talking, reviewing understanding, and, most importantly, finding those few A-team people that have meaningful input.
Definitely not something someone early in their career should be given, but these types of problems usually benefit from devaluing manager input, since they have a disconnected/warped perception of reality. I take these projects only if the understanding is that I'll be solving a problem, taking input from all involved, rather than implementing a specific solution.
my reading of it was more like some past experience where I tried to do what you described only to be reprimanded for "overstepping boundaries" and that not being my job function, especially when the manager/PO/etc insist on being an exclusive gateway to information despite repeatedly failing to do that job correctly.
I think we all agree on the importance on your last part where you need to suss out whether they want an obedient code monkey or someone to solve the problem. probably through an iterative dialog where the parties involved get to explore and update their understanding of the problem, the technical and non technical constraints and the solution space.
A strong signal to leave a sinking ship. I suspect even if you somehow pull a rabbit out of the bag you wouldn't be rewarded for it. I stuck with one project like that when I shouldn't: I finished the work however the useless salesperson was screwing up the communications with the client and didn't get the project across the line to them.
One of the most demotivating environments to suffer.
This siloing is what causes fear and uncertainty about project, and also what makes developers atrophy their "talk to users" muscle.
It is what made me leave jobs in the past and made me fight to change how things work at my current job.
If this is the case, I would say you're not being "set up to fail", which was the explicit description the GP gave. "Set up to fail", to me, implies that whoever is tasking you is not tasking you to actually solve the problem, either because they're too clueless to know that what they're specifically tasking you to do won't work, or because there is some other hidden agenda in play.
Perfect wording! I tell people the hardest problem, in my work, is getting people to see there is a problem. The crazy workarounds that people will come up with, to avoid the root issue, are really incredible.
- as you said, this can’t be too soon in your career : gathering requirements and knowledge is something that you can’t do without experience
- you need to be 100% confident that you are working in an environment where you will be rewarded accordingly. Working hard on those projects just to see your manager be promoted will be an absolute emotional disaster.
If one of those requirements aren’t met, you are good for burning out.
In the past there was no choice: developers would talk to users and stakeholders and collect information.
Today there are few opportunities for a junior developer to do this.
There was constant complaining from both sides: from product that "tickets opened to us are horrible, support/customers are ignorant" and from customers that "nothing ever gets done here".
In the end nothing of that was true. Nobody was ignorant and a lot was getting done.
Yeah, that's not agile. The entire point of agile is that you close the loop and the reason you break your work into smaller chunks is so you can deliver them faster in order to gauge the feedback of that chunk and better understand what should be delivered in the next chunk.
If you're not closing the loop, you're not doing agile, you're just doing waterfall with more bureaucracy.
Things being up in the air and vague is often an opportunity to step in a tame a wild forest of ideas into a real application.
It definitely takes a certain type of mindset to harness that energy and herd the cats though.
All one text file (300 baud VT-100 CLI). The line printer saw lots of use.
No comments.
No subroutines (just lots of GOTOs).
Helpful variable names, like A32Q3
It was mid-1970s vintage. The original developer was a Bob Ross-looking guy that I wasn't allowed to bother.
I was on my own.
My most effective tool was a magic 8-ball. Lots of guessing.
It helped me to ensure that I would never do that to anyone, ever.
It did help me to be better, in the long run. Once you have seen Hell, everywhere else is Heaven.
Yes! I have been in such a situation once (I didn't quite understand what I was supposed to do and no one else did, either) and, to this day, I remember it as a cautionary tale when I think of moving to a new position.
The people who think that they can shine in this situation by being proactive underestimate the lack of understanding: it is not a blurry task, it is a task where you are told to do X and no one knows what X actually means. You can do great things, but they will not be able to deliver on the requirements.
But there was an actual developer team as well, including all the original founders - who knew the system perfectly. So things would get posted and you'd be like "oh, that's a good starter task..." and instead a "core" developer would pick it, do the one line fix, and then just push it straight to the master branch (or PR it but it would get approved within minutes in their timezone) while I was still trying to get my bearings in the code and tests.
Within a month or two the manager who hired me had "resigned", and then I was let go near the end of my probationary period. The whole time I never had any solid work assigned beyond "oh figure out where you can be useful".
It's also possible the project is simply too hard -- maybe doing it right would take an experienced engineer two years; but since that approach seems like obviously too much work, you flail around assuming there must be some alternative.
This is, sadly, pretty common for junior programmers and people who are new to a team.
Every damned time I set out to implement changes necessary to ensure the maintainability of the project, the boss would bring in an architect (who otherwise never even looked at the project) and he would pull the rug out from under my feet.
Every damned time. By the time they realised I was actually useful and regretted the situation I was checked out and ready to walk out the door.
I had 18 months of pure hell. Only after I left did I realise I was in a sort of abusive relationship. I heard later he said that my problem was I refused his help. It was a miserable experience, made worse because I pretty much always blame myself if something is wrong.
Has anyone ever actually found this to be true?
I work in a place with 99% test coverage requirements and it's honestly still a super brittle system that everyone is afraid to make big changes to
In my experience, automated tests don't promote quality engineering, they just calcify whatever quality exists
And before you get into it, TDD is not a panacea for this problem
I've been occasionally pushing coworkers to restructure code and tests into a higher level style, defining an API boundary and calling it from the rest of the code (as if you're writing a library except it's inside your codebase instead of installed), then writing tests to that API. That does make for some easily refactorable code where the tests don't need to change at all.
That pseudo-library style is kind of what "unit test" originally meant: a business unit, not a code unit. Examples simplified it too much and people copied the style instead of the substance, and the original meaning was lost.
For another example, some of our recent projects have been on updating daemons, those have also been really good candidates for this since there are clear entry and exit points.
Very interesting. It sounds a bit like what happened with the term "Hungarian notation".
I don't consider it a problem, but it is to those who want to see big numbers.
As for hardware-related problems and race conditions, testing at a higher level of abstraction seems like it'd help more than it'd hurt - in the former case ensuring graceful handling as part of the tests, and in the latter case making the race conditions more likely to hit (and hopefully fix).
It’s a bit of the old classic “reading from the file never returns an ‘access denied’ error, until it does”. There’s ways and means, but let’s not pretend this is simple.
In todays development world, the unittest is primarily a developer tool to help speed up development. One should not be afraid to delete unittests if they are doing refactorings. But the long term value is always in the integration tests.
The important thing is not the size of the code affected by the test, the important thing is that a test should verify a single requirement isolated from other requirements.
I believe the original distinction between unit test and integration test was that integration test was for when parts developed in isolation by seperate teams was finally integrated. But this tended to be a disaster, which is why continous integration was adopted.
Obviously, when reading the rest of the papers, they are clear on the fact, that it is the specifications that the programmer have developed for, that should be tested. That was at the time, synonyms with individual sub routines, and as such it was both the smallest unit and a logical testable unit at the same time. Since then, we've come a long way with programming smaller units with better composability.
I'm not sure I agree that the original meaning of a unit test was lost. Perhaps, only one part of the definition was carried over to modern development practices. In any case, I always stand by the fact that long term value is in tests of the "API". Everything else is implementation details.
When I was at BigCompanyA, we had 95% coverage requirements as a management fiat. The company generally was very management-decree-driven. People would unit test individual methods and helper functions and each “unit” of code. If you wanted to change an API, you had to dig through a sea of broken tests because each little bit of code was individually tested. We literally had unit tests that validated one line methods for string concatenation of a prefix (in Java). Management said add tests with each commit so everyone added tests without thinking about what is valuable.
At BigCompanyB, our testing was engineering driven, not some management metric being tracked. The goal was to test “public interfaces” and ensure that these tests captured all the helper methods along the way. This helped catch dead or extraneous code if you couldn’t write a test to exercise a particular path. Changing an API didn’t require changing a bunch of silly tests. We still had equal >95% coverage, and it was very useful.
Basically you need to actually think about what makes sense to test and write logical tests.
You could quickly make the changes you want not worry about what else breaks, then run the tests. The tests that you expect to fail would fail, and you fix them. Then you find other tests that you didn't expect also failed; you'd review the impact of your changes on those parts of the system and make appropriate changes until the tests were happy.
Because the testing was so thorough (and quality), you had high confidence that nothing unexpected was broken.
It was "move fast & break [tests], then fix them and deploy" :)
Unit tests which dependency-inject mocks of other parts of your codebase are 99% worthless.
Source: Spent many years writing the latter and they caught close to 0 bugs. Moved to Elixir ecosystem where integration tests are the norm, and they catch bugs on a regular basis.
I never understood the fascination with Unit Tests. For Testing to be useful the code being tested should have a certain degree of Complexity (algorithmic/behavioural/state-transition/etc.). But what i see from most unit tests is mere "busy work" as if mock-testing a trivial class/api/etc. would somehow make your code better. A similar criticism is also applicable to TDD based programming.
Me neither. The legacy forms of testing that "unit" was trying to differentiate from have died out. All tests are unit tests today, which is why most developers just say "tests". "Unit" is already implied and adds no additional information not already understood by "test" alone.
Same goes for integration tests. Integration tests, as it was historically defined, died out. It seems anyone still trying to use the term today is using it to also just mean "testing", or what was historically known as unit testing. Like unit, a pointless qualification.
It depends.
I've written a number of apps where I'm fearless with making changes because I trust the tests I wrote. One of the apps has been running for 7 years and it's had a lot of big updates, refactors, etc.. I'm breaking every rule there is on jinxing things but there hasn't been a single bug introduced due to those updates and there's ~85% coverage. It's only a ~3k line Flask app though, but it does get deployed straight to production with no staging environment and gets hundreds of requests per day. It writes to a DB, Redis, interacts with multiple 3rd party APIs and sends out webhooks so it does quite a few "external" things.
I've never been a fan of TDD and personally I think tests that really hit your DB, Redis, etc. help a lot more than mocked out unit tests or a billion unit tests and nothing else. I tend to write more tests that really test things together. Not full blown Selenium style tests, but I do really write to the DB and other data stores in tests and often use a framework's built-in test client for making HTTP calls. Everything can still be really fast too (~100 tests in 2 seconds is my usual rough guide for having an assortment of "real" tests with Flask).
I've come to the conclusion that the idea that "unit tests" should test functions/objects in isolation with completely mocked dependencies is based more on the slow speed of those dependencies in the past than what actually makes for good tests. Now that we have faster computers and storage devices, and easy/fast store creation, we should move past this.
Obviously, this is very dependant (no pun intended) on the dependency in question, but as a minimum, anything with SQL should have a test that hits a real SQL DB (PG in docker for example) at some point.
External dependencies (such as PostgreSQL, Redis, what-have-you) are a pain though. I feel pretty strongly that just a single command ("go test", "cargo test", "npm test", etc.) should run all the tests on all platforms, reliably, with a minimal of fuss and messing about. Things like docker-compose or whatnot quality as "a fuss and messing about".
That said, in the current job, we have a dedicated Postgres container that comes up with with a just file command. The setup, test, and teardown of schema all happen within the standard test system of the tested platform (pytest or go test specifically).
https://engineering.atspotify.com/2018/01/testing-of-microse...
> Has anyone ever actually found this to be true?
Yes, but you have to have the right kind of test coverage, and that's the tricky part.
Yesterday, I refactored a bunch of functions that changed a bunch of unit tests. However, since we also have integration/system tests on that code, I'm confident that I haven't broken the code as a whole. Without those system tests, I would not have confidence that the change would be successful, and probably would not have refactored.
In another codebase that hadn't been touched for a year, as part of a feature change I refactored an SQL statement to what I thought was a more optimal design and immediately broke a bunch of tests. Based on that, I was able to understand the original intent of the SQL, and updated it in line with the feature change. I added test scenarios for the new feature, but left the existing scenarios as is.
Without those tests, would have broken the system in a subtle way.
We have a 300kloc monster, and I find that going up from 0 to 60% test coverage has given me appreciably more confidence that the system still works after any change.
Sure, the test code is almost twice that size, and breaks almost as often as the code, but my confidence in the system itself is definitely higher.
To prevent people from writing tests that need too many mocks we now have explicit dependency injection. So much easier to reason about stuff, and prevent’s people from not testing the important bits.
In most other cases, we had tests that only covered the happy path and did little more than slow down builds and make refactoring more difficult. In other words, they made things worse. E2E tests, in particular, are the Afghanistan of web development. I’ve seen more time wasted getting useless tests green than I’ve seen wasted on any other programming exercise.
If you have to rewrite tests that means you've changed the user experience in ways that are not backwards compatible.
Which is sometimes valid, but not exactly what is being talked about here. The discussion here is more about changing the code in ways that makes the code better, but still delivers the same user experience – possibly with new features added, but not where anything is taken away.
That is only true for integration tests. You can rewrite a set of local functions without changing user behavior, and then you need to change tests, such refactors becomes a pain when you have too many unit tests but are really easy when you have many integration tests.
If changes to "local functions" calls for tests to be rewritten, that means you've exposed "local functionality", even if by accident, to the outside user. Which means it is not actually local functionality, but something you have exported and are committed to maintaining. Rewriting the tests is not the correct course of action. You need to fix the code that you just broke as the functional contract was violated. With any luck that hard lesson will teach you to be more careful next time.
But you are right that deleting the tests isn't an option as it will break the contract that was entered into with the users of the code. Of course, you can't modify the tests for the same reason, so...
Obviously tests are going to break when you change the code's APIs and functionality. That's expected, and does nothing to boost your confidence in the part of the code you're working in. The point of tests is to improve your confidence that you're not doing things that have an unexpected impact somewhere else (hence the Bob Martin quote in the article about tests being useful even if you have good design).
Tests aren't an alternative to thinking. You still have to consider what changes you're making, and what tests should break as a result. That's just part of your code change. When tests that shouldn't have broken start failing that's when they show their value.
Of course if you need a higher standard of guarantee you can try model checking or other formal verification techniques.
I am always in favor of tests around the boundaries between system components (regardless of what you call them; integration tests, API tests, etc.) The more robust the better.
If I'm working on a system with robust (ideally generative) integration tests covering the interface of a component, I feel I have near-complete freedom to rewrite any aspect of the component I want with high confidence.
Stable interfaces that the user cares about (e.g. CLI, public API, RPCs) make for the highest-value tests.
Funnily enough, "unit tests" is what this was called originally. The boundary is what "unit" referred to. In my experience, these days most people just call them "tests". That they happen at the boundary is implied as by this point it is generally agreed upon that testing anything else is a waste of time at best and sometimes even detrimental.
Which is why nobody in fake internet argument land can settle on what "unit", "integration", etc. mean in the modern age. There is nothing related to testing in need of additional qualification to communicate to other developers what isn't already communicated with "tests" alone.
How have you seen this done well? In my experience, this usually ends up being some hello-world type simplicity that doesn't really represent the real world use corner cases.
For public examples, I'd point you to (e.g.) Jepsen.
I'm not going to deny it requires a pretty high level of time/money to implement. But done well it's super powerful.
There may be a black art to recognizing what you overlooked, but otherwise it is pretty straightforward. Tests are your documentation. The angles needed to be covered are exactly what the user needs to know to use your software – how to use it and, when using it, what happens both in the expected case and when failure occurs (especially what happens when failure occurs!).
Calcification is of little concern as the inverse is breaking changes, and the user does not want to deal with your breaking changes. You can put your mind at ease knowing that once you commit to a documented feature, it should remain there until the end of time.
Please explain this to Product Managers. :)
But only at companies that don’t give a shit about test coverage %. The percent kills the purpose of the tests, which is certainty.
> I work in a place with 99% test coverage requirements
Sounds like your problem is bad tests. In fact, I guarantee they’re bad tests because that’s exactly what coverage requirements lead to
Yes, always.
1. Your input will be incorrect/corrupt/malicious. Sanitize the crap out of your input. Think of every possible way your parser could go off the rails and fail.
2. Your code will hit pathologically slow cases. Thread the needle between the overly complicated but linear thing and the simplistic but straightforward to implement, debug and test O(nlogn) thing.
3. Your code will fail in production in a way that is hard to debug. Put in logging, monitoring, and dashboards. Check them. Alert on them.
4. You'll have to debug your code. Put in tracing modes, use good names, divide things up to separately debug them.
5. You'll have to explain your code. Make it easier to understand for future, drunk, or stupid you.
These days I'd argue that Rust is at the same level of Haskell and Purescript in fearlessness.
Some languages that removes certain fears:
- Kotlin: removes the fear of null exceptions
- Go: removes the fear of forgetting about an error that this function throws
- Languages with ADT (Haskell, Rust ... etc): remove the fear of missing a case
Now, management and leadership were most affronted by the state of affairs, I assure you, and many words were spent extolling virtues and lambasting vices and sin. But let me assure you, it's not that they love eloquent speech, they were simply not allowed to say the real reason for any of it because the real reason was a trait of the org that the founder thought was a key to their success.
They run a meritocracy, you see, and a pay for performance comp philosophy. They run performance reviews very tight, and getting a passing mark is publicly said to be something to be proud of. The natural result being that everyone lives in constant fear, and being put on a big risky project is a great way to not get to vest your whole equity grant. So everyone plays defensively, the smartest people dodge the hard/big projects, and so on and so on.
And they'll never fix it because they are not allowed to. How tragic.
- metrics gamification
- wrong incentives
The article reminded me of the military's emphasis on maintaining a healthy level of fear to prevent complacency.
When working on new projects, there's room for bold moves and experimentation. However, in existing large-scale systems, the impact of our actions extends beyond ourselves to the entire team and company. Let's avoid cowboy coding and prioritize stability and reliability.
You can partly address this by trying to make sure every part of your codebase is easy to understand, but sometimes your code is just going to be complex. That becomes an education problem, you have to work with these developers and coax them into confronting difficult chunks of code and help them develop the skills they need for understanding.
The same applies for development/debugging techniques. I walked one developer of equal seniority through using WinDbg once, because javascript he wrote was causing IE6 to crash. It was my first time doing it, so the role I had to play was the "let's just try things and see if we can get anywhere, there's no harm in failing" facilitator. Better to try something new than to give up. We didn't come away from the exercise with clear answers, but we had learned some useful information in the process by exploring.
Where there is most deafening silence in the area of proving correctness of the codebase and of the resulting binaries. seL4, klee, fp, and coq ran in the right direction with this in some aspects, formal verification still hasn't become a standard practice because the level of effort is still costly and tools to accomplish this aren't readily available.
Then you learn the tests are a monster stack themselves, often taking more effort than the actual features they are guarding.
Then you are part of one of these 'blameless postmortems' and realise everyone has an opinion and there are now 80 outcome actions, half of them complete horse shit but they are high priority now and jammed into your sprints for the next 6 weeks.
Funny, but quite true.
Remember the first weeks of coding C? It took me 4 years to not get cold sweat just thinking about using it when the compiler didn't inform you about a typo and you have to rollback the code all the time to fix problems and learn assembly (in X86, ARM and now Risc-V).
And that was without deadlines or any delivery pressure.
Today I realize C is for mad people, but I learned to respect the thing while no longer being afraid to use it in a real scenario.
Sort of "play to win" instead of "play to not lose"
I imagine confidently coding and doing stuff like:
if(count < 0)
panic();
if(count < 0) {
log_message("count is not supposed to be less than zero!");
error_recovery();
fallback_handler();
// etc...
}Incidentally, AI not having that kind of feelings could be a drawback for it creating code.
Even running away can get more error-prone with fear.
Funny, reading the headline I thought the article was going to be about the polar opposite. Over time, I've found the headline to be absolutely true, but I have also found that many invest in tooling and overabundant testing out of fear.
eg, "What if we need to recreate our entire stack from absolute scratch?"
I mean yeah, from that perspective terraform is totally useful, but what's next, automate the entire creation of the company, including customer acquisition and hiring? How often do you really need to recreate everything from scratch?
As a more junior engineer I used to have the confidence that things just wouldn't break, or at least if they broke it wouldn't be such a big deal. More often than not I was right. As a senior engineer, I find myself just pushing my junior engineer tenets. Usually everything will be fine, and even if it does break, we can fix it.
Write tests and invest in tooling if they're helpful. If they're not, don't be afraid to just write code by ssh'ing into a server.