Boss decides to add y,x,z options "just in case"? - YAGNI
Engineer wants to get the datamodel correct up front to avoid costly rework and data migration in the future? - Not YAGNI
Fowler fortunately states this distinction: "Yagni only applies to capabilities built into the software to support a presumptive feature, it does not apply to effort to make the software easier to modify."
The key here is avoid costly rework. The counterpart to YAGNI, is Avoid Painting Yourself Into A Corner. APYIAC. Actually, that is first and foremost, and YAGNI is secondary. How do we know this? Because we know change is universal and unavoidable. Planning as if change doesn't and will never occur is like planning without acknowledging weather.
This one is a bit like the TDD advocacy that says you shouldn't need to do much design work up-front because you can let the tests drive the design along with everything else.*
*Except for the part where you refactor your code, which by definition shouldn't be changing its behaviour and therefore can't be driven by adding tests to your test suite, where exactly the same design issues will immediately arise.
The key is the adjective, presumptive. YAGNI is about features that aren't needed now and it's uncertain whether they ever will be.
"Yagni only applies to capabilities built into the software to support a presumptive feature, it does not apply to effort to make the software easier to modify."
If we exclude features we definitely do want from consideration, so we're only talking about presumptive features, and if we exclude actually implementing those presumptive features, so we're only talking about "capabilities built into the software to support a presumptive feature", and if we then further exclude things that "make the software easier to modify", then how much development is really left for YAGNI to affect?
I can't speak for anyone else, but when I write new code, it is generally either implementing some specific functionality that we want right now or improving the common foundations of the system in some way. The usual reasons for the latter would be either gaining some immediate benefit, such as improved performance or robustness, or making the code easier to modify so we can then gain some benefit. By Fowler's argument, I don't see much of a middle ground where YAGNI would mean anything, unless maybe a developer is in the habit of starting to develop code for some specific feature when there isn't actually any requirement for it yet, but why would anyone do that anyway?
I've always taken YAGNI to be a cautionary principle about over-engineering and over-generalising software designs before there's sufficient expectation that the extra flexibility will justify the implementation cost. I'm not sure I've ever seen it used (or needed) in the context of someone just randomly adding arbitrary functionality because they felt like it, and it had never previously occurred to me that someone might restrict the term to only that scenario as Fowler seems to be doing in the quoted comment.
These aren't crazy features, but they're unnecessary and may never be necessary. As to "why do their managers and technical leaders allow it," often they are the very people who spearhead it. Or in my case, our team was embedded in a non-software company, and our leaders didn't understand exactly what we did, so they were hands-off.
There's a particular design failure mode where an emphasis on 'easier to modify' results in a codebase that's difficult to follow. You're making some future code change easier at the cost of all current code changes.
You can infer a concern about overall cost of modification into his statement, and claim that such a person is increasing the cost of code modification, but it's dangerous to leave it unsaid.
At the end of the day it's a matter of the quality of the tests. Bad tests can be worse than none (because at least you know to be afraid).
By comparison to Smalltalk, most environments seem broken to a seasoned Smalltalker. The main refactoring tool in Smalltalk would happily refactor your tests, so you'd just do your refactoring and re-run the tests. No interference with any reasonable implementation of TDD needed at all. You just counted refactoring as writing the tests and then immediately ran the unit tests.
There are other, more subtle issues as well. For example, you might want to add an iterator for something, even though your current task doesn't need iteration. However, you have strong feelings about how the iterator should work, so you add it because it helps you take control of the development. In those cases, it's easy for the rest of the group to say, "Thank you for that nice spike. However, that functionality is YAGNI right now. Let's take it out and we can discuss how it should operate when we're working on that feature". Basically, it's a way to stop people from bullying by intentionally creeping the scope of their work.
And finally, sometimes you honestly think something is needed and someone points out that it isn't necessarily true. It might be needed, but without getting more details it's hard to know right now. In those cases it's often better to remove the code as "YAGNI", keeping the design simpler. This allows it to move in multiple different directions -- some of which may obviate the need for that code. If you leave the code in, then it becomes a bit of a manifest destiny -- you write your code consistent with that extra complexity, even though you may not need it at all.
The above hopefully answers your actual question. Below is a very long discussion of my opinion of how XP development should ideally be practiced with respect to design. I hope you find it interesting -- or at least enough of a departure from what you've seen before to make it interesting for you.
Avoiding YAGNI, TDD and avoiding up front design all dovetail in an XP style development team, but my experience has been that the dovetail becomes a monstrosity if you are not actually doing XP. This is possibly why you seemingly have a negative opinion of it.
The main thing I tell people when I'm explaining XP is that "no big design up front" is not usually discussing the kind of design that they are thinking about (for example a week or so of trying to understand how to implement something and coming to agreements on design by drawing diagrams or writing example code). It's more about avoiding the kind of old-school design phases that lasted 6 months or more. When I was first starting out, we had minimum 18 month development cycles (and at one place we had a 5 year development cycle!). So we did 6 months of requirements analysis, 6 months of design, 3 months of development and 3 months of QA. But in reality, when you started development, you realised that the chunk you bit off was so big that it was basically just fiction. It turned into 12 months of patting ourselves on the back for being brilliant analysts and designers followed by 3 months of hacking and 3 months of absolute pure panic.
XP attempts to reduce the size of these phases and to distribute them more evenly into the development effort. It does not attempt to avoid them. With our old-school ways of doing things, our analysis and design was completely wasted because it was too disconnected with the actual development. With XP, you try to (as much as possible) marry the analysis, design and coding.
Of course that's usually not possible. One of my biggest problems with "Agile" teams is that they do so little analysis up front that they have no idea what they are building -- and it ends up being the same big hack we did in the old days. They avoid the panic because they just don't do QA and bury their heads in the sand thinking that their hack-a-thons are producing brilliant solutions.
Ideally you have a good 2-3 month backlog of stories that have been thought out enough that it only takes a day or so to implement each one. The key is that these stories are changeable and you "groom the backlog" at least once a week to see how your understanding of what you are building has changed since the last grooming session. By the time you commit to a sprint (very complicated discussion, but I like 2 week sprints), all of the stories should be implementable with only minor details remaining. While a "story" is a "reminder to have a discussion", if you haven't had the discussion before you started writing code, you are pretty much screwed.
Additionally, there is no way you can write stories of that size without having a basic understanding of the overall design that you are going for. Again, if you get to the point where you are going to put the story into the sprint and the developers don't know (or don't agree) how it should be implemented, then you need to insert a spike story instead.
This handles the "strategic" level of design. There is no "design phase". Rather it is spread out over development and updated over and over again. Any overall guidelines should be captured in a "coding standard" (which is not just a document describing how to indent your code, but rather a document that describes the overall "flavour" of design and the kinds of typical solutions that are preferred).
"Tactical" design is dealt with very differently, though. The idea is that code is design. It's just a very formal description of the design. Importantly, it's the only design that counts. It doesn't matter what your design diagrams look like if your code is not implemented that way. In most XP teams, diagrams, etc are usually considered transitory artefacts. The design is embedded in the code and you should write the code in such a way that the design is easy to see. That is not to say that you shouldn't draw diagrams, or utilise other kinds of design tools! It's just that you shouldn't attempt to persist them in any volume or (heaven forbid!) generate your code from them. The maintenance is just too expensive and error prone. Get used to understanding design by reading code -- you have to do that anyway.
The design aspect of TDD (if you are reading it as Test Driven Design) is a particular method. I could easily double or treble the size of this post and still leave you none the wiser. However, the basic thing to understand is that test code is very hard to write if you did not provide enough access in the design to write the test. Very, very often, as you expand the system the access that you provided for the tests makes it easy to access from the production side. Similarly the design is malleable because there is less coupling, though sometimes at the cost of cohesion. One very wise person (whose name I completely forget) suggested that a good way to understand how this works is to remove all global variables from your code. After doing this, start removing as many instance variables from your objects as you can. Once you've done that, consider ways of removing even more. Observe how that impacts your ability to easily write tests and to refactor the overall design of the system.
Although you always have to consider your strategic moves in design, coding (and especially TDD) is about tactical moves -- If I do X, then it allows me to do Y. At some point, you realise that this gives you an ability to move strategically. But that strategic decision is almost always better to discuss outside of the code (or in a spike), because it affects the whole team.
One of the main goals for XP style development is to end up with a design where it is easy to change direction. This is great for organisations where changing direction is important. Sometimes you are in a big stodgy company that literally doesn't care what the software does and never asks you to change directions. In that case, I probably would not worry about it so much. I've worked in groups like that and it doesn't matter what kind of lode stone you strap around your neck -- you won't get into problems. But in most places (and especially startups!) being able to change directions at the drop of a hat is extremely valuable -- sometimes to the point of rescuing a company. When you have discussions like "I know we asked you to do Y 3 weeks ago, but we were completely wrong. We need Z now. Is that possible?" and you reply, "Yep. No problem. Changing software is our job", then you know you've got it right. Getting it right is super hard, though ;-)
I think one of the reasons you don't see people talking about strategic ideas of design in XP is simply because they take it for granted. If you don't have a strategy for what you are doing, you are doomed. This is one of the reasons why I don't really like doing outside-in approaches, except if I'm spiking. Certainly, I consider advice to always do outside in as being exceptionally bad advice. It's a great technique to help you discover what you need to do, but I don't recommend it otherwise.
Anyway, I realised I launched a book off of your 2 sentence off-the-cuff comment, but I often find that people take popular misconceptions of good ideas and then treat the good idea as if it were the equivalent to the misconception. Now, I'm not going to say that my interpretation is mainstream at all. I'm also not Kent Beck, so I have no authority to say what XP is supposed to mean, but especially when you think an idea won't work, it's easy to choose the worst possible interpretation of it as being the most authentic. Hopefully this gives you food for thought.
Any decisions that can be unmade later with relative ease, should be made as quickly as possible just to get them off the board (opportunity cost). If Do X vs Don't Do X is reversible, then YAGNI can apply - do whichever is cheaper and fix it later if you're wrong.
What pisses people off is when people try to block conversation on an irreversible decision by shouting YAGNI at you. There are a lot of adjectives for this behavior and none of them are good.
Reversible Decisions+YAGNI is pretty much your answer to the bikeshedding problem. Instead of wasting hours trying to decide what color to paint the shed, just budget painting it 5 times, and get back to figuring out if you can find a cheaper vendor to provide you millions of dollars worth of ultra high quality concrete. Or whether that crack the geologists found in the bedrock means you need a different site altogether.
How much more expensive are options X, Y, and Z in the future compared to now?
How much more expensive is a correct data model in the future compared to now?
If it has a 50% chance of being needed in the next two years and will be 10% harder by then, YAGNI.
If it has a 50% chance of being needed in the next two years and will be 1000% harder by the then, do it now.