20.5 Years of XP and Agile (2020)
henrikwarne.com
henrikwarne.com
- had a manual build
- deployed by hand
- did all testing by hand
- never touched code except for new features
- might not even use source control
- comprehensive source control, or even total BOM control
- automated builds
- automated tests
- continuous integration
- continuous refactoring
- incremental design
What's left as purely "Agile" is the branded process stuff. E.g standardized meetings and titles like Scrum Master, SAF, etc. Given how prescriptive the business culture around it has become, it is a kind of anti-agile.
So death to Agile, long live agile.
Personally, I'm still an XP fiend at heart. Give me: a single list of prioritized work; a test suite I can trust that runs in < 15 minutes; and a dedicated team with good comms; and we'll tackle any problem you give us.
- We happily adopted a CI system from another team in 1998.
- We had a decent unit testing framework and a fair bit of automation for integration testing (although yes, some aspects of UI were hand tested periodically).
- We continuously refactored and improved existing code, especially if we touched something in it or around it. The only practice I think was poor then was mixing the refactoring and new work into a single commit, which I would never do today.
- I was using source control even in school in the late 80s, and I’ve never come across a team that didn’t, although I’m sure they did exist at that time.
- As far as deployment, everything I’ve worked on has been shipped on a very episodic basis (anywhere from a month between shipping to a few years), so most of the automation I’ve seen there from early on was about deploying internally for testing, and that’s always been more or less automated.
Although I did work at one place that tried Scrum for a little while around 2003-2005, the only practice I’ve generally seen adopted from that in the remaining years has been having periodic check-ins on status, e.g. monthly demo days to show off accomplishments from the last month. These are relatively low-prep although not impromptu, and that practice isn’t uniform.
With all that said, some things I have seen change, which aren’t mentioned by parent:
- Much more emphasis on developers writing tests. In the 90’s test teams were sometimes as large as development teams, and usually way behind the development team in progress. There was a belief that developers couldn’t possibly write good tests, or simply wouldn’t.
- I never worked anywhere where it was “docs first”, at least not to the extent that people mean that when they talk about older development practices. We did sometimes write the equivalent of a 1-pager to get everyone aligned on what particular work was about, and had a meeting to discuss and determine if everyone was in fact aligned. I think that was very useful, and sadly what I see more often than not now is “code first” meaning that you write code and then try to use code reviews to deal with everything, which means that what could have been an hour of writing something up and an hour meeting turns into someone spending days coding something that has to be thrown away because it’s so clear during code reviews that the approach is the wrong one. So now this “agile” approach means a throwaway work sometimes because people aren’t communicating in any meaningful way before writing code.
[edited formatting]
I'm happy you had your experience. But I will testify I've seen lots of places that would slap your hand if you touched code to "clean it up". When "testing" was two people running a week-long manual script, narrowing the scope of changes was a required practice!
As for developer testing: a core idea of XP is that we developers should take responsibility for the quality of the code we ship. Test-first was a practice to support that goal. Automated testing wasn't completely unknown, but many shops had zero automated tests. Or if they had automated tests, they were clicky-clicky tests run by a ui-automation tool, and were written by the testing team. Developer written tests were rare back then. I remember interning at MS in the 90s and how proud they were that their target tester:developer ratio was >1! The creation of JUnit was an important historical event for us.
As for the lack of communication and wasted work: I completely agree. Turning a story/use-case problem statement into a concrete design has to happen sometime. We used to do a whole-team tasking meeting for each story we opened, and group estimate of our tasks on the whiteboard. Combined with pairing, whole team rotation, and continuous integration, there was very little opportunity for someone to get lost.
I think one thing I have seen over the years is people making extreme assumptions based on experiences they hear via the grapevine, and seeing more perspectives seems like one way to counter that.
For example I have seen the claim here that CI systems were somehow new in the same timeframe that XP and Scrum were first written about and being adopted, and that testing was always an afterthought back then, etc.
If anything I think we were often building more reliable and higher quality systems in that timeframe, if only because there was more alignment that testing and fixing bugs mattered. Today I get responses like “quality isn’t just about bugs” if I am critical about the quality of a particular product. As if what users want is 2N as many features, none of which quite work rather than N features that are all solid.
So it was definitely not a general purpose reusable system.
I’ve talked to other people working at other large companies in the 90’s and they had similar experiences. I think either multiple people at different places recognized the need and value of systems like these before they were widely written about, or word was getting around in some circles. Not sure which.
Hudson came along later (2005?) But that whole scene started on CVS and jumped to Subversion. CI was well established when git showed up.
For me the essence of XP was always programming in pairs in front of a computer. That notion was somehow lost in the article and you also don‘t mention it.
I miss pairing. It is powerful on its own, and is shockingly effective when combined with the whole XP thing. But you can still get a ton of value from the rest to XP even with async comms.
This are CI/CD technologies, and perfectly fit into a waterfall deployment too.
From Wiki:
"In 1994, Grady Booch used the phrase continuous integration in Object-Oriented Analysis and Design with Applications (2nd edition)[8] to explain how, when developing using micro processes."
Cruisecontrol and XP were building on an already known concept. By 2001 hardware also became cheap enough to have a dedicated build servers.
The XP community sometimes give a distorted view of waterfall. It present waterfall as a dogmatic approach run by incompetents. In reality waterfall was a lot more advanced, with testing, build tools and a phased delivery approach.
One of the reasons people dislike staunch agilistas is because they try to lay claim to anything that turns out to be a good idea and deny everything that turns out to be a bad idea.
The Joel test was published in 2000, the very first question on that test is "do you use source control". It was known back in the 90's that using source control was a good idea.
https://www.joelonsoftware.com/2000/08/09/the-joel-test-12-s...
From Joel: “The truth is that most software organizations are running with a score of 2 or 3, and they need serious help”
XP and agile had absolutely nothing to do with the use, or lack thereof, of source control. At best you could argue there might be a correlation due to the sophistication of the shop itself.
I feel more strongly that continuous integration and continuous testing became widespread as part of the early agile movement. Again - none of those practices were invented there. Booch published his "Booch Method" in 1991, and Microsoft had monster overnight automated testing when I was there in 1992. But Beck wrote SUnit in 1995, and then JUnit when the Smalltalk diaspora fled to Java. XP was about evangelizing practices the authors had found to work and turning them to 11. CruiseControl came out of Thoughtworks in 2001, and the XP book literally coined the term "continuous integration".
I don't know how else to interpret this other than the list given are practices of agile, I responded because they definitely were not.
You may be right about the consultants bringing in better practices overall, but that goes back to my point that agilistas love to take credit for things that are not agile.
I would also argue that cloud was more the motivator for good CI/CD than agile.
At the end of the day it's all mixed together, but when you try to attribute something that's just good software dev to agile when it has nothing to do with agile itself you open yourself to the criticisms I've leveled.
To be clear - I'm asserting that the Joel test was part of the broader delivery agility current in the industry. This current sprang from the lessons of the first web boom, and the fantastic changes in delivery possible on the web. In the shrink-wrap floppy world, yearly releases left so much time to just wander around and get lost. Head would spend weeks not compiling, and savvy devs knew the right times to merge head into their long-lived feature branches. Joel was explicitly reacting against that, and to SEMA, RUP, and the 90s planning driven movement where sub-module teams would write against fixed apis that hadn't been implemented yet.
Maybe I'm exaggerating. I agree that he differed sharply from the XP and other agile folks in some ways, and would never have signed the manifesto. But who was Joel talking to when he wrote: "Another great thing about keeping the bug count at zero is that you can respond much faster to competition. Some programmers think of this as keeping the product ready to ship at all times. Then if your competitor introduces a killer new feature that is stealing your customers, you can implement just that feature and ship on the spot, without having to fix a large number of accumulated bugs."
Or Joel again: "The other crucial thing about having a schedule is that it forces you to decide what features you are going to do, and then it forces you to pick the least important features and cut them rather than slipping into featuritis (a.k.a. scope creep)."
You can argue that people were already doing these things, but clearly Joel thought there was an audience that wasn't (including early 90s Microsoft!).
Compare that to the manifesto: "Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage."
Anyway: Single-backlog. Ship at any time with a single click. High quality. Clear comms. Dev focus. Blah blah. It's not Scrum, but I'd definitely call that agile as we understood it in 2001. If you think that's just good practice, and has nothing to do with "agile", I'm happy to retire the word. Time to move on.
or guns should be credited with the advent of plumbing because guns were popular when plumbing became a thing.
The entire reason I pointed out that the joel test existed was to point out that industry leaders were actively trying to get people to become more sophisticated completely outside of agile. Why aren't you claiming agile is the reason people started creating timelines for software releases? Oh what's that, agile is really into NOT having those but releasing when it's ready?
It's almost as if different market forces were at play.
At some point you have to put your own biases to the side and realize that you do your own agenda more harm than good here.
Here's the thing: in some ways I think what we were doing back then was better. I just noted XP was essentially what we were doing anyway. Different teams and projects essentially chose their own internal methodology depending on what worked best for the project. Waterfall was what was presented to the project stakeholders.
What does that mean? Paradoxically, we actually had more flexibility back then! Thanks to the tooling, which more or less standardizes the methodology, and all the Agile Coaches which have been trained on that tooling and standardized methodology - Agile today isn't very, well, agile - at least not from a development perspective.
People point out that Agile allows for the requirements to change. No it doesn't. That's not a magic wand. Requirement changes still have project delivery impacts. Agile doesn't make that go away. It just provides for smaller iterations which facilitate the management of requirement changes. Know what? Iterative Waterfall, which we had been doing since the 80's (actual Waterfall is an artifact from the 70') also allowed for that.
Now I live in a world of WaterScrumFall, which is absolutely the worst of all worlds.
Now we have agile coaches (per team), agile leads (all teams' agile lead), scrum masters (the guy who does the meetings), and project owners (giving requirements), and lead project owners (planning) all trying to get developer time and attention.
Right size this, OKR that, attend this attend that. Wasn't agile always about finding what works best for the team and doing it, with consideration of tracking what works well and what doesn't, and refining the process to fix it?
Now it seems to be some sort of glorified job role, so even if the team get it right and run well they're patronised into attending non stop meetings from non technical staff trying to justify their jobs and whose own job role even states if the team is running well and self organising, then they need no involvement.
All I hear these days is, how can we fix that? And the team answers; the delay is off team, in either third parties, business itself, or we've worked around the bits that we can.
The dev team is now being asked to fix other teams' delay causing problems and then effectively silently shamed for not doing it week after week (because it gets raised again and again).
This is end game and wasting time if you're team reaches this point, yet the coaches still pound that shaming in like there's something that can be done in team. Of course these meetings aren't meant to do that, but when has human psychology ever been considered (being asked how to fix something a dev has no control over is now a common theme in questions I'm asked)?
IMHO agile in its current bastardised form should be taken out the back and shot. When it became a rigid set of rules (oh but "that's not how it's meant to be applied" is the war cry!) which promotes long term job roles preaching it, adds nothing to the teams' ability to adapt, and the team is already meeting most sprint goals and deliverables, the team has hit peak agility already.
If something is to hard to implement, maybe it is the problem.
In my limited experience, Scrum usually comes with a bunch of people who think Scrum principles come from some holy book and act accordingly.
You can't estimate how long your tasks will take because you are flooded by interrupts (coming from outside the Agile team or even from within)? Well, shame on you. Sprint review will show a decrease in completed tasks and everyone will wonder why. At the next planning you want to add stories without story points? The horror!
You are part of an Agile team and also have to do stuff for another team? Good luck closing your tasks without missing stuff that was assigned to the Sprint.
Daily meetings take more than 15 minutes? That's not how things should be!
Kanban is a much more reasonable approach, because its focus is on "flow" and not "whatever we decided three weeks ago without taking into account things outside of this team".
> Scrum usually comes with a bunch of people who think Scrum principles come from some holy book and act accordingly.
> You can't estimate how long your tasks will take because you are flooded by interrupts (coming from outside the Agile team or even from within)? Well, shame on you.
It sure has a book (in fact, many books), but that doesn't mean it should be treated as the <enter favorite holy book, if any>.
Before ITIL4, ITIL v3 (a famous IT service management framework) didn't mention Agile. Changes must be requested (including the actual configuration items you want to change), someone else evaluates them, if the risk is too high you need approval for a specific board, and so on.
Companies were following ITIL practices to the T, and some people said "don't you realize this whole thing is ridiculous in most cases?".
Cue to Agile. Sure, Agile itself promotes adapting to the environment, people not process,... But then Scrum with its rigid practices is treated as a sort of religion, leading some people to say "don't you realize this whole thing is ridiculous in many cases?". Pretty ironic.
> think the fundamental issue of scrum/agile is thinking one can estimate accurately in the small/short-term and do it continuously.
I agree. Doing stuff and estimating stuff are clearly different skills. And it is well known that many programmers estimate time by doubling, or even multiplying by 10, the time they expect to require for a task. So you have some people who vastly overestimate, others who underestimate,...
If you use agile, and it doesn't work for you, it's because you did it wrong.
It's the law of attraction, in software development.
As a _philosophy_, Agile gets those things done. It's fine.
As a _process_, as most places that "do Agile", it exists for it's own benefit and not for what it _does_. In almost all cases, it's a people problem more than a process problem, but no process will solve those sorts of people problems.
Well kinda. There’s wisdom in crowds (sometimes). If it doesn’t just naturally gain adoption then that has to count as a negative. Maybe the positives outweigh the negatives, but I think that’s arguable.
Dunno about the last, but I have gotten a bunch of value out of Beck's TDD book and Fowler's refactoring book.
Meanwhile Kent Beck [1]: "I get paid for code that works, not for tests, so my philosophy is to test as little as possible to reach a given level of confidence"
This is an absolute nightmare for retention and recruiting. When I went to university to learn about computers and technology I always dreamed of writing user stories and doing other mostly pointless clerical work.
It was never appropriate. It was created by consultants to sell consulting services. In that way, it's a huge success. As a practical development methodology, it's always been a disaster.
I found it quiet genius move from their sales teams. Because usually customer went well actually i could also use feature x, y and z that didn't made the MVP product.
This is not how it works. "All revolutions are conceived by idealists, implemented by fanatics, and its fruits are stolen by scoundrels" — Thomas Carlyle.
It was never for the developers.
Your first sentence became correct after managers and MBAs co-opted the term and turned it into "AGILE" which has almost nothing to do with "agile".
Maybe we should call it !AGILE ;-)
But the article highlights the good points of agile, and I agree with them:
- Iterative development (smaller steps with faster feedback)
- Unit tests (and building with unit tests in mind)
- Code Refactoring (which you can do confidently when you have unit tests)
The rest of XP/Agile is not necessary, and in some cases even detrimental. YMMV
But isn't this... OK? Extreme programming is probably defined as a certain set of practices, and if you are doing a subset of that, you are doing something else. This doesn't make you into a bad or a lazy person; it just means that your process is something other than extreme programming.
It's the same with scrum. It is extremely common for people to pick some practices from the scrum framework, skip others, and still claim that they are doing scrum. It gets very confusing when people do this.
Scrum had some serious problems, too, but nobody wanted to admit to them.
The former eventually leads teams to lowball everything to make sure they always complete everything at the expense of trying to accomplish more. Which will then lead the company to wonder why everything is so slow.
Getting people at the top to understand that challenge is the hard part.
My point being: other aspects of companies and teams are far more important than the latest fashionable project management methodologies/frameworks/philosophies.
I find it hard to care about these trends.
If you have an approach that forces the people doing prioritizing to weigh Return on Investment (benefit or value / estimated time) to largely guide the priorities, busywork with little benefit like you describe won’t be prioritized.
It’s critical.
And if people mean different things by the word scrum, it is rather hard to tell whether they are doing it properly :-)
Scrum is pretty difficult to do properly (the famous "easy to understand, but difficult to master" formula; although I would argue that it isn't that easy to understand either, at least not something that someone unfamiliar with it can understand in a day or two). It requires certain changes within the organization, which few organizations are willing to adopt.
I am seeing problems left and right. Mostly because people often copy the activities (they would often even call them rituals or ceremonies, as if to make it even more obvious that they are treating them as mysterious quasi-religious practices) without understanding what they are intended to achieve.
For example:
- Lots of daily scrums in which people go around the circle saying what they did yesterday and what they are going to do today. No sense of a common sprint goal, and no indication of collaboration in reaching it.
- Lots of sprint reviews that are just demos
- Lots of sprints without goals, that are just timeboxes to finish a certain number of tickets/stories
- Lots of teams that remain hierarchical, and instead of a real product owner have some kind of a middle manager (who might also assume scrum master responsibility, because he is manager)
- Lots of teams focusing on story points, velocity, and estimation
- Lots of teams that don't adapt previously formed plans to the emerging reality
I remember reading the scrum manual but I don't remember it saying anything about it being necessary to show esprit to corps and a "collaborative spirit" during standup.
It did say how to conduct a standup ceremony though...
>Lots of teams focusing on story points, velocity, and estimation
Well, they did make these things a part of scrum.
There is no such thing as a perfect set of rules or practices - only things that are useful in certain contexts. It is the practitioner's task to decide which principles apply, and to what degree.
Saying "You're not doing [brand new popular thing] unless you do it this exact way" is a trap laid for neophytes who lack the experience to discern that this is just arrogant bullshit, denying them the exercise of their own brain to judge what applies where (and learn from their mistakes).
Which is fine. They defined a set of practices, called it XP, and if you don't do these, you're not doing XP.
If the other person and I are on par with the environment then we're more likely to have a discussion about what we're wanting to achieve and how best to go about it, but then one of us will go away and knock out a first cut for the other person to review.
I do wonder what differs between groups of people who can actually do pair programming and those that (like me) that can't.
Occasionally I do side-work if Im not driving - e.g. look something up, message devops, a very small PR etc. so the driver can maintain focus.
I find it to be much more effective that working alone simply because so much stuff that would take me 20 minutes to to go down a rabbit hole ends up being caught by an extra pair of eyes in 4 seconds.
And then I moved teams... And realised it's all about the people and culture around you.
I probably wouldn't sign up for it again when joining a new company, because at its worst it's absolute torture, but I still believe that at its best there's nothing like it.
Personally, I'm not willing to get rid of _all_ my comfort zone to get something out the door more efficiently, where shareholders and C-levels get rich off my labor, and if I'm very lucky, I get to keep doing that.
I like working in teams with ownership over their outcomes, and reward for their success
Second, there's a chemistry element. When there's good chemistry it can be quite effortless and the chemistry can grow - as you start to develop an intuition for how the other person thinks there's less communication and mental jostling.
I've had remote pairing sessions that have felt almost as fun as online gaming.
It was high-energy, though, and an extended period might well have been fatiguing.
To me, this is one of the worst ways to work. I very much prefer to communicate in writing, and at prearranged times. To me, the idea that there's another person working with me on the same problem at the same time is as absurd as having another person cooking in the same kitchen with me, or painting on the same canvas as me. Whatever result I can produce on my own, it will be ten times worse if I had to do it with someone else working on the same problem with me.
The original Manifesto was actually about being "Agile", developers being developers. How to really focus on being the best developer. Moving fast, focused, cutting away un-needed tasks.
I'm continually disappointed that it spawned some giant consulting industry that became an onerous layer on development, a wet blanket.
For example, if testing is good, then it must be done to the extreme. Basically this means you will need to have a 100% test coverage to comply to XP.
But history proves that extremism and fanaticism are bad and harmful in every area of life, including in software development. You can, for example, read about the consequences of testing to the extreme here [1]
> Any program feature without an automated test simply doesn't exist. Programmers write unit tests so that their confidence in the operation of the program can be part of the program itself. Customers write functional tests so that their confidence in the operation of the program can become part of the program, too. The result is a program that becomes more and more confident over time -- it becomes more capable of accepting change, not less.
> You don't have to write a test for every single method you write, only production methods that could possibly break. Sometimes you just want to find out if something is possible. You go explore for half an hour. Yes, it is possible. Now you throw away your code and start over with tests.
[Emphasis mine.]
I can see how this is interpreted both ways, so I'll leave it to you. For what it's worth, I found the book to be an easier read than most books on coding practices.
[0] Extreme Programming Explained by Kent Beck, Chapter 10 under "Testing"
However, the framework needs updates in several important areas: operations, data science, end of the PO figure and incorporating other actors as part of the feedback cycle, tech debt management, how to deal with uncertainty more efficiently, and so on.
Unfortunately due to a lot of issues in our industry, agile got a lot of bad rep plus a lot of luggage it turns very hard to have a dispassionate discussion that the IQ of the participants does not drop 20%, at least. It's a shame, but I really would like to see some updates on this.
Around 10 years later, some methodology came along and working with the users ended. This methodology was suppose to give the programmers more time to work. We ended up with frustrated users who started moving to Lotus 123 or Excel. Back then I developed (on the side) many processes where my users could get the data from the mainframe for use on their PCs.
Agile came along later with prepackaged software, AMAPs(?), SAP, Oracle and many others that I have forgotten. Now we have many many users spending at least 50% of their time (some much more) testing upgrades instead of analyzing their data. End result, users are just doing what they can to get by whatever GUI screen is presented to them, not caring at all what data they are entering in. And most real work occurs in excel to the point where users are downloading gigs of data to their PC to get away from the canned, hard to use Enterprise Systems.
Instead, the bureaucracy did what does when it is threatened, it pretended to pay lip service to the agile principles while subverting them in order to survive and thrive.
Bureaucracy is just the human political animal expressing its awful nature in organizational systems.
Instead, the bureaucracy did what does when it is threatened, it pretended to pay lip service to the agile principles while subverting them in order to survive and thrive.
Amazing how Orwell never ceases to be relevant.
Also Scrum as outlined by people like Jim Coplien would probably be kind of fine for certain set ups, but only a few agile coaches, POs or SMs know Scrum that well, instead it's a weird mix of apocryphal, illogical practices and superficial knowledge that, coupled with an inexperienced dev team, is similar to a giant Molotow cocktail that will sooner or later explode.
Scrum itself is like the rules for holding a good meeting. It's good common sense advice that I don't think you can actually reasonably object to.
If it's not in the Scrum guide, it's not Scrum, truly find something objectionable in that: https://scrumguides.org/scrum-guide.html
But in achieving that flawesseness, they made it so broad and vague that you really need to add additional practices to it. And then you got to be careful, mature, and thoughtful or else it's gonna turn into a shitshow... but if you just do Scrum you'll end up as a shitshow anyway.
PS- to me personally the biggest flaw is that so much hinges on the definition of done being correct, and that is at least half the battle in dev and can almost single handedly save or sink an organisation in itself.
With Agile, at best, you will build an amateur treehouse.
Sure, prototypes that test the viability of certain parts and aspects of the system are very useful. But having those prototypes and toy models then be “free-styled” (without a clear design document to be referenced - what exactly is the system supposed to do?) into a repurposed role to deliver the whole system’s functionality and reliability is the recipe to disaster. Or in other words: technical debt.
- Where’s the Systems / Systems Integration perspective?
- Where’s *Repeatability*?
(middle management does like Agile to justify their many times useless positions)
PS: I believe “Agile” explains a lot of the 737 MAX owes…
IMHO following any sort of methodology dogmatically is unlikely to give great results, and the best path is probably somewhere in between two extremes. Waterfall emphasises "plan ahead" --- a concept that seems to have completely disappeared from some types of software these days.
What do people consider as the current best dev methodology?
I think both are very naive ideas (as I think overall about Agile and XP). But, before I get to the "why" part, I want to first describe my perspective on how testing changed over time.
The perspective on testing before the "shift left" happened was that testing is supposed to be adversarial. I.e. it was important that the testers don't know how the code works or what decisions did the development team make to solve higher-level problems the software was meant to solve. This was important so that testers wouldn't turn the blind eye to the software problems they would've otherwise gotten used to, or would have accepted the trade-offs before even getting into testing. The reason is very similar to how in clinical trials the participants aren't told whether they get placebo or the real drug.
This is also why the testing would normally start after the software was in a shape ready for testing (i.e. after a reasonably long development interval). This is also why a lot of it was manual: there wouldn't have been time to develop utilities for testing since the testers didn't know what the software was like.
But, of course, there were also commercial incentives to finish earlier. This is how "patch version" became a thing for example: a typical testing cycle would be to receive the system from the developers, to deploy it, to validate it (i.e. to ensure that the minimal required functionality is there), and then to try to "break" it. If the system broke, the developer would be notified. They then would have to fix the problems and deliver the fixes as patches. These patches then would be given a version number, and the testers would be required to use them to patch the deployed system in place, in order to continue testing.
A system shipped to the customer would usually contain a bunch of patches, and the installation process would include patching as well.
The downside of this system is the time it takes to get to the working product. The upside is the better overall quality of the product. The better quality comes from the testers working from specification rather than from their familiarity with the code.
----
So, unit tests. From the perspective of someone who had to manage CI for rather large organizations: unit tests give the least value for the most effort (among different test types). Because unit tests deal with the minutia of the code details they tend to change a lot, they grow stale very quickly, and they require a lot of maintenance because they tend to also multiply very quickly. Besides, most of the unit tests in every CI run test the code that didn't change anyways. They just burn electricity for no effect. It's very hard to come up with the system that would only apply unit tests where the code changed.
In the end of the day, given the limited time and resources, if there's any test a developer should be spending their efforts on it's the end-to-end test. They don't change as often, and they at least can guarantee that the program, as a whole, works sometimes (unit tests have no useful guarantees about the whole program).
----
When it comes to designing for testing... it's also complicated. There are many dangers in doing this. Testing code is often a security backdoor. Testing code can also cause performance problems. There's a very delicate balance between the ways of how the system can be instrumented for testing and how much this instrumentation actually contributes to the quality of the system. My approach here is that as much as possible, the testing needs to be external to the system. This is to prevent tests from affecting the system, and especially from leaving undesirable artifacts in it.
----
Agile, or any other simplistic software development management process is just that: simplistic. It doesn't mean it "doesn't work" -- it means that it works sometimes. But, I wish we spent time on studying the process and discovering better way of doing things instead of trying to shoehorn things we meet in practice into existing simplistic process descriptions to justify their existence.