Department of Defense: Software Is Never Done (2019) [pdf]
media.defense.gov
media.defense.gov
I really like how they talk about "individuals of talent" in the document. Much of the DoD focuses around process, to great effect. The problem is that you can't take a process and put software drones on it and deliver software of any complexity. That goes against much (but not all!) of the DoD paradigm, it is interesting to see they might be changing.
Of course, they will meet great resistance, after all, the first job of a successful contractor is to extract as much money as possible from the government with as little work as possible. "Never finish a contract" was the mantra we had learned from the giant contractors.
Glad to be out of there, but it was a really interesting experience!
Agile software development is a tough pill to swallow for non-engineers. The lawyers and business people can't wrap their head around the idea that building iteratively could actually reduce bugs and improve quality - because the way they work is via process planning docs and checklists.
I’m including bug fixes here but nothing got stalled. We had a working iteration loop of Test -> Develop -> Test -> Review -> QA -> Release
Lots of automated checks, investment in infrastructure and good architecture made this possible. Ironically one of the smallest companies I ever worked for
Or did you mean some quality in particular?
Every other place always sings the praises of agile and/or scrum but it’s always just waterfall with ceremonies.
I feel like the collective of how software is developed evolved little from circa 2006
The notion that gets talked about at Netflix is “context not control” and I think it would go along way if everyone looked at software development management this way. Too many people try to control software development lifecycles instead of just providing engineers enough context to get their work done and trusting that reasonable people will make reasonable decisions
Not perfect but a more than above average way to approach the problem yet this kind of thinking or its permutations have not caught on at all. It’s still top down process oriented models
Nothing about Agile says, "Don't ever test and validate this system before deploying." Anyone who believes that is, frankly, a moron.
If you have twenty versions of a thing and multiple deployments to fix and fine tune it, that means to this person there are 20 plus examples of "mistakes" you made in building it. They were looking for evidence that the software was built with "poor quality controls" and ironically interpreted iteration as evidence of poor build quality.
I think one way to go about it is to just put the Agile bits inside a Waterfall.
Which is frankly how it works anyhow.
You can characterize early sprints at 'research'.
Later the sprints are just 'sub projects'.
I think people could be convinced that 'feature refinement' is a real thing. Put it out in the field, doesn't work the way we want it, and so it needs to be refined.
I know that I would hide 'Agile' to anyone outside the team and just talk in simple, easier to understand phases.
It's also possible to put checklists in those phases, and if you say someting like '75% features' at such and such phase and assume '25% of that 75% will be reworked' it's not irrational either.
But you are right, culture matters.
It would seem they don’t do a very good job teaching how to think about uncertainty, risk, and unknowns, which is what agile is good at addressing. If you want a suit to lose their mind, tell them they can only pick two of feature, quality, or schedule.
Waterfall seems a lot better when you have done something before and there's nothing to discover.
I've said this here on HN before, but the only Waterfall projects I've seen it work reliably were where:
1. The systems were small (in which case any method would work)
2. The systems were trivial (in which case any method would work)
3. The systems were well-understood by the people making them (in which case any method would work)
It does not handle large, non-trivial, or novel systems very well. Almost without fail they will deliver late, over budget, or deliver the wrong thing. This can be a win for contractors, though, who get to bid on the rework or sustainment contract for their own work.
But there was lots of boring bullshit in there too ;)
My (outside) impression has changed the more I've heard over the years, to the point that I assume the DoD and military are acutely aware that the drones aren't really interchangeable, and that at every level they cultivate people under them they know "get shit done" and hand the important projects to them. Those people in turn have those under them that "get shit done" they hand the important things to, and so on, all the way down to privates in the military. The important thing being that those people that get shit done are given a lot more leeway in how they accomplish things (which might later become doctrine of successful), and this is specifically how those orgs evolve over time.
Again, that's my outside perspective from someone that's interested whenever I hear it and reads/listens to history sources (podcasts,etc).
Within military troops, yes. That's an active goal, especially on the officer side where the whole career path is pointed towards increasing levels of responsibility and leadership. The problem is not in the troops, but in the program offices.
I was on a spectacularly failed Waterfall (big-W, they were explicit about what they were doing) where they handed the task of managing a large, multi-billion dollar software system (from the DOD side) to a 1st Lt history major. She had no preparation for the role. Software acquisition in DOD is, mostly, a clusterfuck. I felt bad for her because she legitimately did not know anything more than the basic legal and financial side of things, she had no technical comprehension and had to trust people (contractors) who were lying about the system status and quality (what was delivered, after she had moved on, was years late and had a fraction of the target capabilities).
Program offices are not generally composed of people who have real-world experience with software development. Remarkably they have great engineers, good finance people, but the software person (often singular, even in programs that are almost exclusively composed of software systems) is usually not from a software background or a straight out of college EE major (more often than not, occasionally a CS major).
Ugh, I can see that being a problem. I remember my practical experience in projects after graduating with a CS degree, and it was close to non-existent (a class in development methodologies without experiencing it yourself is just a survey of all the things you can do badly if you were to be in charge).
What they mean by that is that the person who invented the phrase "waterfall method" did it specifically to tell the world about something that did not work. In fact, even going back to the NATO software engineering conferences of the 1960s, they were discussing how development needs to be agile.
But sure enough, few big organisations got that memo.
However, where I am, there is a huge emphasis on "getting shit done" well ahead of schedule. I'm lucky to work with smart and motivated folks who together, work well and we are wildly successful.
My biggest gripe, shared among many of my coworkers, is the level of micromanagement. Multiple business analysts, product owners, stakeholders, SCRUM masters, managers, etc. who all share essentially the same job - delegating and scheduling work. It makes every project 10x more difficult and complex than it otherwise should be.
An example just from this past week: Project A: we are delivering a single product. Three managers, two SCRUM masters, multiple business analysts. We engineers have four different JIRA/task-boards to keep up with. All of them overlap, three of them are essentially the same board in different spaces. Each day we have 2-4 hours worth of meetings spread across stakeholders amongst those three boards where we regurgitate the exact same information.
It's repetitive, backward, and a time-waste. Some engineers are currently "revolting" by saying what I'm saying now - in the meetings themselves. We've had multiple blowups with management.
Having meetings and discussions spread so wide leads to a lot of confusion because us engineers appear to be the only conduit with which the analysts and managers are communicating. Often times, the different stakeholders will ask for the same features in different meetings, other times they ask for different or conflicting features which leads to confusion and the need to schedule more meetings to clarify as they are not communicating amongst each other.
For tight hardware integration, iteration on design isn't appropriate because the hardware design was locked in to begin manufacturing.
Quite often software life cycles miss the hardware rapid prototyping phase and so there's a desynchronisation of iterative design.
Within the framework of the hardware, yes you can iterate on the specific implementation but the contract with the external hardware and the timeliness to deliver was fixed.
Agile struggles without agile being applied end-to-end, and so you get agile/waterfall mixes.
In my experience, they can still work. In my case we have annually field deployed (and difficult to update) firmware. Firmware feature additions and changes are an up-front known quantity (even if they turn out to be insufficient for the in theater usage). We can't change them. That's the root where any agile approach reaches an external impact limit.
Our little bubble is agile in a waterfall world.
Imagine you have a group of illiterate managers who have just hired a group of writers to write some company policy documents.
You could have the managers (carve / draw / modern dance) their requirements in a waterfall manner.
Of course you could get one of the managers to sit down with the developers and tell them how the policy's work each day, embed with them Agile style.
However in the end you can't be everywhere, sometimes there aren't enough people in the company who understand what is needed, or even be sure that the embedded people agree on what is needed.
These are business problems - but they are also literacy problems. We have a literate society and so there are enough literate people to write policy documents to cover a whole compmay.
We need to make enough people who can code, so they can write the policy documents of the future.
The only thing I'm not convinced about is whether programming is something of a functional specialisation or if it will (GAS already?) become a literacy-like thing.
I mean, we generally think of numeracy as something widespread in the West for the past few decades, but the average person really does not do even basic arithmetic very well. Are we really going to do a better job of teaching people programming, or will we have to resign to the reality that most people won't learn it well?
The bit that's gets me is how fast does earth spin (without which sun rise is a bit hard to explain). Until it was accepted that air was just a thin atmospheric layer, it becomes insane to imagine we spin at a 1,000 miles an hour though a air medium that extends to Jupiter. How do you keep your newspaper still.
Kids today have Mickey Mouse cartoons explaining space rockets, moon to mars, earth spinning etc. We just accept it as true without trying to be clever about it
The same is true for literacy and numeracy - just having a society where you absorb character arcs, sub clauses, chapters, publishing, versions etc makes so much difference. Mathematical symbols were the names of Ed Sheeran albums.
Look at something simple like version control - the idea is fairly simple and hot is almost perfect for it, but oh god have you tried to use Sharepoint. Once you are in the WYSIWYG mind set, version control breaks horribly and most most business users should have this all the time but it's not in their world view. Explain it to them and they say "but how does your nreasoaper not flap around in thousand mile hour winds"
(#) Look for episode on the podcast "Subject to chnage"
Fully 25 years after computers-in-business, especially for any significant size of business, was virtually universal.
Humans don't manage businesses anymore. They are essentially all software managed. Business management is now really meta-management of business management software.
>>> Business management is now really meta-management of business management software.
It really ought to look like :
for each emp in employees_by_dept('accounts'): emp.getvacationplaennedthismonth
or something
That would be news to most tech companies, these days, that seem to be obsessed with hiring huge armies of inexperienced engineers, relying on the process to make them magically deliver code like they were masters.
I worked for a couple of major corporations, and got lots of waterfall. A the Japanese company, I quickly learned never to utter the dark majik phrase "A G I L E," as it conjured demons from managers.
Also, looking at that PDF, I realize that it is a slide deck.
That presentation must have been a soporific.
Also, you can edit your comments which is better than tacking on additional self-replies just a minute or two after.
Software can be considered done with respect to any given set of requirements. Now if the requirements change, then of course, the software may need to be updated - but even then, it should be designed in such a way that the impact of requirement changes are minimal. Software should, to some extent, anticipate requirement changes. Of course this is not easy, but to claim that it's impossible is to admit defeat. You need highly skilled people making those architectural decisions.
Some software is subjected to constantly changing requirements and these can be said to never be done but that's far from being the case for all software. Libraries and modules are software too and these should be generic enough to handle requirement changes without code changes since they should not concern themselves with the business domain.
I have completed open source projects which did not need to be updated in years and worked just as well. I've worked on other projects which had to be updated because of vulnerabilities found in dependencies but these did not require any code changes on my end.
I consider it a huge red flag if some library was working fine for 5 years and then people decided to change its interface in a non-trivial way.
I was tolerant when async/await was rolled out to major programming languages; this affected and continues to affect dependencies but it's a meaningful conceptual improvement and helps with future maintainability. I hope that programming standards organizations will slow down after that in terms of changes which affect function interfaces because these decisions have huge flow-on impacts.
It's totally possible to produce completed software (especially libraries and frameworks) but the industry has to aim for it and expect it.
The current goal of industry seems to be job creation, not the creation of robust working products. Accepting that software can never be done right from the beginning is going to be a disaster. It should only be tolerated in the context of requirement changes.
While a piece of software may satisfy its requirements, it will never satisfy them perfectly or optimally. All software contains bugs, unexpected failure modes, and undiscovered security vulnerabilities. I am convinced that there are essentially zero exceptions to these claims.
Furthermore, the environment in which software runs must change over time. This causes "bit rot" as (say) dependencies stop receiving security updates or platforms become unsupported.
Finally, there's the human factor. When software runs on old technologies, it will become progressively harder for new developers to understand. There will consequently often be a steady decline in available talent.
For these reasons, even if the requirements of a piece of software never change, it will always require ongoing maintenance to keep it running, much like a piece of physical infrastructure.
(edit: typo)
About changes in tech and environments. It's possible to build portable software. Whether or not software needs to be portable should be treated as a requirement. I disagree with any argument which suggests that it's not possible to foresee this.
It's not always the case that old technologies are harder to understand. This is mostly because software is still a relatively new area in the grand scheme of things. There are some old projects which I find easier to understand. Especially those built with languages which stood the test of time like C/C++, Java, Python, JavaScript...
/bin/true (humbly submitted as a non-zero exception to those claims)
Imagine Photoshop, they continually add new tools which let you get the intended results with less manual work. The end state of photoshop is you just open the file and it immediately shows you the finished picture you wanted without having to do anything. We are decades away from that.
I had a discussion the other day about how to tell when a web service can really be considered secure enough for somebody important to store sensitive data on (Eg, medical software); when do you say “the requirement has been met”? Does it become un-met next week when a new exploit is discovered in a package you rely on?
>Software is made by people and for people, so digital talent matters. The company I was at did not acknowledge this. To them, engineering hours were fungible. Two weeks from an engineer who is doing the bare minimum and coasting to retirement (in DoD contracting, there are lots of these type) is the same as a mid-level engineer who is up-to-date on the latest technology and working their hardest to push the project forward. The latter is usually paid less than half the former in the industry.
>Software is different than hardware (and not all software is the same). What works in other industries doesn't always work in defense. It's a unique space. CI? Good luck running automated "intercept a hypersonic missile" tests. This doesn't get enough attention. However, there is a lot that can be done in pure software testing, even for DoD applications. They view it too much like hardware development and that hamstrings their software efforts.
The bigger issue that is industry-wide that I noticed in my time there, is that the incentive system in the industry is broken. Contractors take an educated guess up front on what it will cost them to build a system for the government, and the government chooses from those bids, weighing capabilities, cost schedule, etc. The contractor who wins, then tracks their progress to completion by how many hours they have worked on the contract, relative to their total. This is how they get paid. For instance, if it is predicted to take 100 hours to deliver a system, and that system is $1 million, each hour would earn the contractor something like $10,000 (obviously there are other non-labor costs and this is a hand waving simplification, but to an extent is true). So, there is limited (bonuses that are tiny relative to the value of the contract) incentive to come in on time. There is a great incentive to over-hire, pay staff to sit around doing nothing, and collect money for it. This disincentivizes innovation, because you don't want to have a predicted cost that is astonishingly low for your next contract; you want it to be just low enough to undercut your next competitor. That's how you maximize profits, and us taxpayers get a terrible deal.
Oh god. Flashbacks to my time as a contractor to Aus DoD. Over 18 months I produced a small handful of xml schemas, a single java class that called out to their approved FTP library, and ate enough doughnuts to put on 20kg.
Feel free to ping me dale@1099fedhub.com if you’re curious about going down that path.
But for very small very well contained libraries I would argue that they sometime _are_ done (for many years, not forever; in a sense that any changes you can come up with add no relevant value at all, including no relevant perf. improvements or better maintainability).
At least if written in a language with nice forward and backward compatibility, cross platform support and a stable "base" ecosystem.
(So not C,C++ due to not default package manager/build system and too much divergence between platforms (e.g. unsigned long), not JS because their core build tooling changes to much (through it's getting better!), etc.)
https://en.wikipedia.org/wiki/Unix_philosophy#Do_One_Thing_a...
> As stated by McIlroy, and generally accepted throughout the Unix community, Unix programs have always been expected to follow the concept of DOTADIW, or "Do One Thing And Do It Well." There are limited sources for the acronym DOTADIW on the Internet, but it is discussed at length during the development and packaging of new operating systems, especially in the Linux community.
As if we software people didn't have overinflated egos already...
This DoD report talks about 100x developers existing (and of course HNers imagine ourselves towards that better-than-average end of the scale).
The DoD here even uses the (lowercase) term "special force", which sounds like we're not nerd typists, but super-cool elite operators.
That's one place you'll need some people who can find brilliant solutions or insights that most people simply cannot, no matter whether you throw 100x times the number of people at the problem.
But if they're having trouble attracting and retaining as many 1x-3x everyday software developers as they'd like, I agree, it'd be easy to imagine an attempt that fails miserably due to stereotypical bureaucracy.
Paying close to FAANG TC levels would be a start, since it would cover the monetary factor, and i think the prestige factor would quickly follow. (I bet few people think adtech, fintech, or yet-another-microblogging/messaging/profile-app are actually cool anymore -- only the money is what makes those companies prestigious.)
Maybe pay 80% of FAANG, so the FAANGs will still get the people who only want money, and you offer better non-monetary value to make up that 20%. (80% should be close enough they can afford to live in the same neighborhoods and send their kids to the same schools as FAANG, rather than be lower class that could upgrade just by grinding Leetcode for a few months.)
Then manage those units like they are very serious excellent product development organizations where everyone wants to work. And figure out how to do the culture better than all the companies that got too much into class thinking, and too much jumping through the hoops for interview and pursuit of promotions, rather than focus on the mission and team.
These were more of a template on how to develop, not a concrete recipe. Scrum then established a process where there was need for a family of processes, continuously evolving, ever changing.
The very fact that "you absolutely have to have a specific ceremony (i.e.: Retro when no topics)" is a red flag which imho puts the cart before the horse.
Why is there a "Scrum Master" anyway, if individuals are encouraged to interact and own the process? Suppose an individual wants to change the process, and reaches consensus within the team/community. Why would a Scrum Master / Manager even be able to veto any decision?
Why is there a Sprint when SW development is actually a marathon? Uncle Bob talks about this in several of his Clean Methodology books.
I don't think that Agile is dead [1], although it has been taken over by people eager to sell you a *cough* certification (Scrum or any other). It's not dead because its genius is that it's constantly evolving, and you can always interpret it differently since it's only a template. Scrum, on the other hand.. well..
For the future, I look forward to not putting an equal sign between Scrum and Agile, even though apparently Scrum is how you "do" Agile, according to the industry.
[1] https://www.simplethread.com/agile-at-20-the-failed-rebellio...
This situation often (IMHO) makes it much harder to deliver high-quality software. No amount of surface-level UI testing replaces the knowledge you get from actually using a system yourself.
Line of Effort A (Congress and OSD):
Refactor statutes, regulations, and processes for software
* [A1] Establish one or more new acquisition pathways for software that prioritize continuous integration and delivery of working software in a secure manner, with continuous oversight from automated analytics
* [A2] Create a new appropriation category for software capability delivery that allows (relevant types of) software to be funded as a single budget item, with no separation between RDT&E, production, and sustainment
Create and maintain cross-program/cross-Service digital infrastructure
* [B1] Establish and maintain digital infrastructure within each Service or Agency that enables rapid deployment of secure software to the field, and incentivize its use by contractors
* [B2] Create, implement, support, and use fully automatable approaches to testing and evaluation (T&E), including security, that allow high-confidence distribution of software to the field on an iterative basis
* [B3] Create a mechanism for Authorization to Operate (ATO) reciprocity within and between programs, Services, and other DoD agencies to enable sharing of software platforms, components, and infrastructure and rapid integration of capabilities across (hardware) platforms, (weapon) systems, and Services
Create new paths for digital talent (especially internal talent)
* [C1] Create software development units in each Service consisting of military and civilian personnel who develop and deploy software to the field using DevSecOps practices
* [C2] Expand the use of (specialized) training programs for CIOs, SAEs, PEOs, and PMs that provide (hands-on) insight into modern software development (e.g., Agile, DevOps, DevSecOps) and the authorities available to enable rapid acquisition of software
Change the practice of how software is procured and developed
* [D1] Require access to source code, software frameworks, and development toolchains —- with appropriate IP rights —- for DoD-specific code, enabling full security testing and rebuilding of binaries from source
* [D2] Make security a first-order consideration for all software-intensive systems, recognizing that security-at-the-perimeter is not enough
* [D3] Shift from the use of rigid lists of requirements for software programs to a list of desired features and required interfaces/characteristics to avoid requirements creep, overly ambitious requirements, and program delaysAnyone know if Congress has done any of these things in the last three years, since this report has come out?
I take issue with the title. Software is constantly "done" and then new requirements arrive, causing a new version of the completed software to be built. A suspension bridge is considered "done" even if it's getting continuously repainted.
Look at how the TV series industry is hard trying with cliffhangers or story telling arcs without a closure...
This issue runs much deeper:
It is how the economy operates: you need a roof to protect you, every day, you need to eat, every day, until you die in decades. Many economic activities, if done honestly and fairly, cannot justify a permanent income.
Chill down my spine.