Hacker News Reader

>suspicious that Washington's culture could really accept a lot of the Agile and Lean notions that are commonplace in the high-tech world.

given my experience with thorough, to the letter, under supervision of no-expenses-spared-top-notch experts implementation of Agile/Lean/Scrum - 2 years non-stop across pretty much the whole company, only very-super-important-critical-projects got exception - the government would love it - a lot of planing, reporting, meetings and other crap boggling down everything with actual product development slowing down to the crawl as a result.

Btw, the original healthcare.gov - the one that failed and required that emergency super-team fixing - was developed using Scrum, and as result it had all the characteristics of a Scrum project, chief among them - bad, to the point of straight failure, integration between components.

I agree and share your cynical view of Agile/Scrum, at least how it's often used in top-down organizations.

In my experience there are two factors required for a successful project:

a) Developer autonomy sufficient to modify core design/implementation if necessary--and developers experienced and talented enough to recognize when this is necessary.

b) Developer ownership of the project. By that I mean minimal power difference between "stakeholders"/"business owners" and the developers actually building the product. You don't want developers to feel like consultants working on someone else's project because they won't care when bad decisions are made and they will naturally do the least amount of work possible to satisfy each Scrum ticket.

I share your suspicion of large-company Agile: by and large I think it's at best Mini-Waterfall, and at worst it's the same old bullshit with some new jargon on top.

However, the large amount of bathwater doesn't mean there isn't a baby there. And the places I see doing best at making the necessary culture shift are the ones with a sense of mission. So I have hope.

And honestly, even if this merely means a transition away from pure Waterfall to Mini-Waterfall, I think that will be a step forward. In the commercial world, the rise of the Internet has made 3-year release cycles untenable for most software; people now just scoff. So if we can merely get governments to emulate business in the shift from titanic failures to medium-sized failures, as a citizen I think that's a win. But still, from what I've seen, I'm hoping for more here.

>However, the large amount of bathwater doesn't mean there isn't a baby there.

Applying generic systems analysis (where it doesn't matter whether it is protein building, network protocol flow or software engineering process) one can easily see that Scrum is a highly synchronous low-latency process (that is the "baby" that management loves) which comes at the cost of dramatically decreased bandwidth. In addition, changing a team's process inside, Scrum indirectly completely changes how the team interacts with up- and downstream teams/components - in particular on practice eliminating any chance for early integration and for any iterations of the integrations.

That became obvious to me during the first day at the first Scrum course, especially after i tried to ask those questions to the teaching Scrum expert from the consulting company and received blank face with some blabbing in response. Some time later, different company with different consultants - that first hand 2 year experience i described in the previous post - the Scrum drawbacks on practice happened to be even worse than i thought they may in theory. We actually have great time right now at our BigCo. as after the Scrum fiasco, and quietly dropping the Scrum company-wide as the result, the management still seems to be kind of "in shock", and they haven't brought up any new fad in the process improvement so far, so we're just doing typical old-fashioned per-feature waterfall, and it just works as usual - which is huge achievement compare to Scrum :)

I can't really answer for Scrum, especially as taught by consulting companies. I think of Scrum as a set of too-basic but reasonably good ideas that quickly ended up mostly as a certification scam. I wrote more about that a number of places, including here: http://agilefocus.com/2011/02/21/agiles-second-chasm-and-how...

I think that's especially true for dramatic company-wide, top-down Agile adoptions, which I have never heard of working. I think people doing that cannot possibly understand Agile philosophy, which promotes local control and continuous improvement of top-down control and giant leaps.

My positive experience is mostly with smaller shops, but I've definitely seen many places where companies have grown up releasing early and often and have no problem working in what I'd consider good Agile style. Wealthfront, Etsy, and Spotify are all explicitly and openly like that. But I think there are plenty of other places that aren't explicitly Agile but ended up that way as a natural evolution.

For example, when I visited YouTube to study their process, there were a great number of developers, designers, and product managers all working in relatively independent small teams. They'd release at least weekly but sometimes more often. There was no overarching plan; each team had particular goals to pursue, and coordinated with other teams as needed. They never used Agile jargon and didn't think of themselves as Agile, but I think they were acting in ways that exactly matched the spirit.

I spent 7 years in Washington and can attest the change has been incredible. Civic-minded hackers, non-profits, and startups went from being outsiders to permeating agencies. Much was organic, and some was top-down mandated. Obama saw what happened in 2 elections and these new agencies are peppered with the people that used agile tech to great advantage on the campaign. Agencies like HHS and FCC were pioneers in open data -- not just for government, but for anyone. Look at cfpb.gov - an example of what an agency can look like when it starts from scratch. 18F and USDS are now tasked with bringing that innovation to the rest of government. If you can get to DC, consider attending the Sunlight Foundation's transparencycamp.org this May. It's the biggest event of it's kind, bringing 600 folks from across the globe.

EDIT: changed this sentence: "Much was organic and some top-down mandated" to reflect what I think was mostly due to popular upswell, though there were key executive decisions. The O'Reilly network has a lot of influential folks in DC too.

That's just the thing - I don't think 'playing it safe' is the goal in Washington, or at least not the only one. As is, the goal for contractors is to stay on a project as long as possible, earning more money over a longer period of time. Waterfall is a perfect excuse to do just that.

'Playing it safe' is also the goal of the gov't civilian managers of those projects- they have a promotion schedule to hit and having a 'failure' on their resume jeopardizes that. And since they're only going to be there 2-3 years, the goal is just to keep things propped up until they rotate out.

I agree that's an issue, but I was thinking more of the decisions of the people who hire the contractors. The right Agile approach to a project is to structure it as a series of small wins and losses, rather than rolling the bones on one big win/loss. It's my hope that calculus will appeal to career-minded officials, especially since the Agile approach lets you demonstrate value earlier (thus reducing drama), lets you bet more on your wins, and dramatically reduces the chances of large failures.

I had managers yell at me when at the sprint review meeting we realize we were going the wrong way and decide to scrap some of the work of the last few weeks...

Imagine politicians with a stake in the project's failure hearing that: they will go full Benghazi on it.

Better a few weeks than a couple of years and millions of funding.

Not necessarily in the eyes of certain people, and it happens in the corporate world, too. If you're only in charge of the project for a period of time (no longer in that political role, promoted, whatever), then "it doesn't need to succeed, it just can't fail" is a term used widely.

You don't want to be seen having failures in your project (even small, agile failures that are for the good)... but you don't care if the project succeeds.

If you can plan out a waterfall project that won't complete until you're "out of office", then there is no negative consequences for you... even if you KNOW it will fail once you're gone, that's the next guy's problem.

Sad, but true.

This certainly true, but it's my hope the culture will shift to make the waterfall option impossible.

Back in 2004 or so, I heard of a major company that only updated their website once a year. Even then that was considered ridiculous. Consumer expectations have shifted massively; plenty of places now release every week or two.

I have hope that the government culture could shift similarly, so that software projects that don't deliver anything for long periods are seen as troubled no matter how many Powerpoint decks and GANTT charts they produce in the meantime.

And I think the way to change that culture is by having enough projects that deliver early and often, so that people say, "Well, those guys are shipping monthly. What's wrong with your team?" So instead of keeping people from being blame-y, we get them to be blame-y about long project cycles.

I think that's definitely happened a fair bit in industry, so there's some chance it will happen in government, too.

Not in an election year.

If it's either a failed public project, or spending millions of dollars (that can be spun as creating jobs), what do you think they're going to take?

Just sprinkle some remote coders among their states, and they'll double down. Works for Lockheed Martin.

Ala "improving gov efficiency through smart use of tech"...I've worked in mil/Fed IT/telecom/technology-development for nearly 30 years. The problem isn't a lack of desire by the metal-benders trying to employ better/faster/cheaper tech, the problem is getting the Jabba-the-Hut career-protected SESs and their minions out of the way, along with instituting Fed procurement and contracting processes that actually facilitate a delivery timeline that at least keeps up with Moore's Law.

The challenge that USDS will need to overcome is one of culture and regulation bureaucracy, not one of throwing a unicorn cavalry at various IT dragons. You go pure "change from within", you'll fail. You go pure newbie/private sector, you'll fail. You need to find fresh and innovative eyes and pair them up with existing Fed ITers that haven't yet been jaded into submission.

Like you, I'm excited to see how it turns out. They need to succeed. If they fail, it will be a long-term failure, and that will do no one any good.

It should come as no surprise that Mr. Park is on board; He built two public healthcare IT companies (ATHN, CSLT). He can definitely recruit good tech talent.

The question is: Can he convince others in government that, as another post on the HN front page says right now, we should replace middle management with APIs?

Given that article was completely unpersuasive to me (as a fairly long-term HN reader) I seriously doubt it[1].

But then I don't think that should be a goal. (Good) middle management is precisely what Washington needs more of.

[1] The author would be well advised reading Peopleware, and understanding that middle management is often (usually?) an exception handling layer that handles the boundary between systems.

That's a great question. At 18F, we <3 APIs[0][1][2][3][4][5]. Not only are we big believers ourselves, but we strive to help other agencies improve their API practices.

[0] https://github.com/18F/api-standards

[1] https://github.com/18F/jekyll_pages_api

[2] https://github.com/18F/API-All-the-X and http://18f.github.io/API-All-the-X/

[3] https://github.com/18F/api.data.gov

[4] https://github.com/18F/fbopen

[5] https://github.com/18f/?query=api etc, etc