Hacker News Reader

Yeah. I aint saying who I am, but I'll anonymously corroborate this. Nic had a lot of good ideas, but he micromanaged the hell out of low-level product decisions, forced us to use specific broken products that made me swear the vendors must have naked photos of him somewhere. He threw new requirements out of left field in the middle of demos that were not real requirements written down anywhere, but for whatever reason, we had to change course in the middle of development and do it anyway.

I'm sitting here right now on a Friday afternoon while the Air Force is on a four-day weekend ahead of labor day, trying to deploy to a broken ass application his DevSecOps reference architecture forces everyone to use, but it doesn't work because it uses a checksum algorithm disabled by FIPS-compliance hardening in this environment, which we have absolutely no control over. The biggest impediment to even getting this far was another vendor enterprise service he forced us to use that was broken until July. We were stuck just waiting on a bug fix to be provided. And, of course, we have to use Iron Bank container images for everything, but Iron Bank container images are themselves perpetually broken. They do security hardening, but no functionality testing, and their process of pushing breaking changes to the same tags can break you in production unexpectedly. And you can't pin to the actual sha because Harbor only holds onto five orphaned shas at a time that don't correspond to a tag.

He's touting a lot of accomplishments here that are accomplishments because pushing broken functionality and calling it done is a very easy way to say you delivered faster than a normal DoD program that has to actually prove it does what it says it does.

This is Nicolas. I'd welcome to get your proposed fixed so we can fix both Iron Bank and the other issues you mentioned here. Do remember that I'm not the one mandating FIPS validation. You know, there is a concept called cybersecurity.

The issue with FIPS compliance is with Anchore. It's actually fixed in 10.2, but that version is not yet in Iron Bank, so delaying deployment to IL5.

The issue with Iron Bank itself is a lot more structural. We report it every time a specific container breaks, and it eventually gets fixed on a report-by-report basis, but the base issue is they're doing two things completely wrong:

1) The process requiring disconnected builds in the container build stage to avoid pulling in dependencies from the Internet, combined with lack of expertise on the part of people assigned to the container hardening teams in terms of how to actually build software, results in them usually copying down the upstream official container, and making a naive copy of the desired executable from that container into a UBI base image. That happens to work for dynamically linked executables when the upstream and UBI have the same glibc version, usually right after a UBI release. Then it later breaks spectacularly, and when the issue is explained to Iron Bank engineers, they don't understand and simply call it fixed when UBI releases again and glibc is aligned by chance with upstream.

2) They push functional changes to tags. You can pull some <image>:8.4 one day, and the next day the exactly same tag will have different environment variables set, different paths, will have executables removed (notably, the jq image used to include aws-cli for some reason, which it shouldn't have, but once it did, you can't just remove it). Normally, the fix for this is pin to a sha instead of the tag, but Harbor doesn't hold onto the shas once you have republished to the same tag five times, and Iron Bank is continuously rebuilding those tags. We've reported it and Iron Bank technical leadership is at least aware this is a problem, but figuring out how to fix it has never been a priority for them.

It's interesting how often someone high up pushes a vendor solution that won't work, and when you explain that it is not workable, and why, (and not just you but basically everyone knows) you would expect them to say "thanks for discovering this critical flaw, we'll can them immediately and select a better solution" or, "we'll have a couple of architects look into that and report back" at least. But instead, they look at you like you just created an embarrassing problem for them and now you're the enemy.

One reason might be that they are receiving some kind of compensation from that vendor. A more charitable explanation would be that they want to cover up a big mistake by loudly affirming that it was not a mistake, and you are in the way of that. Either way, the fact that they don't really want to know whether it was a mistake, and if they know, don't want to fix it, tells you all you need to know about them.

>He threw new requirements out of left field in the middle of demos that were not real requirements written down anywhere, but for whatever reason, we had to change course in the middle of development and do it anyway.

To me this sounds like the fundamental difference in approach between "agile" and "waterfall" (and underscores his accusation of the DoD using broken "water-agile-fall"; which is more commonly called "water-scrum-fall", but I digress). You can't use an agile process for a project whose contract demands requirements (and all that implies) up front. It would require a complete revamp of the entire DoD contracting process. Which may or may not be something that can or should be done, but with the understanding that it IS the way it is, for a very good reason: to prevent fraud. Which was a painful learning experience going back to the very beginning of the Federal practice of providing national defense. The current system is NOT perfect, but you don't throw the whole system away in pursuit of "cost savings" or you'll attract every grifter and con artist out there into this business looking to scam every penny they can from taxpayers.

>He's touting a lot of accomplishments here that are accomplishments because pushing broken functionality and calling it done is a very easy way to say you delivered faster than a normal DoD program that has to actually prove it does what it says it does.

100%. But dang, wouldn't it be nice if we really COULD do all this? And given my limited exposure to P1, and the traditional delivery systems, it seems to me there's a very worthwhile middle-ground to pursue. It didn't seem he was interested in pursuing an incremental approach. It seemed like very much an all-or-nothing mission to him.

> It would require a complete revamp of the entire DoD contracting process. Which may or may not be something that can or should be done, but with the understanding that it IS the way it is, for a very good reason: to prevent fraud. Which was a painful learning experience going back to the very beginning of the Federal practice of providing national defense. The current system is NOT perfect, but you don't throw the whole system away in pursuit of "cost savings" or you'll attract every grifter and con artist out there into this business looking to scam every penny they can from taxpayers.

The thing is, at least for software dev, DoD projects don't know their software requirements any more than private sector ones do.

Plus, the current acquisition system is by no means more fair or less riddled with fraud. The WWII model of much easier acquisition regulations run by bureaus or offices with deep technical expertise and independent auditing did quite well.

Just because we intend to reduce fraud with a given set of policies doesn't mean we should assume that the policy actually works, or that the other problems such policies introduce are to be ignored as a result.

The only truly intractable problem I don't see a way around is the interagency nature of some of the requirements. In many cases, they aren't really requirements in the sense of being formally written down anywhere. It's worse in the IC than in the DoD, but present in both, where enterprise services meant to interoperate with systems run by other components have to go through a trial period after releasing to some sort of near ops environment. Every single involved stakeholder has to sign off on this.

Who can solve this? There is no common authority. Theoretically, that is what the DNI was supposed to be for, but they don't have any kind of IT expertise.

> He's touting a lot of accomplishments here that are accomplishments because pushing broken functionality and calling it done is a very easy way to say you delivered faster than a normal DoD program that has to actually prove it does what it says it does.

Is your experience that normal DoD software programs have to actually prove they do what they say they do? Because that has not been my experience at all in the Navy.

We work on something for years until political and leadership support for working without concrete progress can finally no longer be maintained, throw whatever we have at that point at the users, and then force people to use it because "we've already invested $XXM into it and worked on it for years and all the rest of our plans assume that this system is there."

My experience is admittedly entirely from the IC side before I came to Platform One last year. And yes, when it came to ground processing and dissemination of geointelligence products, we absolutely did prove we were doing what we were supposed to be doing, to an extremely exacting standard. I'm talking exact specs on false positive rates of cloud detections. Written explanations of pixel by pixel differences from the last release's test keys traced to an RFC. Verification that something worked equally well over cities, jungles, deserts, ocean, mountains. Sometimes getting dinged for ridiculous attention to detail stuff like accurate multi-tile pixel registration of a mosaic collection over Mount Everest, just to prove it still worked in the worst possible spot on the planet for our topographic earth models.

When you see people above asking "what are the examples where we're a decade or more ahead of the commercial world," satellite imagery is one of the places where we're a decade or more ahead of the commercial world. I don't know if you recall Trump accidentally declassifying a collection on Twitter a few years back and imagery experts skeptical that it actually could have been taken from space given the resolution. That is not even remotely the tip of the iceberg of what we can do. I would not have even believed what we can do until I saw it.

So I actually am used to being in an environment run by technical experts in physics and image science who know exactly what they want, write down clear, specific, and exacting requirements, and actually meeting those requirements.

But you know what? We had the infrastructure to do it. A complete clone of the operational system with a cloned production data flow, putting the next release through exactly the same load, so we could detect the differences and any bugs immediately. Actual continuous integration because there was somewhere we could integrate to. Then I come to Platform One and it's here's an AWS account and some buzzword tech products that say they enable GitOps and CI. Figure out how to use them and stand up your own servers from scratch.

And you know what? I'm a confident person. I perfectly believe I can do that. But not very quickly. And if my five person team is supposed to do design, development, test, operations, and maintenance all by ourselves, it is never going to happen quickly. Process can't save you from underprovisioning of resources. But apparently the government is just not willing to spend money any more. It's hard to see how. It's not like the federal budget is getting any smaller. But I have no idea where that money is going. At least some of that used to be going toward extremely superior radical technology that the public is skeptical of because it's all classified and they don't hear about it, but trust me, it's there.

Thanks, and yeah that makes sense. Until Kessel Run (and later P1) got off the ground I don't think I've ever encountered any team in DoD able to effectively use cloud or "DevOps", except from the Intelligence Community.

Even with P1, I would point to their rapid provisioning of MM chat as an example of where they were able to move quickly... but that was the P1 team fielding it, not P1 fielding a platform that downstream teams can jump onto quickly.

Even now there's been little point to trying to adopt P1 directly, as I understand it you're supposed to onboard through a separate software factory instead. And that's hard to do in the Navy lol (we're trying with things like Black Pearl to build on top of what P1 has done but that launched a year ago and there's still little ability to onboard new efforts).

> But apparently the government is just not willing to spend money any more. It's hard to see how. It's not like the federal budget is getting any smaller. But I have no idea where that money is going.

The federal government spends so much on IT that's its well into "waste, fraud, abuse" territory. I don't know where it goes either, but I think it's 20% labor of staff who are no-value-add middlemen, 30% on cybersecurity paperwork documenting the lack of security, 30% PowerPoints documenting the lack of progress, and maybe 10%-15% on the actual technology/cloud/development.

I'm heartened to see that someone is able to do better (even if they need the 'classified' shield to keep the technicals focused on technology) but I really wish we could hurry up on replicating the IC / Kessel Run model to a wider part of the DoD.

>> Other places, where it matters a lot it's like nothing you've ever seen or will likely see in the commercial sector for decades.

That's something I'd really like to see. How does that kind of difference come about? My guess is that it requires a certain degree of funding and commitment that may be impossible in wallstreet companies. But what else does it take for an organization to get there?

> Other places, where it matters a lot it's like nothing you've ever seen or will likely see in the commercial sector for decades.

Would love to more about this as it sounds like you are implying that their practices and technology are significantly ahead of the commercial sector.

> - Nic never held a clearance, and as a result was never actually involved or aware of most of the programs that he intended to impact

I honestly don't know how he'd ever expect to succeed in a job like this without a clearance. Without it, he'd have little practical knowledge of how the actual air gapped networks (SIPR, JWICS, etc.) are designed and implemented.

Even if he didn't know the classified technical details, there's a lot of differences between deploying software in these environments vs the wide open public internet.

From experience, the type of exploits that would be devastating to a big company just aren't a big risk for most apps on these networks. OTOH, classified apps have all sorts of bureaucratic rules that the unclassified world doesn't have to consider.

For example, if this guy is coming from Silicon Valley, he might spend 80% of his time worrying about exploits that cost corporate American billions / year and which are due to unpatched systems which become targets for script-kiddies pounding every IP on the internet again and again, looking to take advantage of dozens of known vulnerabilities.

On the classified networks, these types of exploits just aren't as likely. There aren't fleets of bots in China and Russia trying known root passwords on every machine with port 22 open or trying to access open Mongo or Elastic Search systems on whole IP ranges.

On the other hand, a private sector SecOps guy isn't even considering that they absolutely cannot store multiple big program's databases in the same data center, staffed by the same contracting company. Because there's a huge risk that some rogue IT contractor with general root access in some small signals intelligence "datacenter" in Hawaii or Germany will end up having access to way too much sensitive data that is shared by multiple intel agencies and military branches, and it'd be disastrous if he decided to start taking raw disks home with him each night a la Snowden. The Air Force and other DOD agencies certainly must ensure that a single contractor doesn't have access to too many data sources.

And with everyone moving as fast as they can to AWS, Azure, etc, I'm guessing these type of procedure are far more important to the Air Force than worrying about some development team deploying an app on an older docker image based on some old version of Java or Python with a few known exploits, which could cost a private sector company a lot of pain and money the minute it was deployed on the open internet.

I don’t think you want your airgap to be the only thing standing between your IT infra and total disaster.

Especially in the military it seems entirely plausible that bad people gain access to your equipment.

To be clear, I am not gone nor was I asked to leave. In fact, I was asked to stay. I do have a clearance as well and yes sorry I didn't let us get locked into Pivotal like you did and cost taxpayer millions.

Thanks for commenting on the thread.

Can you refer us to your opinions on the matters expressed in the article in a public friendly fashion?

The idea that he could fix Air Force IT in 6 months if empowered seems absolutely ridiculous given the size of the organization. What do you think the US gov needs to change to get better at it?

He did mention that's it's frustrating that officers put in charge of IT work have no IT experience.

That's not true for many other specialties in the military, where the officers will have extensive experience in the area they are leading.

Doesn't solve anything by itself, but it would help.

The fact is: you can't wave a magic wand, and suddenly crap out a person with high-level experience in these very young tools, who also happens to have officer-level rank in the USAF (or at the very least: experience with satisfying the operational and compliance requirements in this industry). I have no doubt such people exist. But not in the numbers that would be needed. It will take years or decades of focused effort to build that workforce.

Exactly right, but the military services haven't even started to implement the changes that would be needed to grow and maintain a cadre of digital talent.

Fred Brooks (a name some of you might recognize) recommended in a 1987 report for Defense Science Board on problems with DoD software that DoD needed to improve its uniformed and government software expertise. Even there he said such personnel would likely have to be heavily supplemented by contractor help -- but at least they'd know how to do the technical management.

Fast forward 30+ years and there are pockets of talent but if anything the problems are larger as digital products have become more important to business operations around the world.

Recent example from Reddit on DoD IT: https://i.redd.it/dyp2nyaca5l71.jpg

No, but they aren't even trying. They treat it as a generalist post. There are scholarships for pilots, nuclear engineers, and so on that place them in specific jobs in the military. That's something they could start now for IT.

I've seen much smaller organizations fail to adopt "DevOps"; pretty much textbook against the recent "Puppet yearly state of Devops report": cultural issues, and legacy software. It boils down to the organization SAYING they want to make this transition, and not doing what's necessary to change. "Sufficiently empowered" is a nice blanket term, but it's not always that easy. At my previous job, the "cultural" issue was specifically 3 individuals, who were highly placed in the organization, and they were #1, #3 and #5 employees from when the company was originally a startup. Nobody, not even the CEO, was "sufficiently empowered" to tell these fully-vested founders how to do their job or run their teams. When it came down to it. And they prevented a company of 500 others from succeeding at changing how they did the work. Now; we did hire a very good devops-oriented cloud architect, who was making a lot of progress, despite the resistance, and he may actually succeed (keyword: "strangler pattern"), but they're already 5 years into this effort.

So; indeed, the size of the USAF, AND all the contractor developers they depend on, and the command-structure of this combined "organization"; I wouldn't want to contemplate the level of "sufficiently empowered" that would be required to change that in 6 months. You'd have to summarily fire a LOT of people, and magically find skilled replacements, and then magically overcome the knowledge gap, while maintaining operational readiness for warfighters.

This is a very insightful and contrasting response.

Do you have any other articles/materials that we can reference for additional information related to this topic?

I encourage you to leave this comment on the article itself on The Register since you already made it public here.

Other places, where it matters a lot it's like nothing you've ever seen or will likely see in the commercial sector for decades.

It’s weird how the federal govt is like this across the board. Most things are “fine” being held together with bubblegum and duct tape. Some things matter a lot though, and when they do you get to see really smart people apply themselves in ways that are cooler than the movies.

Likewise I will take this comment with an equal amount of salt.

Thanks for posting this, the lack of a security clearance seems telling. Even contractors often have them. Also, a person in a real position of responsibility would never behave this way, and to be fair to him, they would never be treated this way.

^^ lol, political hit jobs representing in these comments

Of course Andrew did his best to get KR locked in to PCF which is now taking us 2 years to walk away from. Well done friend.