Why I fought for open source in the Air Force
opensource.com
opensource.com
As noted in the article, the biggest things for them were being able to review security, fix bugs, and affect the roadmap. While I didn't add their requirements to the core system, I added a number of hooks and interfaces they specifically requested.
For contractual/legal reasons, they had issues around making the fixes themselves or even providing them to me but I received a lot of "highly specific explanations" to issues and approaches on resolving them.
Getting started with Hanscom's team was painful on the contractual side but excellent to work with once things were rolling.
The "DoD's clarifying guidance" he's talking about is almost certainly this[1]. It's got some good information about open source in general, even if it is a little dry.
If you can prove that there is support and training available for a piece of software and you can get it approved by your local information assurance officer then you can put it on the Air Force network. As far as getting it approved by your local IAO; let's say you want Python on your computer and the IAO says no then you can point to your local Redhat server (which comes already built with Python) and point out that you have already accepted any risks associated with Python by using Redhat. Another good resource for getting management to buy off on any risk is the EPL[2]. If the software is on that list then someone else in the Air Force is already using it, and the software has already gone through an approval process.
Edit: I just realized my second link might be CAC only ... hopefully it will be useful for any contractors/military folks that browse HN.
[1] http://dodcio.defense.gov/OpenSourceSoftwareFAQ.aspx [2] https://cs3.eis.af.mil/sites/afao/Lists/COTSGOTS%20Software/...
Also, doesn't the quicker timeline of updates/patches/releases make FOSS harder to verify? Is the regulatory landscape changing at all to accommodate this?
Also, I find the 18F Open Source Policy [1], which I saw in a recent HN topic elsewhere, seems like a good "default stance" for government use of open source and reiterates some of the same points in the article, and maybe we'll see some of that attitude prevail in the end.
[1] https://github.com/18F/open-source-policy/blob/master/policy...
They largely circumvent a lot of the process with help from policy makers at the highest levels.
Even then, I'm pretty sure they don't offer hosting/authorizing deployments from party sources. So they essentially have a monopoly on the ability to use OSS freely for bulding government websites.
According to https://www.govcode.org/repos, there are about 1,300 U.S. government repos on GitHub that do not belong to 18F.
If you're generally interested in how we do ATOs, https://pages.18f.gov/before-you-ship/ato/ is a nice internal guide for our devs.
(I work at 18F)
Establishing an ATO for end user systems, specifically hardware/software that is intended to connect to the military tactical networks comes with a very high barrier of entry.
Being an OCCFLD made up of media specialists (ie photographers, videographers, illustrators, reproduction specialists); they have requirements for hardware/software that fall outside the generalized platform/support provided by G-6.
Overwhelmingly, they prefer Apple/OSX but building Apple-based systems that support the wide range of security scanning, observation, and auditing requirements simply isn't possible. AFAIK, nobody has established a baseline for Darwin and the cost/risk to do so within the schedule of the systems lifecycle is too high. As a result, Windows is only option.
The systems I supported directly included laptops running a custom configuration (ie Adobe, etc) as well as tactical deployable media production studios that provide print, media, large format, networking, and archival capabilities in the field. At one point we added VLC (ie due to it's superior playback format support) but had to nix it after discovering that it had a CAT I CVE.
Things may be different outside the DOD but for the USMC specifically, MITSC are the 'gatekeepers' for the Marine Corps when it comes to establishing an ATO baseline and meeting the requirements for ATC (Authority to Connect) to the MCEN-N (ie unclassified tactical network). Frequent scans, remote access, and auditing aren't recommendations so much as an absolute baseline requirement. Don't get me wrong, they do their job very well.
When it comes to fielding COTS systems, lifecycle sustainment is a huge problem. Depending on Windows based systems and Dell Enterprise networking hardware makes the problem much worse. Unfortunately, that's what everything is standardized on and systems that can't be bought with a long-term support contract are no-go. Diverging from the standard is a recipe for failure.
I didn't deal with the ATO process directly. Instead, I provided field support, user feedback, and maintenance of the systems.
I established a distribution channels and did my best to provide periodic updates to all the systems within my AOR to meet the requirements of the ATO/ATC. I even spent months trying to hack the bureaucracy -- with little success -- to get the systems connected and operating at their full capacity.
Throughout that process I discovered numerous missing prerequisites buried among a morass of policy and documentation. I'm no stranger to reading dull/dry specifications but the literature surrounding ATOs and their application is a minefield. One misstep can lead to the loss of certification so it's wise to be cautious.
I tried to raise the issue with my superiors but as the saying goes, 'shit flows downhill'. It's not really their fault, they don't know what they don't know. I've learned not to take it personally, life of a contractor in the field is one where you'll inevitably (and hopefully infrequently) accept fault for failures beyond your control.
What really bothers me is that the Marines have to go forward with limited/degrading technical capabilities and little/no support. No amount of effort, willpower, or policy will solve significant technical problems that require technical solutions.
-----
Just to be clear, the following isn't specific to COMCAM. If anything the leadership I had the privilege to work with made an exceptional effort to always do things the right way, despite some of the ridiculous barriers we had to face. This is based on a wider perspective gained from working with many organizations across many military installations.
The DOD doesn't suffer from technical debt so much as technical poverty. Any benefit I had from being good with technology going in became a weakness. The military is a black hole of communication and information sharing. Don't get me wrong, it was an amazing opportunity to 'level up' my people skills but seriously...
I often see people complain about rampant waste and the lack of transparency in government and the DOD specifically. I can honestly say from first hand experience, it doesn't come from lazy or malicious behavior. When limited time/resources restrict decision making to a gray area between doing things the 'right way' and accomplishing the mission, it's the duty and responsibility of our service members to choose the latter. Waste and opaque behavior is a symptom of systematic failure (ex lack of knowledge, communication, unnecessary bureaucratic barriers, etc) at a much grander scale.
I probably have a unique perspective. I went in with no prior military background and a strong technical background. The former makes it very difficult to accept 'broken' as norm, the latter provides a unique perspective of what can be done to fix it.
-----
I'm a huge proponent for OSS, having contributed and/or authored many projects. From what I've seen, breaking new ground and introducing OSS to the DOD on a larger scale is currently beyond reach.
There are pockets of exceptionally talented/capable individuals with the technical background required to make it possible. Except, they represent an extreme minority and are already over-taxed with supporting the existing systems. I'll spare you a the diatribe on why the alternatives consistently come up short.
I have read all of the literature 18F has to offer (except a few blog posts). 18F sets a very good example for others to follow. Unfortunately, the platform is limited to internal use and the organization's capabilities are limited to serving the latest 'hot button' issues within the DC political spectrum.
Much like SV represents an extreme monoculture fueled by exceptional technical ability and a flood of VC funding. DC represents its own extreme monoculture fueled by the concentration of power/influence and a flood of taxpayer funding.
Beyond the bubble the state of things is much messier, broken, and in need of help.
> Even then, I'm pretty sure they don't offer hosting/authorizing deployments from party sources. So they essentially have a monopoly on the ability to use OSS freely for bulding government websites.
That's not quite the case. Many agencies use a ton of OSS to build websites and other services.
https://github.com/cfpb/source-code-policy https://github.com/uscensusbureau/open-source-policy https://github.com/18F/open-source-policy
The policy links above refer to those teams releasing their own code as open source, but you will find huge amounts of outside open source code being used there, and in many other agencies.
The Department of Defense is famous for (in 2003!) making sure their department knew that open source software was A-okay to use, and people refer to their amazing FAQ all the time:
http://dodcio.defense.gov/OpenSourceSoftwareFAQ.aspx
(I'm an 18F employee.)
"We'd like to buy your web browser."
"Well, we actually give that away for free."
"Yeah but we need to buy it."
It's 2 hours long, but worth a listen if you have the time: http://www.imore.com/debug-19-wil-shipley-next-delicious-mon...
In the UK and Canadian military, there is no single overall software person, although there are technical authorities who define standards, etc, that should be used when software is developed.
FWIW, the UK Ministry of Defence actively supports the use of Commercial Off the Shelf (COTS) software. This isn't always OSS, but is at least better than building everything from scratch. A good ongoing example is the use of 'serious games' to support military training, e.g. as image generators inside simulators or as standalone desktop training packages. There are various COTS packages that fulfill both of these and other roles.
[EDIT: Typo]
By 'general purpose' I mean the baseline configuration that all non-POR (Program of Record) systems use. Ie, hardened Windows, MS Office, security monitoring, CAC authentication, and DOD certificates. There's no chance in hell that the 'general purpose' configuration will ever be changed to include OSS because it's intent is to be the 'lowest common demominator' of connected systems.
This person probably works as a grunt for the AF branch equivalent of G-6.
The COTS systems you're referring to are POR (Program of Record) systems created and supported by the product development branch. I think for the AF, it's the Air Materiel Command. They fall outside the scope of general systems support, therefore they require their own infrastructure for maintenance/support.
Source: I used to support a POR system.
The A6...?
But even though all new software produced for government should be FOSS, there is little movement because all the small authorities buy licenses at a speed and cost too low to allow productive n of new software.
A coder must develop the code first then try to sell it. All the risk is on the coders, but they won't get paid unless they sell licenses - which defeats the whole point
In short if there are 60 councils in the UK who all want a new system for one of the 2000'legally obligated services they must provide, they will generally ask for a license of say 1/10 of the cost of production. If they all do that then the UK overpays by six fold.
If the first one went FOSS and ponied up the money the UK saves six fold.
However no council will pay for everyone else's free software.
So we keep over paying
"""The Council would like to hear from organisations willing to share information about their e-recruitment systems to gauge the likely level of interest in the project from the market. This will enable the Council to gain a better understanding of the systems available, the ways in which they could be supplied, an indication of the likely costs, and help determine the most effective way of packaging and scoping its requirements for any future procurement opportunity. As such the Council is undertaking this soft market testing exercise to engage with organisations and share information."""
So I am looking to build some forms of mutualised contract where the first authority to want finds say nine others and agree upfront payments.
Anyone with ideas pease shout
One solution is to have everyone agree to put limited resources on a shared project that they will all use, but this has problems. Many projects fail or don't go as well as hoped. The distributed nature of it may increase the possibility of failure and the politics as well.
Another option is to get everyone together and agree on specs. Then outsource it with the requirement to the provider that the result be open source. After that first development effort you can all collaborate. This should go over better in government than in industry because your different groups aren't competing with each other.
As, phkahler said, "Nobody wants to go first because they pay for everyone". That is the precise issue that game theorists call the "snowdrift dilemma" and that Snowdrift.coop is working to solve.
This sums up one of my major frustrations of being a programmer for the Air Force Research Lab(AFRL).
You have branch and division managers that are completely incompetent trying to micromanage Software Engineers and Project Managers.'Why' isn't ever a thing they think about because they're afraid to admit as managers what they don't know why, and they aren't interested what would make software and integrations more effective. Within projects there was a great deal of expertise that was squandered by poor management and management structure within the Air Force.
I worked there for 5 and a half years, then quit because I couldn't deal with the frustration of upper level incompetence and blatant fraud waste and abuse.
How's that? Yes, now in order to use any OSS you have to have support and maintenance contracts from OSS vendors like Red Hat, Cloudera, VMware, there are a bunch of them. Their formula is to offer a productized version of the community editions that are closely managed and generally running a few iterations behind the latest dev versions supposedly for security and stability (though in my experience neither is often the case). At a Marine Corps shop I was working with last year it was actually more expensive to bring in OSS and support for this very reason, given they already paid M$oft licenses. Are they cheaper generally yes, far from free though.
Last thing I will offer up is the security profile of most OSS is atrocious. We routinely pull dozens of critical CVE's from systems built using OSS for the simple fact that fixes are difficult to obtain for libraries that other libraries depend on. Massive problem as evidenced by current state of our (lack of) cyber defenses. On the whole I support OSS and am a primary developer of it but you can't just drink the kool-aid.
I admire his advocacy for use of open source software, but his primary pitch is a reduction of costs. Any organization that switches to open source for the purpose of promised dramatically reduced operating costs is fooling themselves. The true benefit of open source is much more than savings.
I understood it the other way around: the bureaucracy was willing to pay the bill for the closed source options, thinking a large maintenance fee meant fast support. The author, on the other hand, wanted a system that would be more agile, with faster updates, easier maintenance and standards:
"it was horribly inefficient, a maintenance nightmare, not user friendly, and agility was measured in decades.
Our job was to take that mess and fix it. The idea was to build a standard hardware and software platform [...]."
So, as I understand it, easier work for him and his team and indirectly lowering the barrier of entry for future vendors, allowing more competition or a way to fix things themselves. Pretty much the true benefits of open source if you ask me.