How Go helped save HealthCare.gov | Hacker News Reader

How Go helped save HealthCare.gov

changelog.com

371 pointsra7148 comments

Greetings, folks: as we know, headlines don't tell the full story, and no, spoiler alert, Go alone did not save HealthCare.gov. A lot of people pulling together and working very hard over many months did. There are very many stories about the rescue. This is but one part about how indeed, a modest but important service written in Go, yes, did help, a lot.

It is worth noting that, 7 years later, almost all of HealthCare.gov has been rebuilt, and substantial parts of it are in Go. This is in part what the company I co-founded after the rescue has been up to. (If this kind of thing -- building modern, reliable digital infrastructure for government -- seems appealing to you, [we are hiring][1].) This is a pretty interesting story in its own right.

[1]: https://adhoc.team/join/

Neat. Curious: What's your stance on open-sourcing all civic infrastructure, like Healthcare.gov?

I think it's a good idea (not a magic silver bullet, but considering tradeoffs, on balance better than proprietary platforms), but it's entirely up to the government folks in charge of particular projects.

I'm curious what you think the greatest downsides would be? It seems the typical ones (you can't sell open source stuff) don't really apply.

VA has attempted a lot of open-sourcing as well, including most of VA.gov [1], including the project management of it, mostly as an ideological stance from the VA product owners we work with.

Open-sourcing hasn't cultivated much in the way of public engagement with the projects, but it's done a lot in terms of making development easier for the range of contractors + VA employees we have, and (IMO) nudged toward better decision-making with the underlying knowledge that everything we do is publicly viewable.

Other federal agencies routinely come to VA asking to learn from their digital modernization efforts, and I suspect the open-source stance has been a big part of that.

(I also work at Ad Hoc. It's great!)

[1] https://department-of-veterans-affairs.github.io/va.gov-team..., repository links at the bottom

Nicely stated gunsch:

> Open-sourcing hasn't cultivated much in the way of public engagement with the projects, but it's done a lot in terms of making development easier for the range of contractors + VA employees we have, and (IMO) nudged toward better decision-making with the underlying knowledge that everything we do is publicly viewable.

Care to define "gunsch"? Other than the expected gross definitions from Urban Dictionary I'm coming up empty-handed.

That is the handle of the person who wrote the comment.

But now it's a word, meaning: to mistake someone's username for an unfamiliar natural language word.

To gunsch v.

The name of the commenter they replied to.

It's worth noting that the VA's most critical software -- its EHR system, VistA -- is public domain with the source code available.

However, despite being a critical, successful piece of open source software that had massive investment over decades, it's being abandoned for a commercial system, Cerner, in a $16bn project.

https://ehrintelligence.com/news/va-cerner-implementation-co...

Why is that happening? How successful was the open source software?

As far as software success goes, all healthcare providers at the VA have used VistA daily for a couple of decades (actually CPRS -- it's like an MFC app or something on top of VistA). A relative who works at the VA likes it, and is not excited about moving to Cerner.

Why Cerner? I don't know.

https://projects.propublica.org/represent/lobbying/300928017

VistA has been a very successful open source project and is being continued in OpenVistA [1]. Originating as a software project in the Veterans Administration (named VA MUMPS) in 1977 it only took on the name VistA in 1994[2]. Internationally it has been used freely in hospitals in the U.S., Mexico, India, various countries in Africa, ...[3].

It was a truly open source project within the VA with programmers customizing this national patient record software in cooperation with doctors (to meet their needs) locally and sharing the modifications nationally. Perhaps its greatest technical challenge (besides complexity arising from decades of evolution) was finding programmers to work with the MUMPS language that it was written in. FYI MUMPS is a language with an integral db -a concept which was out of vogue for some time.

Political challenges are another story[4].

[1] https://sourceforge.net/projects/worldvista/ [2] https://www.hardhats.org/history/hardhats.html [3] https://worldvista.org/AboutVistA [4] https://www.politico.com/agenda/story/2017/03/vista-computer...

That thing is actually mostly designed by HP and RedHat, had really weird design choices and wasn't cheap either.

In my opinion none of them were capable of designing a user facing application. It was also built as a client/server desktop application. Not really a good choice in my opinion. Sure there were no offline PWA at the time, but by contrast if your SAP backend dies all the local applications in a hospital e.g. writing the release report also dies.

Needless to say given it's immense complexity it was also impossible to use this monstrosity elsewhere.

But replacing it with Cerner in a $16bn project is just sad.

There is a start on VA.gov already happening in this direction. The front-end codebase and backend (Drupal 8) CMS are both fully opensource for VA.gov.

* Frontend - https://github.com/department-of-veterans-affairs/vets-websi...

* Backend - https://github.com/department-of-veterans-affairs/va.gov-cms...

Paul's company is working on this project too (whom I work alongside with but with a different company)!

The U.S. Digital Service is really a big factor in this, see the playbook, specifically play 13:

> If the codebase has not been released under an open source license, explain why.

source: https://playbook.cio.gov/#play13.

So, see how this is now flipped. It is now "hey, you need to open source this by default, and if not, you need to explain why you didn't." and not the other way around!

Update: See Andrew Gunsch's comment in this thread too!

It would only make sense to open source something like that if people would actually go work on it. There may be lots of security flaws in the system but it's not guaranteed that open sourcing it will result in a fast fix from the community. This isn't OpenSSL.

A better approach might be to just budget for and pay a team of good developers a market wage to make it better.

To be fair OpenSSL was mostly ignored by industry and the community until heartbleed.

“At the time of the Heartbleed attack, the OpenSSL website listed just 15 active developers, most of whom contributed to the project on a volunteer basis. But not all changes to the OpenSSL software are written by these 15 people. Rather, these developers help to filter and organize suggested changes from a larger community of people who make occasional contributions.”

https://www.vox.com/2014/6/19/18076318/heartbleed

If you want the government to have more competitive Federal salaries for developers then write your Senators.

As a note, I’ve seen some truly awful software written by highly paid devs, so, just paying more won’t necessarily fix the problem.

> As a note, I’ve seen some truly awful software written by highly paid devs, so, just paying more won’t necessarily fix the problem.

High pay is necessary, but not sufficient.

With high pay, you MAY get a good product — if you attract competent people on all levels.

With low pay, the mind-numbing godawfulness of the end result, if it's ever delivered, is practically guaranteed.

Sometimes, the high pay is to offset the terrible nature of the work, not to attract better developers, e.g. having to work on the terrible ad-hoc, micro-managed code of many previous devs.

https://www.gov.uk/ is substantially if not entirely open-source, and this appears to have had benefits above and beyond "community contributions".

[ EDIT ] And, contra to your apparent contrasting of "open source" and "paid developers", gov.uk is both.

Re: your edit, I am also baffled by this idea that open source and paid developers cannot exist.

In this case it's the government which technically doesn't have to actually make money (budgets, tho), but it can work in non-governmental situations too. Maybe via consulting/support on the technology, grants/donations (OpenSSL, now), maybe just making it open to foster contributions and long-term maintenance[0]. Maybe the actual software isn't the company's main focus and open-sourcing the code is just a way of ensuring accountability and quality. IME, people are actually far more conscientious when they know they code/docs could be read by anyone for all time... even if it probably won't.

[0] I'll grant that this is probably quite rare. That's not the point... the point is that there are many potential reasons for open-sourcing.

While it is true that there are more developers paid to work on proprietary software than on open source, there are lots of open source paid jobs out there. Some are listed on the FOSSjobs site and resources wiki. There are also a number of them in the monthly "Who is hiring?" threads here on HN.

https://www.fossjobs.net/ https://github.com/fossjobs/fossjobs/wiki/Resources

The logic is different: if public money paid for the code, the code shoupd be public.

So paid developers work on, but the end result is publically avaliable.

I see, that's fair.

That's exactly my thinking.

> building modern, reliable digital infrastructure for government

And how about well supported? How resilient is the technology going to be in 15 years and how easy will it be to find developers? How much are those resources going to cost versus say .NET, PHP, Java, etc.?

Go is sufficiently trivial that anyone who dares call themselves a programmer can learn it in two weeks if they already have experience in some even remotely-related language.

You could have said the same about pre-generics Java. The problem is that when the language is simple and minimalist, people do things with the language that are byzantine and arcane, and you can't learn those in two weeks. Example: classic Spring's XML configuration inner platform.

No you couldn’t. I watched both Java and Go since they got released and Java really quickly went into the over-engineering and complexity camp long before generics. I worked as a consultant doing Java long before generics and the culture of over-engineering was already there. The language also really encourage it.

Go is an entirely different beast. The language, libraries and whole community is really worshipping simplicity in a way Java never did.

Just look around a bit at the standard library. There are pretty much no setters and getters. No inheritance hierarchy to speak of. Not even constructors most of the time.

It is not without reason that Go angers a lot of people that think they know how to do software. It breaks all the rules they have kept sacred, that they think all “real” programmers should follow.

Think about it this way. If Java is SLS assembled in a clean room never getting done, then Go is more like SpaceX assembling a rocket in open space with guy who normally weld water towers. It isn’t how it is supposed to be done but it works and they get shit done 20x faster.

News flash: this is what they said about COBOL, C, C++, JavaScript, closure, etc. as well as architectural patterns such as OOD or functional programming. Every language, and every platform has it’s nuances. You can do silly, non performing things in any language or platform. To assume a particular language will solve all your woes is outright naive and exhibits no memory of past software engineering concepts.

You are basically saying that no language is a silver bullet and any of them can go badly wrong, which is trivially obvious and not a productive addition to the discussion. If you have specific criticisms of this approach, that would be much more constructive.

When you claim history is ‘not productive’, you have taken the fool’s path.

> The language, libraries and whole community is really worshipping simplicity in a way Java never did.

Isn't that because it's essentially "new"?

I don't think so. Go looks to be focusing mostly on systems-level problems and the kinds of applications that orbit that concern.

Java, on the other hand--at least when you control for the confusion around whether "Java" referred to the JVM, the language, the base class library, or the security model--was looking to take on C++ and the kinds of software people were looking to write in 1996 in C++.

You can write line of business software in Go, but that doesn't really feel like its purpose in life. The language fights you a little bit when you try to do that.

It's been released (or gotten a 1.0 release) in 2012 but in development since 2007, so relatively as old as Java 5 since Java's inception; compare the features added to Java with those to Go: https://en.wikipedia.org/wiki/Java_version_history vs https://en.wikipedia.org/wiki/Go_(programming_language)

Nope. Java complexity came straight from Sun and friends. J2EE / EJBs (came in 2000) did not come from some randos, it was official Sun specification and supposedly standard way to develop any enterprise application.

I think Go has held up pretty well on this front, and I've also found a good heuristic to detect if libraries aren't doing this well. Check and see if they use the empty interface type a lot (this is the All or Any type in go). It's a code smell for me if I see a library using it a lot.

This project might not have a million lines of code but how maintainable is the codebase going to be in a few years once the original authors have moved on to other projects? Is the code organized to be easily readable? Java is very expressive but that's a good thing from a maintainance standpoint.

Also, the ecosystem is very young so you don't know which libraries will still be around in 3 or 5 years. You might find yourself reworking large parts of the code for betting on the wrong horse.

> Also, the ecosystem is very young so you don't know which libraries will still be around in 3 or 5 years.

This is precisely my concern. It's all good and well that we have these "Google level engineers" taking over government projects. But when the boredom runs out (or they go back to the private sector), who takes over?

I suspect the first people who wrote gov't applications in COBOL felt the same way...and you know how that story ends.

I think the obvious answer is to make it compelling for those kinds of engineers to consider civil service as a long-term option. Right now, 18F positions on the general schedule, which is probably where ICs would land, are time-limited up to, I think, four years.

That said, we should perhaps not have Google engineers live the meme of bringing their flavor of complexity everywhere they go, but I feel that having permanent FTE positions within government agencies specifically for bolstering up the competency and cost-effectiveness of information technology projects is overall a good thing.

My suspicion is that the majority of the projects that agencies like the GSA undertake (through its TTS arm, of which 18F is a constituent part) are ho-hum boring business IT projects, where most of the complexity comes not from the technology itself but from having to deal with getting a handle on the meandering specifications laid down by stakeholders (read: the plebiscite via representation in Congress).

I further suspect that the engineers that are recruited by 18F and the USDS tend not to go into the weeds on immediately building out, e.g., systems like Chubby and Stubby and D/GFS and Bigtable for the sake of it but rather tend to focus on trying to solve the problem at hand. I feel that the requirement to submit timesheets (and, hence, justify time spent) every week tempers that "every problem is tempting" feeling. Anecdotally, it did for me when I worked at a similar three-letter organization, albeit in the private sector, for the better part of half a decade.

I've been coding for decades and Go is the best language I've used in terms of being able to easily read and understand other peoples code. It also "feels" very maintainable to me, but it's hard to speak empirically about that.

I've also found that most libraries in go hold up well over time. In fact, I just recently needed to write a go program that could use some code i wrote many years ago and haven't touched in nearly as many years which used a few external libraries. I literally copy and pasted my old code into my new project and it just worked.

One part of the ecosystem and community is that they tend to pull up their nose at using too many libraries, especially if they're too different from the standard library. A lot of people come into e.g. the Go slack channel and ask "What is the best web application framework?", because that's the go-to thing when you do e.g. Java; the common answer is "YAGNI", because the standard library is good enough. If not, there are some really lightweight libraries on top of the stdlib that do things like adding route matching, or slight conveniences to do SQL.

I did just this, learning Go to do MITs Distributed Systems course. It has the speed and easy of dynamic scripting language plus a ton of sane defaults where JS and Python have quirks, and is compiled. My understanding is the major downside is the performance has stalled at 2-5x C++ which is not enough for systems infrastructure projects.

What do you mean 2-5x C++ performance?

I think he means C++ is 2x to 5x faster to run than Go. I saw many benchmarks and not sure if that is true though. Go is still one of the fastest out there.

According to micro benchmarks C is about twice as fast as Go.

https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

That is already pretty fast, and I guess the difference in practical scenarios would be even smaller since external systems like DB's won't change their speed if you call them with another programming language and because production code is not always as optimized as benchmark code.

Sorry, I mean in benchmarks Go took 2-5 times what C++ did to complete.

As others have noted, this is one of the areas Go really shines at (and I am not a Go fan). The lack of language features really forces most code into a limited form of TOOWTDI and because it has pretty opinionated choices about loops, conditionals, errors etc... you really don't see much Go code that makes you scratch your head.

Plus - at least so far - it hasn't aged; they are reluctant to make backwards-incompatible changes, and right now they're really humming and harring about whether or not to add generics. The biggest changes you'll see in Go codebases from 10 years ago vs modern will be Go's module system / dependency management and the `context` package sprinkled as an argument to some of the API methods.

Compare that to other languages (from the top of my head, don't pin me down on details or inaccuracies please):

- Java: Added generics, annotations, lambda expressions and streams, new date/time libraries, modules, switch expressions and multiline strings.

- PHP: Added classes, types, null coalescing operator, etc

- Javascript: Classes, ES6 features (arrow functions), imports, and offshoots like Flow, Typescript.

- Obj-C / Swift; ARC, Swift itself (4 versions), I haven't been up to date with this much.

TL;DR, most languages really shift over time, often 'borrowing' features from other languages, but Go resists these changes because it's intended to still be read and compile in ten, twenty years' time.

Personally, I'm working on a management interface for a network application; the existing UI has been around since 2012, I'm trying to build this to last at least as long, probably longer, and I think Go is a great choice for it. The old application was built in PHP 5.2, which is vastly outdated now (and updating is challenging because on the one hand, the versions of RHEL used in production don't supply newer packages, and on the other there's 65K lines of badly written PHP that would likely need to be updated).

Conversely, the new Go back-end is standalone and self-contained, I try and maintain the habit of updating the Go version and all dependencies monthly; so far I've never had to make any code changes or had to rethink my architecture because of Go adding a new (architectural) feature. (I probably have the benefit that it started after `context` and the module system were in place though). I feel a lot more confident in its future proof-ness. And I feel confident that anyone looking at the code can maintain it, although that said I'm cautious about the more magic parts like GORM, which I should probably swap out with lower-level SQL at some point but I don't really want to.

why wouldn't it be comparable to any other modern, reliable digital infrastructure? Those are not terms I would use to describe something that was going to become a horror show to deal with in a few years.

This is something we think about and discuss a lot. We take the implications of using the word "infrastructure" seriously. There is a healthy scepticism among the team about new & shiny. We try to be conservative in our technology choices, recognizing that the externalities of those decisions will be borne by others, including the taxpayer. That said, we also have to be mindful that users' expectations about technology change over time, based on their experiences with the consumer world, so we also have a pressure to at least be conversant with new stuff and evaluate it regularly.

I think this is a rich space for practitioners to comment on. We are in the midst of a mass conversion of the delivery of government services into digital channels. It's important that we approach it as if it were physical infrastructure, for a host of reasons.

My understanding was healthcare.gov was a failure until a small team took over and redid the full thing in ruby on rails just to get it all working, and then later it was ported to something more performant. Is that true?

There are many gross oversimplifications of the healthcare.gov rescue experience.

It was a long ago, but here's hoping someone from 18F can chime in: https://18f.gsa.gov/

No. The original site was a Java/J2EE application, and that's what the small team you refer to helped to stabilize on the rescue. We got it working well enough to make it through that first enrollment year. There was no rewrite into Ruby on Rails.

Later, and over time, key components of HealthCare.gov were redesigned and rebuilt. Rails was used for some, Go for others, and there is still some Java from the original design.

I was checking out the application for one of your positions and found this required question:

> Please provide your salary requirements or range.

I feel frustrated when I see that you haven't listed a salary range for your open positions, yet expect applicants to provide this information. I think it puts prospective employees at a disadvantage. Also it can waste applicants' time if you'll never pay as much as they need or expect.

I think your mission is valuable. I want public services to serve the public well. I hope you consider being more open about potential salary ranges because people like me would feel more comfortable applying.

This may be illegal in California where employers must share their range and cannot ask candidates their salary history.

No, I don't see anything here that would be illegal in California. There's no requirement to include the salary range in a job posting or anything upfront, the obligation is to disclose the pay scale for a position "upon reasonable request." In fact, this is specifically called out as being after an initial interview.[1]

Employers can't ask candidates their salary history, but they can definitely ask the candidate for salary expectations.

[1] CA Labor Code Section 432.3 (c) An employer, upon reasonable request, shall provide the pay scale for a position to an applicant applying for employment. For purposes of this section, “pay scale” means a salary or hourly wage range. For purposes of this section “reasonable request” means a request made after an applicant has completed an initial interview with the employer.

Like many CA laws I don’t see how this is practical. Unless there is some reference to market ranges or something might as well say range starts at minimum and up to whatever made up budgeted amount, depending on experience.

And one is free to skip those companies. If those are the practices to entice you before you start, imagine how they would be when you’re working there.

You could say that for any labour regulation though. Like, how about skipping companies with non-competes? Or how about skipping companies paying below the min wage?

The reality is many people aren’t at the liberty of strictly choosing employers; and labor laws help raise pay for everyone at the detriment of corporations and investors b

Yes, assuming it will depend on the quality of the candidate, much better to publish a range and say "based on experience" Especially because salary requirements may be soft, and also dependent on a benefits package. I'd take a lower salary if there was, for example, an excellent 401k matching program, flextime, etc.

Agreed. I would not consider applying for a role like this where there is no clear commitment to salary or day rate, especially where there is a lengthy 3-step application process involved.

They (adhoc) pay scale for everyone ... as they are focused on pay parity. Everyone starts at the same equal level pay per job title and you can not negotiate your worth. Also, you have to take a lot of tests and go through a gauntlet of interviews. Personally I was exhausted after the first one in which they detailed their long drawn out process and dropped out. So, I'm gonna go through all that and I cant negotiate my worth/years of experience?

I will say its a good place/company for starting your dev/design/tech career.

Starting out your career with an inflated salary is better than anything they can offer.

What do they offer?

entitled much?

Agree. In the UK most salaries are stated upfront, especially contract work.

Personally I do not even bother looking at a job unless there is a salary/day rate stated.

Or you could just apply and say "1 Billion dollars".

...in your best Dr. Evil voice.

In the UK it's only contract work that states a pay upfront. Usually a daily rate for tech work.

Permanent roles don't state salary. Exception being a few companies like government that have a fixed band (way below market).

FYI, looking at your job descriptions as a non-citizen, this line is pretty cryptic: "Some federal contracts require U.S. citizenship to be eligible for employment." Is this a really vague way of saying that only US citizens should apply?

Not likely. It's actually quite literal: Some federal contracts do require U.S. citizenship to be eligible for employment.

Government contractors often have postings for multiple contracts or positions and not all may have this requirement.

Ad Hoc employee here -- this is spot on.

This abysmal response suggests that there are no non-US citizens currently employed there and nobody knows how it would work if one was hired. Good to know! I’ll be interested to hear when someone finally guinea pigs it.

Yea, I do know that. What I don't know is whether that makes up 95% of their work and the other 5% is maintaining a COBOL app? Yea no thanks.

edit: "quite literal" is absolutely worthless. It would be "quite literal" to post a job saying "You will definitely be paid", and it would convey no information at all.

I think the point of the question is not what are the literal facts, but rather what do they mean in practical terms for applicants.

I think it would be very important to spread the word about how healthcare.gov was saved. Are there any good sources on this?

It's not just US that fails with these kind of projects. In Sweden we have seen many expensive government IT infrastructure projects that effectively flush tax money down the drain. The users find them horrible to work with and they are riddled with security issues.

As a developer in Sweden I feel very frustrated about this. If I knew of a similar company in Sweden I would seriously consider joining it.

It could be done so much better. But how did you do it?

Edit: And I'm thinking not so much about the technology part of the problem but everything else, since finding competent developers that can solve the problem once you have a clear direction and decent financing seems like the easier part?

Edit 2: It feels like one of those things that could make a really eye opening documentary given the right direction.

Having seen such failures in Germany, I think the key reason is that government IT contracting requires two very different skills: Government contracting, and IT. And "contracting" skills are much more important than the latter: It doesn't matter how good your IT skills are, if you aren't good at contracting, you aren't getting the job, but if you're good at contracting, you can get the job, deliver a shitty product, and still make money.

Good IT firms are busy working on free-market projects where they have to deal with 10% of the bureaucracy. It's not worth competing against established players who know how to play the government contractor game much better than you, especially if bidding rules may force the government to pick a contractor that they know will deliver a subpar product because it's cheaper/because their bid is written by someone who knows exactly how to write them.

I suspect that if you want to compete in the market, you need to have a "bureaucracy" team that's at least as big, if not significantly bigger than, your actual software development team. Even in an ideal market that's going to drive prices up.

Edit: It seems like in this case, this quagmire was avoided by having the government build a skilled team in-house, avoiding all the overhead of an army of government bureaucrats interfacing with an army of contractor bureaucrats (which is necessary due to the processes that ensure fair contract awards).

> this quagmire was avoided by having the government build a skilled team in-house

Oh, this has been my feeling of how to solve the issue as well. Government in-house projects seems to have better quality overall in Sweden too.

Paul, I love the Ad Hoc Homework resource. Thanks to your team for sharing it!

Holy crap dude. It looks like a sea of white faces. (it's partly the glare, but still) Are you facing any challenges in finding more diverse candidates?

Specifically: the author used Go to create a routine that would collect user data from submitted JSON requests to the site and then email them at a point later in the day when server load was low so they could sign up for healthcare. (Pre overall operational fixes / improvements)

Transcript Anchor: https://changelog.com/gotime/154#transcript-83

I think people on this thread saying "any language would have worked" miss the point.

The solution was to find a simple, obvious to deploy and maintain and monitor piece of code, doing precisely the thing needed to solve the very specific problem they had, and nothing more.

And this is precisely the kind of culture that has been built around the go language from day one. So, in a way, it's absolutely not surprising that go was chosen.

Finally somebody said it! You cannot divorce language from its engineering culture. Just compare how Go libraries and tools are made compared to Java. In the Java world everything is over-engineered to an extreme and if you as a new developer try to keep thing simple you will be quickly put in your place and told “that is not how proper Java programs are written.”

And Java is not alone in having that kind of engineering culture. You find it a lot of places but it is the community where I find it most noticeable.

Maybe Kotlin will change this, given time.

Unfortunately Java seems to bleed its culture everywhere:

Modern PHP development seems to be about how much of a pyramid of class you can build to serve a single request.

Then you have all the new typescript frameworks, proudly inspired by Spring.

(Former 18Fer here)

If you're interested in getting involved in government-focused work, there are a ton of fantastic organizations hiring right now.

One warning, before you consider applying to any of these orgs: please do not see yourself as a savior. There are a ton of smart people already in government who have tried everything before. More smart, talented people are great, but appreciate and respect those who have come before, who have been doing the Hard Work for a long, long time.

Be humble, be willing to learn, and be ready to listen.

Gov agencies to start: 18f.gsa.gov usds.gov

Gov contractors to start: http://adhocteam.us/ https://www.navapbc.com/ https://truss.works/

And selfishly, I'm Head of Product at http://www.demandstar.com where we're building software to help city and local governments bring the procurement process out of excel files and outlook inboxes and onto the web. Hiring Lead and Senior frontend engineers. Email me zcohn@demandstar.com for more info.

Are there any open-source projects you'd recommend helping out with?

[Disclaimer: the government is a Big Place and there is lots of variety.]

With the exception of open source infrastructural tools that are used by the whole world, there probably are not many open source government-run projects that allow contributions from non-governmental employees.

There are all sorts of reasons for this:

- Some are legal. Lawyers get anxious about the soliciting or accepting free work. There are ethical issues around contributions - how do you know it isn't a bribe?

- Many are technical. Someone has to review that code before it gets merged in. Who does that? Probably needs to be a government employee, but many government agencies don't have much, if any, in-house engineering resources - and what few resources they do have are already quite busy maintaining the current crop of software projects. How is this maintained once the open source contributors go on to other projects?

- Many are a mix of both. How do you ensure the code meets proper security and compliance standards? Accessibility requirements? Records retention requirements? Who is responsible for fixing one of these issues when an issue is discovered? What happens if there is a support issue? Who can the case management worker who is out in the field contact to get help?

This isn't to say these issues can't be solved. But it would take a lot of work to solve them all, and it would be difficult to solve them all in a way that would be cheaper than just hiring internal resources, buying a COTS solution, or hiring a vendor to build what you need.

That being said - if you aren't ready to switch jobs, there are lots of government-adjacent open source projects - especially in the Open Data world.

I'd recommend checking out your local Code for America brigade as a good starting point: https://brigade.codeforamerica.org/?_ga=2.169184225.19486385...

Thank you for the thoughtful response. I will look into Code for America

Is go what saved the project or good technical leadership and cultural changes in the thinking of design and architecture of software?

The title is “Go helped save” not “Go saved.”

Yes, but what little detail there is sounds like something which almost any language could have done acceptably. I'd believe that Go would require a few less servers than, say, async Python but the title really should have been something like “A little architecture goes a long way”. The approach they described is no doubt considerably more polished but people built websites using the same techniques back in the 90s to avoid SPOFs and then XHR made that even easier to do robustly.

To be clear: this isn't to say that their work wasn't good but really seconding the point they're making about how the original design was the wrong architecture for a public website and there's no easy way to band-aid around that.

This was my complaint about a go port at my employer. There was an old C++ service that was really slow, so the Go advocate made a bug-for-bug compatible version of it in Go, and they migrated over to that system. A lot of work went into touting how awesome it was, until I asked "so, you could have done this in C++, or really any other compiled language?" and they said "yeah, I just wanted to help build enthusiam for go. there's nothing specific about the language that made the resulting system any better"

There's always a tension between maintaining an existing codebase and trying something new but in that case I'd personally jump on a Go codebase instead of C/C++ in a heartbeat just for the maintainability improvements. There is a meaningful difference between “can a turing-complete language do this?” and “how much time will you spend dealing with avoidable problems or overhead?” and in the case of unsafe languages I find the arguments relatively compelling for picking anything safer to use in practice.

and obviously they can now get hired for Go and you can't.

what's the complaint, stop falling for these pedantic loops and win the game

The complaint is that as long as you're transparently wasting employer resources on a resume gambit you may as well just embezzle anyway and avoid inconveniencing your coworkers.

no no they're helping build enthusiasm!

Evangelization is a paid gig at times.

Neither that code author nor I has any trouble being employed. I learned go and then stopped using it because (in my field) C++ and Python are what is used most commonly.

https://medium.com/the-u-s-digital-service/mikey-dickerson-t... from the guy on the front lines. He has fascinating stories.

I just want to point out that I was unable to fill out my 2021 marketplace application on healthcare.gov until I stopped blocking optimizely.com in Little Snitch.

That’s terrible. Nobody should be compelled to provide their private health information to a single, unelected private company to obtain government benefits or comply with the law.

It's been like that since the initial release unfortunately.

Realistically private companies support all sorts of government operations and have access to all sorts of extremely sensitive information. There's just something extra creepy about seeing it happen in your browser. (It's not immediately obvious what can be gleaned but most likely more than what you would gather from a cursory glance.)

Here's a few of the domains from just the front page:

    adsrvr.org
    advertising.com
    chartbeat.com
    chartbeat.net
    digitalgov.gov
    doubleclick.net
    google-analytics.com
    google.com
    googleapis.com
    googletagmanager.com
    govdelivery.com
    gstatic.com
    healthcare.gov
    newrelic.com
    nr-data.net
    optimizely.com
    qualtrics.com
    quantummetric.com
    tealiumiq.com
    tiqcdn.com

Edit: After a bit more looking I really think they need a bug bounty or something. This site has been dinged numerous times for infosec practices and it doesn't seem like the message is getting through.

I have quite a few of those domains blocked, but other than optimizely, blocking seems to have no effect on functionality.

This is what you want to see in your Optimizely code from Healthcare.gov:

    <!-- Facebook Pixel Code -->
    <script>
    !function(f,b,e,v,n,t,s)
    {if(f.fbq)return;n=f.fbq=function(){n.callMethod?
    n.callMethod.apply(n,arguments):n.queue.push(arguments)};
    if(!f._fbq)f._fbq=n;n.push=n;n.loaded=!0;n.version='2.0';
    n.queue=[];t=b.createElement(e);t.async=!0;
    t.src=v;s=b.getElementsByTagName(e)[0];
    s.parentNode.insertBefore(t,s)}(window, document,'script',
    'https://connect.facebook.net/en_US/fbevents.js');
    fbq('init', '1706186366368131');
    fbq('track', 'PageView');
    fbq('track', 'Lead');
    </script>

lol

it's infuriating that google is so embedded in government. i'm routinely thwarted from using government websites, especially in CA, because i block google (especially captcha).

if you're in gov (or not), please advocate for using/building open-source, privacy-conscious analytics tools rather than giving all of our sensitive information to uncaring corporations like google. the right to privacy extends to our personal data and google shouldn't be a information tollbooth between us and government services.

At a glance, it looks like optimizely does A/B testing. Couldn't they do that without receiving personal data?

Because optimizely is a marketing department's wet dream for invalidating a dev department. "See, we can add content by ourselves! we don't need any highly paid developers!". Meanwhile, it jacks in jQuery to add DOM nodes on a page that might not appreciate that method of content injection.

Hmmm....

label0: Pick any project x, and a technology y. If x is a success, publish content on how y saved x. else goto label0.

This also works for compound x:

label1: Pick any ailment x, and a compound y. If x improves with y, publish content on how y cures x. else goto label1.

Yes, it's the word "save" that's the problem. A title like "Go's role in improving Healthcare.gov" would be much better, less click-baity.

Because I can say SQL saved my data analysis project. MS Word saved my writeup of the project. Email saved my distribution of my final report on the project. The phone saved my ability to have meaningful follow-up conversations with stakeholders on the project.

the saddest part about this type of stuff is the gov has software devs on staff not from consultants who are capable of "saving" gov sites. It is often a political reason why things are they way they are(making everything 10x more convoluted for on staff devs)...and people profit more from having external contractors do the work. Parts of govt I'm in..contractors can use cloud services but our teams cannot, and we have to rely on internal systems and bureaucracy.. it's honestly all bullshit.

Yep, one of the core value propositions for bringing in consultants is “they tell the exec what your staff already know. But because it cost a lot of money to have them say it people are more likely to listen”

It’s a real service that provides real value. But companies can avoid the cost if they have a strong culture of trust internally (which is no simple thing to foster)

yep trust! exactly so little of it for whatever reason.. but somehow the contractor company we have to hound for the features they originally promised for that money get away with it, at least for enough years to get millions upon millions

Funny title. Meanwhile, title in an alternative universe, where the project didn't go as planned: "How Go has caused Obamacare to fail".

Oh God this has nothing to do with Go.

I interviewed with Ad Hoc a few years ago. They told the healthcare.gov story (and I googled it as well) but never mentioned Go. Seemed like a good group though

I liked this, Paul Smith had some good stories. This episode and the following one about Github CLI got me thinking a lot how we choose architecture and languages and how it’s an iterative process. Rarely do companies/orgs get it right on the first try. Needs change, estimates are way off, and budgets run out.

I did not listen to podcast as I prefer reading but looking at the title I'd say that if one is in dire straits one needs smart programmers. The language is irrelevant and unless some very specific requirements are to be met good software can be written in any language.

FYI the transcript at the link is really good

Oops, sorry did not notice it. Thank you.

Great writeup. AdHoc sounds like a really interesting company to work for. Does anyone know if there are similar firms working to improve online govt services in Australia?

> And like a SAND attached to it for network storage

SAND -> SAN

Hopefully that's a typo by the transcriber. ;)

Maybe if the team who built healthcare.gov wasn't shit all of this could have been avoided?

Those who cannot work for real tech companies... build poorly performing webpages for Uncle Sam?

Did they rewrote COBOL parts in Go too?

I don't know if I'd be boasting about wasting $2.1 billion dollars :/

How is that even a lot of money?

Do people not understand how many failed projects there are in the US budget?

I guess wasting other peoples money is ok as long as we waste less than the other failed projects?

Were you trying to put forth an argument? Because I don't think you did.

I think the biggest thing that helped save HealthCare.gov is having a blank check from the government.

As of 2014, the site cost $2.1B to create and operate:

https://www.bloomberg.com/news/articles/2014-09-24/obamacare...

I deeply respect the efforts that went into saving HC.gov though the initial rollout was a stunning blunder by the Obama admin for not keeping careful track of progress.

I further would note that such a citizen facing service would not need to have been stood up if Medicare for All was passed. It could all be handled on the backend and people would just walk into the doctor's office or hospital.

How fitting:

American healthcare pre ACA : ACA : Public heathcare

Untyped scripting languages : Go : Decent programming languages

Or maybe I am being too charitable to Go here.

What do you dislike about Go?

Not the person you're replying to, but the lack of generics is the big killer for me. I'm used to being able to manipulate data efficiently via streams/sequences and a rock-solid stdlib which just isn't possible in Go due to a lack of generics.

A close second would be the lack of good performant libraries, which is somewhat due to number 1 and somewhat due to it being a relatively newer language. Try finding a good drop-in loading cache akin to Guava caching or Caffeine in Golang, it doesn't exist.

My take on Golang: Use it for a project, get into the Go mindset of doing more with less.

Then back switch to a language that's actually suited for useful work with getting mired in dogmatism and bring the mindset with you.

Kotlin is a good one if you like Goroutines, Flow/Channels map pretty cleanly to them.

Proponents of Go will say the dogmatism is a strength... it is until it isn't

While generics is definitely an issue, all to often people simply have not learned to use Go properly. They are stuck in usage patterns from their former preferred language.

I remember as a Objective-C developer all these Java/C++ guys complaining. So often it was because they were trying to use Objective-C as if it was Java or C++.

Now as a Julia user I see all these Python guys complaining about stuff which is really about Julia not working like Python.

A lot of people just don’t have time for the language they are using. They just want to bang out stuff as quickly as possible in the way they are used to.

Not saying that is your case but I have seen plenty of complaints regarding generics in Go which could easily have been solved in other ways if one actually understood the language. Go without generics is not the same as Java without generics.

On a meta note, I'm surprised to find myself strongly agreeing with a post under the nick "RhodesianHunter", and strongly disagreeing with this post under the name "socialdemocrat".

Yikes.

I like the dog breed, that's it.

While I agree that different langauges promote different solutions (otherwise we wouldn't bother making new programming languages!), I'm sorry but it's just false that a statically language without generics can be sufficiently expressive.

Generics are necessary to move information of different shapes through a function without loss of fidelity. Without generics, there is an artificial trade off between the variety of the inputs that can be consumed, and the richness of the output that is produced.

Reply on news.ycombinator.com