Programming Language Breakdown for the HealthCare.gov Website | Hacker News Reader

Programming Language Breakdown for the HealthCare.gov Website

randalolson.com

93 pointsyawz59 comments

At least this pile of code turned into something, unlike my local Cover Oregon which has been killed following a $250 million outlay and still no end in sight [0].

Funny that oft-mentioned Oracle used their work with Oregon to tout how their enterprise prowess "plays an essential role in establishing Oregon as a leader in healthcare reform", but the link to the source piece is now dead [1]. I did, however, save a copy of this PR fluff when it came out in 2012, just to see how it would all pan out... [2].

[0]: http://www.latimes.com/nation/nationnow/la-na-us-attorneys-o...

[1]: http://www.oracle.com/technetwork/articles/entarch/oeea-home...

[2]: https://www.dropbox.com/s/jnt15vqdc15aopj/state-of-oregon-18...

Almost as bad as Ontario's eHealth scandal:

> the government spent five years and $647 million on the forerunner of eHealth Ontario: the Smart Systems for Health Agency, which used 15 per cent of its $225-million annual budget on consultants despite employing 166 people with annual salaries exceeding $100,000, before the project was shut down and restarted as eHealth Ontario.

https://en.wikipedia.org/wiki/EHealth_Ontario

Nothing like the massive bureaucracy of government healthcare combined with government run software.

You don't have to be a government to spend half a billion on a system that never works.

I don't see anybody claiming that you have to be.

It is generally an environment and context that can lead to resources being expended in ways that would be very unlikely to happen in more commercial settings, however.

Businesses do tend to avoid such mishaps in the first place, or at least manage to stop them earlier.

Cover Oregon was a disaster. That work should have been done by an agency here in the state. I don't understand the logic of government sometimes.

I once went out on a Tinder date with one of the IT managers for Cover Oregon (in December/January). She was a music major at Denison and apparently Oracle was vacuuming up everyone with a pulse and a BS in DC and the East Coast to manage their contractors. I guess she was doing pretty well too since she could afford to go to Sun Valley once a month from Portland. What I wonder is why couldn't Oracle find managers with more experience?

Incompetence?

I've met a lot of Tier 1 contracting consultants. Smart people, excellent communicators, well presented and great at generating ideas, but with zero problem solving experience they were no better than a fresh graduate.

You need a few seasoned professionals who have seen it all to keep the children in order. Reminds me of my year 12 chemistry teacher who was out of the class for a minute and on returning had the class evacuated within seconds. He recognised the smell of mercury vapour from a burst thermometer the second he walked in.

Are you a Oregon elector? If so, what was your logic? Did you not read the procurement notice? Did you agree with it? If you didn't agree with it, did you lodge a complaint? Or were you too lazy? Were you too lazy to even read the proposal? To read and analyze the resulting award? To read others' analyses? And if you were too lazy, how can you now complain without being a hypocrite? If you're not an Oregon elector, have you done any of these things in your state? Because I don't think an executive would be able to complain about the functioning of a corporation if they don't bother to show up at board meetings or even read the meeting minutes.

But it has all those smiling people in the pictures! What could possibly go wrong...

I've recently been solicited by a couple of recruiters, on behalf of contractors who've got some of that work. One of them in particular amused me by trying to hard-sell a six-month contract job with a firm which organized a "career fair" an hour south of here on two days' notice. When I showed disinterest, he "sweetened the pot" with an offer of a $25 gift card. That was when I told him not to bother contacting me again. (It's not even so much that he tried to bribe me, as that he tried to bribe me so low. I mean, really -- $25? What kind of naïf does this jackass usually deal with?)

If that's the at all representative of recruitment efforts around healthcare.gov, then I'm not sure exactly what it says about the quality of the staff working on the project, but I can't imagine it saying anything good; the best of them are probably quite bright, painfully inexperienced, and not yet equipped to understand how awful this project is going to look good on their résumés -- and I'd tend to suspect that the best of them are a small fraction of the whole. Between that, and the inevitable bloat and quality issues of any federal technology project with this many fingers in the pie, I'm really not surprised the whole thing has turned out to be such a rolling catastrophe.

Based on the last notice they sent out - around 400 people are working on this mess. I got a feeling they are more concerned with just filling the positions then hiring (and paying) experienced developers.

Based on what I seen in hacker news few month ago, a guy trying to hire people on healthcare.gov described it somewhat like this: we are saving the project and are looking for people. We hire them for few months stints (3-6 months), they work super hard that time and then leave. He mentioned occasional sleeping under desk, it did not sounded like a joke.

If true, they have perpetual death march with constantly changing workforce and everyone being essentially newbies on the project. How many experienced people can you get with that sell?

"Saving the project" has become a massive red flag for me.

In my experience, when anything about "saving" is mentioned, it means a handful of people started the project, made a hash of it, then after a few short weeks passed it off to another team who were tasked with "saving it". This second team blames all their failures on the first, thinks that they are making the best of a bad situation, then after a few months passes it off again. The new people comfort themselves with the same "saving the project" narrative for a while more until it again comes time for somebody else to save it.

Each team after the first lasts only as long as they can plausibly believe that everything is the fault of the people that came before them. As soon as they realize that their shit stinks too, they bounce.

In the end, 90%+ of the man hours in a project were put in by people who thought they were saving it.

So I have a theory:

http://www.ohloh.net/p/openjdk/analyses/latest/languages_sum...

The numbers even lineup fairly well in the 2010-2012 time frame which is around when the project would have started.

I still don't understand how they have that much code. 3.5M lines is still crazy - what exactly is all that goop doing?

Tying together fifty states worth of healthcare systems and laws to provide some kind of reasonable abstract interface. On top of systems that may or may not have been designed with this form of use in mind.

Go look up some of your own state's healthcare laws to see how bizarre they can get. Then think about how each state has organically arrived at their own set of bizarre and nebulous laws. Then imagine what the infrastructure built around those laws - adapted over time as they changed and ignored as much as possible - looks like.

People surprised at this number are judging a vast and complicated problem by the (relatively) simple and pretty face plastered on top of it. I'm actually kind of impressed that the number isn't higher.

Maybe they counted all dependencies or libraries. If you add all that to any project, size will grow ridiculously big. I suspect so because the coding part of the project too only half a year or so, so they could not possibly write that many line by hand.

> Maybe they counted all dependencies or libraries.

This is really the only way I can imagine they got to that number.

Spring, a beastly enterprise Java codebase if ever there was one, is still "only" 460K lines of Java code according to cloc.

Most if it is in Java, which is pretty verbose in comparison to many languages.

If Java were actually succinct, it wouldn't be Enterprise™

Does that count auto generated code? I agree that 3.5m loc seems absurd (I work on a complex medical device which has about 1m loc), but to be fair, they do have to talk to a ton of different and incompatible data sources.

AbstractSingletonProxyFactoryBean and friends I guess.

Wouldn't this type of verbosity just increase the length of lines, not the line count?

It depends whether lines count statements or lines.

    if (identifiers.length > enough)
    {
        anyStatement.canPrettyMuch(takeUp)
            .asManyActualLines(ofSource.asYouWant())
            .theOnlyLimit.isYour(imagination)
            .FactoryBean();
    }

import java.languages.english;

import java.languages.english.grammar;

import java.languages.parser;

import java.languages.conceptualunderstanding;

Java is also line verbose. The factory model also leads to not only long names for things but line verbosity in the form of multiple distinct instructions to accomplish something that in other languages might be a short one-liner.

They counted with word wrap turned on.

And column width set to 10.

Lack of code reuse. (other than copy and paste) So they probably do the same thing over and over in different parts of the application.

Reusing code is not trivial, the design has to be good, the code needs to be easy to find and call. It becomes harder as the number of people increases.

I imagine that 90% of that is dealing with the all of the various insurance providers and plans across the states, i.e., glue code/integration logic.

Destroying souls.

With all the healthcare.gov hate, I've been wondering why there's barely any scrutiny towards coveredca.com -- at least I haven't seen any.

I just tried to go through the coveredca.com application process and it was a nightmare. It took me 5 hours, spread over two days, 3 tech support calls (I never call tech support!), and 3 different web browsers to get through it.

1. Fill out the very first part of the application - just basic name and address - click "continue". Error page. WAT.

2. Notice they have an outline view of the application. Use that to get to get past the error page. Now it's asking for my "Certificate of Citizenship" number. WAT. No form I have ever filled out - tax returns, financial aid forms, employment forms, DMV forms - nothing has ever asked me for this. Spend a day searching for it.

3. Finally find the number, fill it in, click "continue". Error page. WAT.

4. Use the outline again to get to the income section. It lists me twice, as if there are two separate brikis98's in my household. It even asks me "how is brikis98 related to brikis98"? WAT.

5. Call to tech support #1. After 45 minutes of messing with the application, clearing cookies, logging out/in, we couldn't figure out how to remove the extra "me". There is no way to completely restart an application. Solution from tech support: sign up for a completely new account - new user name, password, etc - but don't enter your SSN too early or it'll detect the duplicate account. WAT.

6. Sign up for a new account. Start filling out the application from scratch. Error page. WAT.

7. Use the outline trick again. Go to income section. No duplicate "me" this time - yay! However, now it asks for "monthly income". I just left my job, so my monthly income is zero... but I obviously had a monthly income the first 5 months of this year. Call to tech support #2. They tell me to take all the money I've made this year, divide it by 12, and put that as a monthly income. WAT.

8. Finish filling out the application. Get to a page that has a button to "See Eligible Plans". Click the button. Error page. WAT. There is no outline view to get past this one.

9. Call to tech support #3. After half an hour more, the suggestion is to try another browser. I switch to Firefox. It tells me coveredca.com does not have a valid SSL cert. WAT.

10. Switch to Safari. No complaints about the cert. Log in, click the "See Eligible Plans" button. The page loads - yay! However, the UI is broken, with strange scrollbar-in-a-scrollbar setup that makes it very tough to browse the plans. Download PDFs of each one for easier reading.

After all that, I found that the plans were all way overpriced for the comically bad benefits they offered.

I ended up signing up for one, and I've been pretty happy with it, not too expensive, and decent when I've tested it. I also experienced a bunch of errors on the way to getting there.

The worst part, though, was after going through the 500 error gauntlet, hearing the person in charge of CoveredCA go on NPR and just pat himself all over the back for how great a job they had done, and much better the result was than Healthcare.gov. After spending only a few hundred million to create that hellish mess. One of the few times I've ever yelled at my radio.

CoveredCA is a true government offshore success story.

A $360M mess. Powering that beast?

1,714 cores, 9,394GB of RAM, 392,327GB of storage. Every single piece of software is Oracle with the exception of Adobe LiveCycle.

Only thing I've seen on CoveredCA is the folks that cannot find doctors after buying the insurance. I get the feeling the tech might be good enough that the social issues are taking center stage.

In what way is Maven a programming language? Is it simply to disambiguate other XML configuration from the build declarations?

How is 50k lines of build declarations even possible?

That's not actually surprising--it's really easy to have a 1500+ line long POM with how verbose it is. Depending on how many maven modules there are, I could see 50k pretty easily.

Brute force as opposed to a few moments of contemplation for a reasonably elegant solution

Url changed from http://www.randalolson.com/2014/05/22/programming-language-b..., which points to this.

I would argue that the graphical breakdown on the original link is more helpful to grok the breakdown of languages than the original reddit post.

Ok, we changed it back.

Yeah, why was the URL changed?

When a post's content is taken from another source, we switch the url to that source. I didn't realize that the graphic didn't come from there as well, so we changed the url back when a user pointed this out.

It's still borderline, though. Making an infographic is a pretty dubious claim to being an original source, and posts that consist of nothing but an image are usually pretty unsubstantive.

from http://www.enterprisetech.com/2014/04/04/hyperscale-lessons-...:

“The way that a lot of the code was developed for the application was using automated modeling tools” he says. “They configured their modelers to [generate middle-tier Java objects] in a three-tier style architecture. Essentially, they created a layer of abstraction that didn’t need to be there.”

I've got to work on my submission titles.

> Healthcare.gov developer provides estimate in response to "500M lines of code" 5 hours ago

The chart from the submitted blog post does provide a nice visual perspective of the breakdown that you don't get from the source alone.

http://www.randalolson.com/wp-content/uploads/healthcare-gov...

What could you possibly need that many individual yet mostly similar languages to accomplish? Even if we grouped things like XSD, XSLT into the XML category or Groovy and JSP into the Java category it feels silly to need this many languages in a single app.

I can only imagine what kind of things all the long tail files are doing. A smattering of Perl and Python, then a lonely VB and ASP.NET page?

I don't understand, are you saying that there are too many languages? I find it very reasonable: - css, js, html are unavoidable for a web app - sql is as standard as it gets for interacting with your database - shell is also widely used as a glue + mini tools/scripts - maven, xml, xsd, xslt are pretty much all xml, and good luck writing a modern java app without touching it - java is the main language in there.

If anything I'm surprised at the small number of languages.

It would be way bigger for the project I'm currently working on at work: we have python for scripting and integration tests, ruby for chef deployment, java as the main language, some shell tools, js/html/css/xml/yml, some legacy groovy, etc..

It's very common to have little scripts and tools hacked together that are rarely used (or even one-offs) making it to the main codebase.

As for the number of lines of code, I don't know how they count libraries and dependencies, but with maven it's very common to be pulling half the universe's libraries through transitive dependencies for parts of the libs that are not even used. It doesn't mean your code is wrong.

It feels like people are just trying to find reasons to criticize that project.

I've used cloc a fair bit... cloc is a bit sketchy in recognizing code occasionally. I wouldn't trust the "tail" of its output, it's frequently wrong or misleading down there.

Yep. In the same Reddit thread, someone ran cloc on the Linux Kernel, and it even reported one ASP.NET file in there too (just like it did with the Healthcare.gov source):

http://www.reddit.com/r/dataisbeautiful/comments/265yns/mill...

That probably explains the 5 lines of Python there.

This was also my first impression. Clearly this was not a well thought out project from any perspective.

I will double clutch on my rant about government inefficiencies for now, but man alive what a mess.

Sounds like lots of small groups doing their own thing without working together.

Seems like a result of alot engineers and an inhuman level of understanding the systems architecture. Reminds me of the Lsd usage for IBM. There must be an outrageous number of bugs and exploits in the code. Google Chrome is something like 0.06 per thousand lines of code.

At the recent dotScale conference in Paris, Robert Kennedy explained how he managed to fix the site.

See: https://twitter.com/search?q=%23dotScale%20Robert%20Kennedy

So the user-facing portion of healthcare.gov is written with JSF then?

Well there's 2.4 million lines of Java that will have to be maintained for the next 100 years.

wasn't this floating around 6 months ago? do we really need to discuss this again?

Reply on news.ycombinator.com