Hacker News Reader

Hacker News

Comment by john-tells-all | Hacker News Reader

Parent Full thread

john-tells-all

> particular technologies being used, including something called MarkLogic, an XML database

XML database ? For a critical service used by 300 M people?

Brrrrrr.....

View on HN

It was around high-tide of 'no-sql' but surely it cannot literally be true ?

I suspect business rules in XML, so effectively read-only, while dynamic data in a rdbms ? Maybe that's just my hopeful heart talking :-)

nope it was exactly that; transactional customer state in giant xml documents in an xml-document-storing database. Bugs would cause them to grow to 10, 20, 100MB.

Was working for MarkLogic and got pulled into this a couple days before launch. That sort of thing was the least of their worries. They got the model wrong. With ACA, you were not allowed to do policy pricing on preexisting conditions. Age, sex, smoking, and location. That really only leaves one variable - location - for the actuarial folks to create unique policies. You might suspect how this breaks things. A healthy 18yo female in Tampa could have hundreds of thousands of policies. Live on a superfund site... hey, about those 3 spendy options. Those files could be massive - hundreds of MB. The log files... ye gods... those things grew in GB fast.

The other gotcha was Java is single inheritance. Foo > XML. Bar extends Foo > XML. Baz extends Foo > /dev/null the Bar bits in the persistence layer.

The MarkLogic bits did OK. When everything went sideways at launch, almost no traffic was getting all the way to the back end. Because things were getting profiled, there was way more time to tune and tweak the indexes as they tried to re-engineer things. The NoSQL bits helped there, as it made it a bit easier to morph things as it was refactored.

Not to say there were not pain points or they did not have a silly amount of hardware in it in the end.

One of the more memorable things for me was working in the XOC, having an update go sideways in the early AM, and seeing CNN pop up a ticker saying ACA was down. Good times.

According to Paul Smith, who has an AMA about healthcare.gov at Lobster.rs[0], the database was MarkLogic, which started out as an XML database, according to its Wikipedia page[1] (Paul also mentions this briefly his 10th anniversary post). According to another commenter[2], who says he met a MarkLogic manager on a plane flight, MarkLogic had passed some sort audit/certification that only Oracle had previously passed, which may explain why it was used for healthcare.gov.

[0] https://lobste.rs/s/igt4ez/10_year_anniversary_healthcare_go...

[1] https://en.wikipedia.org/wiki/MarkLogic

[2] https://lobste.rs/s/igt4ez/10_year_anniversary_healthcare_go...

Wow, strange stuff, indeed.

I had the same thought

I doubt that was the main datastore. It might have been a configuration database though, which I can see it being an issue given that it sounds like it was business logic causing the issues.

It was. The way they built it out was UML > generated Java, which persisted data as XML. The MarkLogic database could then do xpath/queries (think stored procedures) for the Java layer - so not just POJO > XML, though there was some of that too. It ended up being a non-trivial cluster.

Your explanation made me dry-heave. UML > Java generation explains why their deployments seemed to contain obscene amounts of code that no one understood.

You would be surprised. :)

Source: I was also a member of the Tech Surge. (Paul Smith recruited me to help with testing.)

Reply on news.ycombinator.com