Comment by logfromblammo | Hacker News Reader

Comment by logfromblammo | Hacker News Reader

Parent Full thread

I worked at Epic for a bit more than 2 years.

It was about 3 years too long. Yes, that math is correct.

As far as I recall, MUMPS is essentially the dedicated coding language for a b-tree based database, and the only reason to ever use it for anything is for processing huge numbers of transactions through your NoSQL database every second, forever. I guess you also might want it if none of your employees knew how to correctly normalize a relational database.

Other than the nightmarish accumulation of technical debt continually rolled over since 1979, and the ungodly high turnover of people jumping ship after realizing MUMPS was a golden ball-and-chain, it was a pretty nice place to work. But management had a culture that it needed to force down everyone's throats, and the tech stack was a real resume killer. I still get contacted by recruiters desperate for MUMPS developers, and they make me feel like someone trapped inside a house besieged by zombies. I get really quiet, and hope they don't break any windows.

It only works because they throw a huge amount of money at it, and no one has any incentive to cheap out on medical management software.

EPIC management's (at every level) embrace of M and all things old as holy and glorious was what turned me off of it most. I could somewhat understand the need to keep working with cold in an old, bad language because it was still printing money (most anything related to healthcare does). I could not understand that consideration any possibility of modernizing the codebase was completely off the table--there were no R&D teams looking at new developments in database design as possible improvements, even though the tech most in fashion at the time (NoSQL) had striking similarities to M's own database structure. All effort was spent towards squeezing every last ounce of blood from ossified technology. This persisted well outside MUMPS itself: I was in the server management division, which was a champion of AIX and HP/UX (and in the weirder corners, VMS) over Linux. When I left there were a few people warming to the idea that RedHat might be enterprise-y enough to serve as a base for running M databases, but general sentiment was that most customers would continue to use old school nixes well into the future. Much of our time was spent painstakingly guiding rural hospital sysadmins through manual configuration of every sysctl and OS parameter needed to make Cache happy. Configuration management would have reduced this to about 3 minutes of tossing a Chef/Puppet/Salt/whatever config at a server, but was considered anathema because it was necessary to have that special human touch involved in spinning up a nix environment--surely no automated system could understand or be trusted with the intricacies of setting ulimits to values obtained via a function utilizing, at most, three inputs and grade school arithmetic.

Since I was more in their systems department I had limited opportunity to observe their archaic superstitions about code structure, but the short introduction course to M they provided confirmed that most current and new employees were willing to accept that certain control flow tools or function structures were dangerous because they were necessarily slower than something that made sense in hand assembly programming, since there had obviously been no developments in compiler optimization since M was designed (in the good old days, before C existed).

There were no archaic superstitions about code structure. Those had to be removed, because the code had long ago grown large enough to push up against size limitations. You couldn't put comments in the code. You had to use the abbreviated syntax. If you tried to make anything human-readable, it wouldn't fit.

You weren't even allowed to touch the infrastructural M code. Mostly, you worked on superficial or peripheral M code, or on the GUI in VB6. I heard rumors of a GUI modernization team, but never saw any direct evidence of it.

It was a result of the iron grip of the top management. They didn't want to yield control of the company to the people actually running it. You were expected to use your expertise to do what you were told to do, in the best way you could, while still remaining completely within your lane.

After the 6 months of training, I gave it a year to read the character of the company, and then another 6 months to soak some potentially major family health care costs with that sweet insurance plan, then spent 4 months job hunting. It was the best employer-offered health insurance plan I ever had.

But make no mistake. Companies like Epic are why health care costs in the US are huge and growing. Epic never refactors anything that still works well enough to hold together with some expensive human labor. It is a technology company that runs on well-trained people instead of well-designed code and processes.

The size limits were for object code, not source code. Comments and short command names have no impact on object code size. Ages ago, there was a 32KB object size limit. Now you can use up to about 8MB, which is large enough for pretty much all non-machine generated code. You can, of course, just use multiple routines to avoid the limit.

When did you work there?? I find it surprising you never saw anyone writing C# or TypeScript if it was any time recent. These days there's quite a bit of web-based UI replacing the VB6, though you might not notice if you don't look carefully enough.

If it was any more than a year ago, there were parts of the company that had barely started and web migration due to dependencies. Epic is still slow playing this to make sure they don't have some of the issues they had with the early 2015 releases

Times are changing.

Some functional areas use Hadoop and SQL now. New server development can be done in VisualStudio using Typescript (which gets converted to M).

There are automated tools to set and verify OS parameters.

Most new installs are Linux now. No one uses VMS today. HP-UX and Solaris will be out of use in a couple years. InterSystems licensing is why people continue to use AIX (you have to buy new licenses to switch from AIX to Linux).

LOL, speaking of b-tree based database discussion

See the intersystems cache talk page on wikipedia: https://en.wikipedia.org/wiki/Talk:InterSystems_Cach%C3%A9

Reply on news.ycombinator.com