Visiting Us
epic.com
epic.com
Epic’s main problem is a lack of clear internal code ownership. Everyone owns all the code. This means that even if you clean something up, someone on the other side of the company may come in and mess things up again.
This led to really defensive programming where developers would never refactor, they would simply add a new if case for their new functionality somewhere deep in the code, then prop drill the data down. This led to every core function having over a dozen parameters and hundreds of branches. It eventually became impossible to reason about. Cross team calls were just function calls rather than defined apis. This made it fast to develop code initially, but terrible to own long term. This mainly applies to their Mumps code.
While I was there I felt like Epic was beyond saving, but with a big push there may be something they can do:
1. Enforce some level of code complexity. Best practice is 40 lines per function and no more than 4 parameters per function. Epic probably shouldn’t shoot for that, but a 100 line limit and 6 parameters per function would already be a huge improvement.
2. Enforce strong code ownership. Epic has many people who are there for life, let them cook. Epic should segment off code to certain teams so those owners can fix it at their leisure. Cross team api calls should be clear API contracts. It would require some more discussions to get feature requests approved since not everyone can do anything anymore, but the code would gradually improve.
Epic is too important to fail. I hope things have started to improve since I left.
I think this is better than requiring teams to make all changes themselves which slows things down significantly considering each team has their own roadmap and priorities
Requiring reviewers isn't sufficient. It needs to require a cohesive review strategy that adheres to a long term product vision for the software component in question. And my experience, though not at the two companies you mentioned, is that it doesn't happen and you instead get a lot of thoughtless "yup, looks like code" approvals.
Upholding a strong vision is fine. But if you want to be a blocker, you've also got to be quick. The alternative is a bureaucratic death spiral.
If I were to enforce some kind of arbitrary code complexity threshold for functions I would put a cap on the limit of possible of branching combinations based on parameters within the code. Like around 16 (branching combinations are exponential).
For example a function with 20 parameters but only one if statement is fine. A function with 5 parameters but several nested if statements is not.
My previous company had a step in the CI/CD that would fail if the nesting level of a function got too deep. (Like, "an if statement, which contains a foreach loop, which contains another if statement, which contains...")
- There were no PMs. No one was chasing vanity metrics. No endless barrage of a/b tests. No growth hacking. Instead, product was built the old-fashioned way - by talking to customers; quite often, customers would reach out to us! "Please build time-saving feature x", "support new medical procedure y", "help us publish more research by analyzing z". The heap of ideas was large, and teams were free to apply their own ranking functions. Some top-down strategic initiatives were threaded through all products. This led to every release being packed with things that customers wanted.
- They leveraged their advantages. Plentiful, cheap land → everyone had an office with a door, possibly shared with 1 other person. Productive people flourished. On the flip side, those who were languishing fell faster and deeper into their holes.
- They learned to live with their weaknesses. Not everyone wants to live in WI, and many who are willing to try end up looking for warmer pastures in a few years. Epic ended up being an early-career transit hub. Their attrition rates would cripple most tech cos, but they shored themselves through extensive training programs, and by rewarding the anchors that stayed.
- They took pride in the product. Every month, the CEO personally presented the latest qualitative assessment of each product (as assessed by an independent third-party). If your product slipped from green to yellow, the pressure would trickle down to you. Also, there were no outbound sales, because having the product speak for itself was the sales strategy.
Of course, not everything was rosy, but much has been said here about the shortcomings of Epic, and I wanted to point out what's contributed to the immense stamina that has kept them in the lead for decades.
https://www.newyorker.com/magazine/2018/11/12/why-doctors-ha...
It seems to be the classic curse of enterprise software: it's the bosses you need to keep happy, not the users.
I work as a university professor and educational software also tends to suck, for what I assume is the same reason.
Think that might be a plus. PM/PO has ruined the industry. This way at least one has a direction to the customers, which is something I can't say about large companies.
Another thing I thought was interesting given modern dev practices is that you don’t even touch production code until after about 6 months of training, including exams and testing on the actual functionality of the product, even bits you’ll never interact with personally. They’re serious about making sure you know what you’re doing before you get within a mile of something that could affect patients.
They also have a pretty strong accountability culture. I shipped two fairly embarrassing bugs my first year and had a very serious conversation with my manager about whether I should continue to be employed since I’d used up about 50% of our team’s allowed bugs for the year. But on the other side, once I got my feet under me, they were very good about recognizing improvement.
Epic has a reputation of hiring lots of new college grads. For software engineers that's not exactly uncommon, but Epic actually has a lot of employees working under titles like technical solutions/implementation solutions (or something like that): the people directly supporting the hospitals using Epic. Because these are pretty specialized roles, Epic has a very formal and fleshed out training program for their new hires with classes and courses and such, and it can take months to complete. They not only have their giant campus in Verona, they have an entire training center there, a huge auditorium for allhands, and a very streamlined recruitment process (for a ~21 year old it feels over the top luxurious). Although Epic does hire from more selective schools it seemed the majority of their new employees are from state schools in the Midwest.
They also are private despite their size, just basically don't do M&A and are, relatively speaking compared to other big corporates/tech companies, in the middle of nowhere.
In corporate America this is a highly unusual way to operate. I think it's underrated how big of a "risk" all these heterodox corporate strategies are for an executive and it speaks to amazing ideation and execution on Judy's part. Also, even though Epic does have a decent amount of turnover, she has taken a chance on tens if not hundreds of thousands of young people who didn't have the skills she needed them to have already, by giving them months of training and a really solid start to their careers.
I went to the University of Wisconsin for grad school, and I knew lots of people there who went to work for Epic.
I confess that, before I applied, I imagined that Wisconsin was nothing but cow pastures and football. I imagine this sentiment is shared my many young job-seekers outside the Midwest. Now, having lived there for four years, I have to say that UW is an amazing university, and Madison is my favorite city in the US.
winters would be rough but the summers would be nice. culture seemed neat. kind of like a smaller, colder Austin.
For those not from US this has nothing to do with EPIC Games and I assume no part of Unreal is using MUMPS for those who are reading comments before they click on the link.
This Epic has something to do with healthcare software. And kind of surprised how many people who used to work there appeared on HN.
I get the sense they're slightly less disliked than Meta or Amazon.
The one great thing Epic did for me was get me to Madison, WI, an amazing city of great people where I found a much better job and stayed for many years. I still miss it sometimes.
This 2007 classic explains how a case of MUMPS progresses when you’re a programmer:
Source: was on a team that was performance sensitive enough that I spent a lot of time in the actual transpiled MUMPS code that did look more like this article.
Every small clinic wants special custom formats. Insurance eligibility providers will lie about their abilities. Some insurance companies don't integrate with any eligibility check software. Sending health info anywhere requires yet more interchange formats.
The actual note taking bit isn't too bad, and can be easily modeled with most flexible CMSs even. The problem is that you can either gamble on a startup offering everything that a company that has been in business since before you were born and actually being able to follow through, or you can just go with the established company like everyone else does.
Having seen several startups trying to break into the space, it's no surprise that epic still dominates. The regulatory hurdles alone are not trivial; clinics that take Medicaid are usually required to use an EMR with a CEHRT certification which can cost tens of thousands of dollars to obtain.
I am amazed at some of the software Epic has built for itself over the years. Using its own database product (the backbone of the product they ship to customers), they built their own code review tools, design doc review tools, project management tools, time logging tools, etc. There is a unity and cohesion to the process of getting things done at Epic, better than my experiences at big tech.
It is very easy to answer questions like "how many dev-hours were spent fixing bugs caused by the code written to implement project X?" or "will there be any days next week where every dev who has contributed to codebase Y will be out of office?"
Imo they could really benefit from staffing infra/tooling teams better. Too many product devs, not enough devs tackling the low-hanging fruit that would make product devs way more productive.
Imagine the Idiocracy healthcare scene. But now you have a menu Diving system within a dense jungle of ten million options. I've never had to click through so many steps to get something very simple done. And I've never had so many completely useless features available
The system is also built for the American healthcare system, but they also sell it to Europe with deceptive practices, with no simplifications whatsoever regarding the less overbearing legal landscape.
As an end user: it's a shit experience to use no matter what you end up using it for, the price tag is an Astronomical Unit but the support you get for that is everything but stellar
We DESPERATELY need more companies to structure themselves like this.
Is it because the workers at a normal company would jump ship if Epic’s culture were imposed?
If companies like Cerner, Google, Apple, Microsoft, etc. are all able to acquire and integrate software that was initially developed by others, why not Epic? Surely Epic is not less competent?
Technically speaking I think they see buying software + changing it to be more like Epic (or continue to operate/develop/support it independently) as a waste of money when they could spend that time/money on improving what they already have.
I also think it's probably a branding/marketing promise to their customers that if you buy Epic software, you're not going to have that contract morph into one with some other generic corporate company who causes problems trying to integrate/migrate your setup. Nor will you have wallstreet begging Epic to juice their customers for all their worth just because they might get away with it and it'd make the stock look good temporarily. (I have no idea how expensive Epic is comparatively but I do know EHR is very difficult to migrate from and Judy is known for playing the ultra-long-game).
I think basically the idea is that acquiring another EHR vendor would only be for the benefit of expanding Epic (the company) market share but detract from making Epic's software a better product for their customers.
Since Epic has good relationships with its customers, working with them to build that expertise from the ground up is considered better.
I can't tell you for sure, but the way it was pitched in the recent Acquired episode on Epic is that no, they* aren't able to integrate software from acquisitions well.
Without knowing this in detail, it sounds like the choice is between one system that does everything, and a patchwork amalgamation of systems, databases, UIs, which are not well integrated.
I can't say this for any kind of fact; it's just the impression I get. It seems highly plausible to me.
*Meaning, Epic's competitors
If you're a small company, then you can "acquire" a larger company which then absorbs the original company. You can fund it through borrowing against the company you're buying.
You then let the large company eat the small company despite on paper the ownership being the other way around.
no new execs from acquisitions coming in, no risk of spinning off in different ways because of new blood, just organic growth
Publicly traded companies have accountability to more stakeholders in additional ways.
A private company isn’t necessarily better for being able to avoid that accountability.
One interesting one is vertical vs horizontal software dev. Epic's advantage is deep domain knowledge of healthcare (relative to competitors at least - a Dr or nurse will dispute that lol).
At Google, Apple, Microsoft they want to make software for everyone. Epic hyper focused of their niche and has decades of knowledge built into the business logic. It's also why the aforementioned companies have failed to take a piece of the EHR market despite more technical knowhow and huge war chests.
Lastly, Cerner is a bad comparison since Epic ate their lunch over the years. If anything it might be a data point that their approach is poor. Cerner rev cycle still doesn't work and Oracle has said they are scrapping the product they spent $28B on.
It turns out companies are transient or have been internally infiltrated by such (outsourcing- and ambition-driven) politics that any mission is more supplement than reality, and there's no sense of controlling your own destiny.
So perhaps the dream persists out in the tech boonies in the ultra-sticky EHR domain, goosed by the Obama/insurance mandates to digitize, where developers are trapped by unportable skills. (Or perhaps in smaller B2B companies filling a niche.)
If anyone's interested in Epic and wants one employee's opinion, my email's in my profile.
This is the way.
About 8 years ago I was working on a mobile game where you could purchase specialized dragons and eggs. Some of these could be pretty expensive, but since they were high end items we wrote special GPU shader code for them so they had cool special effects on them. We tested these as well as we could -- we had a room with maybe 100 or so mobile devices -- but of course we couldn't test on everything.
One day an irate older lady came to our office, and our receptionist for some reason let her in (probably thinking old lady = harmless?). Keep in mind our office was unlisted because we didn't want fans dropping by. She had driven all the way up from Arizona to Colorado (although I don't think it was the only reason she drove up), and she accused us of ripping her off, because she had bought one of these fancy dragons and instead of getting what she saw in the promo materials, its wings were black! I didn't hear or see this directly, instead it was the main topic on our Slack chat with everyone being cautioned to Play It Cool.
I didn't think much of it until I realized it was _my_ code that had caused this entire issue in the first place!
Luckily we had a really good customer service guy that defused the entire situation, but that's the first and hopefully only time I've been tracked down in person by a customer for a bug.
Almost fifteen years ago, another of my former employers had an angry customer storm into their office and start shouting at the front desk associate. Soon after they relocated, and hired a full-time armed security guard to process all visitors straight from the elevator lobby. Any further, and you'd need to present photo ID and maybe sign an NDA before he'd buzz you in.
Not long after, at an engineering holiday dinner, an exec was asking everyone about their hobbies. That same engineer proudly shows a photo on their phone, of themself posing with a huge black tacticool sniper rifle.
I might've looked taken aback, because the exec then asked if I shot guns. Nope.
Refer back to the old saying, in the training.
The engineer did good work, and seemed to be a decent person, but it's still a little funny that they turned out to be immune to the layoffs (by that same exec) that eventually got me and almost everyone else.
The conversation then went a bit like this:
Head of support: well you know, there isn't supposed to be a round hole in the middle. It looks like it's been damaged. In fact, it looks like a pen was pushed into it.
Mum: [Aghast] (to the boy) did you push a pen through the speaker?!!
After some shuffling the boy admitted that he had. At that point the mum was just going to leave in embarrassment, but our head of support insisted that she take a spare speaker, after having come all that way. No doubt the boy got an earful on the way back.
Still, I heard working there was quite good. Obviously not FAANG level salaries, but after you left and completed the 1 year non-compete, other health care companies and/or hospitals would pay a good premium for your MUMPS expertise.
(None of the above is sarcasm, BTW)
For it's relatively small size it plays an outsize role in the US healthcare sector. So in the same way ASML is an interesting business due to its niche role, this is similarly... Though with lower stakes
Incredible place, super detailed, and I loved the cafeteria setup they had (great food too.)
I definitely got a feeling that folks got burned out pretty quickly though.
The posting are pretty up front about that requirement too, it's not like that is sprung on the candidate after they apply.
> This extraordinary building is a playful tribute to the timeless movie
Lyman Frank Baum would be surprised to read this :)
That also made it an excellent proving ground for me, to be fair. I've never doubted my sysadmin skills after that.
Epic sucks. It only sucks marginally less than all the others. As a corp, they are expert at abusing mechanisms for lock-in as well as network effects
If you all were forced to use tools as shitty as the EHR's, no one would be a software engineer
A re-write of VistA would be the way to go, but someone paid off someone, and the VA is doing a disaster of a changeover to Oracle's Cerner
tl;dr Healthcare software is a steaming pile of shit
https://isthmus.com/news/cover-story/opportunity-lost-epic-n...
They're also vehemently opposed to remote work, to the point that during COVID they tried to force employees back into the office in August, 2020 (!) in violation of a county public health order (!!!):
https://www.wpr.org/economy/workers-officials-urge-remote-wo...
Epic's Glassdoor reviews are terrible. Several personal friends each lasted less than a year at Epic out of college before finding new, better-paying employment elsewhere. Since Epic is privately owned and its founder and CEO has stated she'll never sell, its corporate culture will never change. It's better than no job at all but if you have other options, avoid.
Not of software engineers, those are decently treated, but the reputation for other employees is questionable.
Epic has 4 major roles(and a bunch of support roles): software development, implementation services, technical services, and quality management.
Software developers are well paid for the area. While it can vary between teams and supervisors, a competent dev should be able to avoid being overworked.
Implementation services travels a ton to go setup new customers. It's definitely a quick burnout position if you don't thrive in that atmosphere. But the ones who do have some of the fastest compensation growth.
Technical services are by far the most overworked because they are assigned to support customers long term. The baseline expectation is 45 hours a week, and most are usually assigned to enough customers that it can exceed 50-55 easily. I would consider Epic to be doing a poor job keeping them from burning out.
Quality managers, who test and document the software, can be overworked depending on team. They are definitely underpaid. They have been the plaintiffs of previous lawsuits against Epic by employees.
The non-compete is only really effective at making employees wait a year before going to work directly for a customer. I've heard of people getting jobs at customers and just not working directly with Epic until after the year has passed. Developers can easily just go work for a different tech companies right away.
The Covid and remote work stuff was pretty bad. At least they backed down in 2020 after complaints to the county. Unfortunately it took a suicide in 2021 for them to ease up on the "must only work remote in the local area" policy before they started bringing us all back to office at the end of the year. At least they never gave us the impression it was long term like some companies did.
Personally I think Epic actually does a pretty good thing training up and employing so many new grads with skillsets that don't find it as easy to get solid corporate jobs as SWEs (a lot of their TSEs are STEM-but-not-CS grads, implementation people seem to be ~anything). They do expect something back from those employees in return, but they're paid quite well compared to their alternatives and given a lot of support/structure to ease into their first job.
IMO it's a place I appreciate a lot more in hindsight than I did while briefly working there, and I don't think that's an unpopular opinion
> I also think their employees might overestimate how much greener the grass is on the other side, because they haven't actually experienced any comparable job
Many people continue to regret their time at Epic even after having left and having seen the other grass first-hand.
> Personally I think Epic actually does a pretty good thing training up and employing so many new grads with skillsets that don't find it as easy to get solid corporate jobs as SWEs
Count the MUMPS training to essentially be a waste of time, unless you like the idea of writing MUMPS going forward.
Also, your endorsement reads as if Epic should be an option of last resort rather than a place where a software engineer should want to be.
https://captimes.com/news/madison-home-prices-increased-the-...
And living in Madison means commuting to Epic in Verona, which means contending with Beltline traffic just as bad as 280 during rush hour.
Staff use cow bikes, cow carts, or cow vans to mooove across campus.
It was cute when Gateway did it, still cute when FatCow does it (it is in their name), gettin’ a little cringe for the late-comers, though.
OT for this interesting discussion about Epic but one of the reasons that Wisconsin slipped to #2 in milk production back in the 90's was that northern tier milk producing states instituted stronger environmental protections back in the 1980's forcing changes in how animal waste in huge dairy operations had to be managed. At the time there was a lot of pollution due to runoff of waste into creeks and rivers causing a cascade of problems with water quality downstream of these huge dairy operations.
A large number of these operations, instead of complying with the new environmental rules that would force better waste containment simply looked around the country for areas that had less strict requirements, lower enforcement, and an established dairy industry. Many of them ended up moving to counties in north central Texas and buying up large tracts of cheap land. They shipped their herds south and went on about their business down here just as they had up there. Erath County became one of the state's largest milk producing counties as a result of this migration.
Texas has been a cattle state for generations but most of the larger operations were traditional ranches of 10's of thousands of acres with lots of open land where cattle herds were measured based on # acres/cow metrics. Feedlots where cattle were fattened just before slaughter were the only places where you had large herds crammed into a small area. The dairy industry bought much smaller properties, up to several thousands of acres, and their dairy herds were crammed into these smaller spaces with the wastes being largely allowed to run off into creeks and rivers. It took years before Texas stepped in to try to get a handle on the agri-waste runoff pollution of their rivers and many large rivers in the state still have contamination issues.
Not coincidentally, this same issue affected swine operations in the US causing concentration of pig farms in North Carolina due to lax regulations.