VA’s foray into Internet of Things faced ‘catastrophic failure’ | Hacker News Reader

VA’s foray into Internet of Things faced ‘catastrophic failure’

mystatesman.com

56 pointsus0r36 comments

It sounds like the VA hospitals are suffering from major failures of protocol and procedure, none of which can be solved by IT systems, even if they work well. Turns out that computerized systems also require that humans follow protocol and procedure in order for them to function properly. But either way, these systems don't operate in a vacuum. Someone needs to analyze the real-world impact of these problems and develop a workflow and procedures that work with the hospital's organizational structure, physical layout, current and planned staff, training, auditing, procurement, patient scheduling, and accounting systems. No software contract for one piece with HP or IBM or Cerner is going to address all of that.

You're Spot on, the problems aren’t unique to IT, it’s a systemic policies / process / procedure failure.

> The VA has also vowed the project would prevent death and disease from unsterilized equipment, a persistent problem at the VA.

Oh my god how is this a problem in the modern age? Stop wasting our money on gimmicks and focus on adopting modern hospital practices.

Dude:

  colonoscopies with dirty equipment 
  led to a rash of hepatitis and HIV

...pretty much in the worst way imaginable.

Following the links the article provides:

  Army veteran Thomas Mayo got a colonoscopy 
  after the VA insisted on it - only to learn 
  it may have left him with a chronic disease.

  6 veterans who took the follow-up blood checks 
  tested positive for HIV, 34 tested positive for 
  hepatitis C and 13 tested positive for hepatitis B. 
  But there is no way to prove [blah blah blah], and 
  some [roll their eyes, accuse the victims of lying 
  and accept paychecks to play devil's advocate, 
  claiming maybe] the infections probably already 
  existed.

You pretty much have to figure anyone that cares enough about their health to subject themselves to an elective colonoscopy, probably has a firm awareness of their HIV/hepatitis status, before hand, including the patient zero carrying the source of the disease in each case. There are likely medical records capable of proving which patients were the infectious patient zeroes, and which contracted the disease as a result of the malpractice, even if those identities are never made public due to privacy law.

Agreed. This is tragic. These are the kinds of problems that I would have expected a military organization to have a pragmatic and robust solution to. I doubt you can rely upon a solid wifi connection in a field situation, so I bet they have some pretty solid procedures in place so that a human reliably find what they need without the need to google it.

Veterans Affairs is a cabinet level department, it isn't part of the DoD.

It serves former military but it isn't military.

That almost makes it worse. The fact that this level of incompetence nd negligence is coming from a cabinet level department is nauseating.

Well, the Department of the Interior runs Indian Health Service with similar results and has been thrown off the internet because of incompetence, it’s not exactly surprising seeing a cabinet level department being this dysfunctional.

I still wonder why the VA isn’t part of DoD?

I believe the primary reasoning is that the DoD is focused on national defense and military matters, and veterans care is a very different thing. In practice, the DoD will always prioritize military preparedness over care and benefits for former members of the military. US history (and I assume most history) is filled with examples of neglect of veterans' needs and failure to pay promised benefits in a timely or effective manner, from the Revolutionary War till today.

The actual promotion of Veterans Affairs to a cabinet-level department in 1989 was in fact driven by the perceived need to ensure that more attention was paid to the government agencies that administered benefits that at the time covered a third of all Americans. Of course, whether that works or not depends a lot on Congress's willingness to adequately fund the department and the President's willingness to prioritize the VA over any of the many other priorities they must balance.

At this point, the funding for the VA isn't the problem. They are beyond incompetent with the latest idiocy of hiring people with revoked licenses. The DoD does a lot, and it cannot do any worse at protecting their own. They at least won't take months to just signup.

How much does a VA doctor make?

They're in the bottom 10th percentile of doctor salaries in my metropolitan area (middle of the country, not very high COL).

Recruitment of docs at the VA isn't funded like recruitment in private practice, so they can't advertise as widely and can't pay for prospectives doctors' lunches (much less whiskey tastings). Credentialing now takes six months, due to the previously-mentioned high profile hirings of people without credentials. There's reasonably high satisfaction among docs actually at the high-performing VAs, but people tend to work there because they believe in the mission or don't want to deal with insurance company rationing of care, not because they want to earn lots of money.

source: family member is VA doc attempting to hire more docs to the VA. If you know internal medicine doctors who are looking for a place where evidence-based care is prioritized and insurance status isn't a factor, tell them to apply to their local VA or one of the high-performing ones they might want to move to! (https://www.va.gov/qualityofcare/measure-up/End_of_Year_Hosp...)

This seems insane how are not all US hospitals using best practice in this regards the savings in insurance for wrongful death suits would be quite large in its self.

It is mind-boggling how anyone can claim tracking equipment with computers all the time is a cost efficient and robust mechanism to ensure sterilization. Where these simple things go wrong complexity is never the answer. Why nobody challenged these assertions is a mystery to me.

Oh my god how is this a problem in the modern age?

The modern age? 100 or even just 50 years ago hospital wards were run by "matrons", fierce and experienced nurses who ran their wards like a Sergeant Major inspected a Guards regiment, everything immaculate. That system was scrapped in favour of what we have now, a fluffy blame-free culture where no-one is responsible for anything. The problem of failing to maintain standards is entirely a "modern" one, sadly.

This sounds like a nostalgia-tinged anti-political-correctness view; a hundred years ago was the end of the first World War, and the controversial introduction of hygene was in living memory.

Could do with some stats either way.

To be fair, you're discounting the possibility that nostalgia-tinged anti-political-correctness views are correct ;-)

A hundred years ago was also the outbreak of the Spanish Flu pandemic which killed 3-5% of the global population, not exactly a golden age for public health.

> the department’s $543 million contract with Hewlett-Packard

> The system [...] has been hailed as a way to potentially save millions of dollars

Does not compute.

At first I thought that was their entire hp purchase number. Covering laptops, monitors, desktops, etc.

Then I realized that nope. It's 543 million of professional services to build a RTLS system. Wow, I'm speechless.

Where did you see in the article that it was just for professional services? The way the article presents it, that number includes everything related to the system (hardware, software, training, and so on).

> Today, the VA spends more than half of its $4.3 billion IT budget on maintaining increasingly archaic legacy systems.

Not difficult to see how they’d claim their Great New Thing would cost less to maintain than the Really Old Thing that nobody likes which was built by someone else.

> employees were shunning another troubled IT program, called Catamaran, that manages supplies through predictive analytics

> employees complained of frequent crashes that made it difficult to order supplies

Also, if they stopped infecting people with Hepatitis and HIV (most likely among other things) due to using dirty equipment they would realise savings through not having to pay compensation and/or treat those people for the rest of their lives. That could also be done by following proper cleanliness procedures but that would require some leadership to achieve.

The VA complaining that tags are too big for small things and HP complaining about the wifi tells you the level of intelligence going around here on both sides.

Sounds like the VA wants out of the contract, so both sides shouting how the other didn’t meet the terms is inevitable. HPE will wind up getting a few (hundred) million to go away and the cycle of failed projects continues.

Honestly, a mature institution like the VA probably ought to be spending a significant portion of their IT budget on maintenance. There's a lot of systems to maintain. And new IT systems are not always the solution to problems, especially problems that aren't actually new.

Since the phrase "archaic legacy systems" is left completely undefined, we don't even know what this means. "Legacy systems" can include stuff that was written a few years ago. I think it may actually be a bigger problem that nearly half of their $4.3 billion IT budget is being spent on _new_ systems. If this half-billion dollar contract with HP is any indication, that money is being poorly spent, and it would be better to invest in the existing systems, instead of trying to solve problems by adding more future-legacy systems to the mix.

archaic legacy systems = paper and Windows XP.

What they should be doing is testing in a single location first to get the kinks on the system out and define facility needs, then expanding to 10 and then moving to roll outs by region. None of the problems that they have experienced should have made it past 6 months.

I'm starting to think that it's purposefully incompetent people at the top that are causing problems, because without a doubt I could have run this program better but I'm pretty sure I couldn't get the job because I don't have a bachelor's.

There is a big story here that is not being told, alluded to in one paragraph of the parent article. "[R]eplacing the decades-old VistA medical record system."

VistA used to be the VA's electronic health record system. It is open source. It used to be maintained by the VA IT people with a host of third-party plugins. The name of the part physicians interacted with is CPRS or VA-CPRS. It has a good track record in improving prescribing accuracy.[1] It's cheap. [1] It's tops in physician satisfaction, for several years. [2] It definitely needs maintenance, and isn't getting it any more. Third party vendors have stopped producing for it, too. A big proximate cause: they're ditching the whole system to outsource to Cerner. Cerner makes a not-very-popular electronic health record system. Cerner did make more corporate donations to Republicans than Epic, the biggest EHR provider in the US. [3] Now, Cerner was already working for the DoD and there is some effort to harmonize systems, so it made sense in some practical ways, but it was a no-bid process. Spending close to $10 billion of taxpayer money without any bidding process... ugh. The lawsuit about that has been dismissed, though.[4]

Jared Kushner was pretty proud that the VA chose "in two weeks" to go with Cerner and ditch open-source VistA. [5]

[1] https://en.wikipedia.org/wiki/VistA [2] https://www.medscape.com/features/slideshow/public/ehr2016#p... [3] http://www.healthcareitnews.com/news/see-what-top-ehr-vendor... [4] https://www.bizjournals.com/kansascity/news/2017/10/24/judge... [5] https://www.beckershospitalreview.com/healthcare-information...

By paying consultancies half a billion dollars on new product development, they could potentially save millions.

thats not even close to the final price. they'll probably drop another $250 mil then cancel the project. they dont even have this running in 1 VISN and there are over 20.

Having first hand experience with the VA I can't say I'm surprised, but really it doesn't matter that it's the VA, because I think most companies doing iot things already are likely going to face failure on their project as well.

In this case I'm mostly thinking about the horrible security environment of the iot ecosystem, especially cellularly connected ones.

"VA officials worried that the department’s $543 million contract with Hewlett-Packard Enterprise Services to implement a real-time locating system, or RTLS, was careening off the rails"

Wow. Let me guess, good ol boy contract kick-back where the good ol boy can't deliver as promised. It's a classic American tale. Great thing the government bid system is blindingly fair and egalitarian. Yup, that's it I bet.

"VA officials have said work here stalled about 18 months ago due to contract issues"

> When it was awarded in 2012, the RTLS contract represented an opportunity for the VA to shed its reputation as an old-school tech dinosaur and chart a path toward what leaders were calling a “transformation initiative.”

That sounds a lot like a mid-life crisis.

That sounds a lot like ”I need a gold star in my CV to boost my career”

Transformation was a $500 word that McKinsey folks and their ilk throw down. It amusing, as the McKinsey people will tell you that 70% of these things fail, but you will succeed, because this time, <reasons>.

They usually deploy this stuff to enable centralization of large organizations with broad geographic or functional scope. These things always fail, because you cannot achieve cost savings and remotely manage the broad scope effectively.

An example of this is the ridiculous project fractured here. Tracking crap with WiFi is what happens when some CIO in HQ “solves” a problem on the hospital floor.

This is usually synonymous with “devops” transformations as part of huge personnel problems (read: no young people want to join / stay) in most large, old companies that are facing a form of a late-life crisis that was plainly visible decades ago to anyone but the willfully blind.

The standard situation of the usual F500 and government technology stack is a Petri dish of half-finished projects by an army of consultants maybe a couple years out of school charged at $300 / hr to the customer and everything is solved by throwing bodies at problems when a square peg was sold for a non-Euclidean hole because the decision-makers are so directly inexperienced in how technology works that 25-year old tech sounds like it’s cutting edge on a marketecture deck (or presume that something written recently couldn’t possibly be archaic already). It’s a lot like failed relationships with dozens of illegitimate children that you’re paying child support on.

why would you track a catheter surly these are single use items that are never reused

The system is intended to cover all aspects of inventory control, not just tracking the location of reusable equipment. It's about knowing what they have and where it is.

Reply on news.ycombinator.com