Building a patient monitoring system with Go and Vue in 3 days
kasvith.me
kasvith.me
The circumstances are pretty exceptional, two things shocked me a bit: 1. The manufacturer already produces remote monitoring software for the device but its _too expensive_ 2. Software like this is usually classed as a medical device, which comes with regulations (e.g. https://www.gov.uk/government/publications/medical-devices-s... in the UK)
This is (possibly depending on use) a biomedical device. These are regulated such that we pull separate cat5/6 and put up chain link in network closets. The Cisco switch in one side of that fence just vanilla corporate IT with email, EHR, Netflix etc; the same switch on the other side and all the stuff connected to it - from the bedside monitor out through the central alarm station - is a medical device.
That isolation is the presumption referenced in the recent GE vulnerability [0] and the challenges of getting bio med chocolate in the corporate (ehr) peanut butter presents significant challenges [1].
[0] https://www.fda.gov/medical-devices/safety-communications/cy...
[1] https://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfStandar...
HL7 and FHIR are a major step in interoperability for health, and they should not be taken for granted.
I've worked with integration projects with EHR (Electronic health record) Systems, and a lot of hospitals still have proprietary formats for exchanging data with limited documentation.
If you ever work on any health related system, consider starting from those standards!
EMR systems lock us (healthcare systems) into proprietary formats to make migration to new clients difficult; likewise, we don’t (didn’t) push for interoperability because exporting data to other healthcare systems cost us more than it benefited us (we didn’t fight it, we just weren’t gonna spend our money to make it happen).
HL7 and other interoperability tech only emerged because CMS reps made their way to -a lot- of conferences and, with varying levels of bluntness, said “find a way to improve interoperability on your own, or you’ll see it show up in federal regs 12 months from now.”
This is a change that came from active regulators doing their jobs correctly, in spite of the active efforts of the tech industry and the indifference of the hospital industry.
So, I rarely say this but, thank you CMS.
I mean, to be honest, it makes sense... locked platforms are just death.. especially if you can't get your shit off it when it shuts down.
I write enterprise medical software for a living and if a hospital had told me they were desperate for a solution in 3 days, I would exhaust other options before writing new critical software.
I want to believe with enough indications of the monitoring itself (Is the screen being refreshed? Is network connected ? When was it last checked?) that this is a viable option and that would unlock a lot of possibilities.
But I can't shake off the feeling of uneasiness (What if it's not working and we don't know it's not working)
PS: I'm not bashing on what was done, just the general idea.
I wonder what useful information you could extract from this. Like a very slight downward trend in blood pressure over a day. Could this be used to detect something like a slight internal bleed after a surgery?
- https://link.springer.com/chapter/10.1007/978-3-319-43742-2_...
I personally would take Grafana with Prometheus (as it has great Go library) and reduce the whole project to some kind of HL7 wrapper.
> We created this software on a request from healthcare staffs. It is not a commercial application. Even with this system, we strongly suggest doctors to visit their patients, take real measurements. As this software was developed fast due to prevailing pandemic situation, we released it with the most urgent feature monitoring. We tested this for long run, with multiple devices as well. So far it worked out well. It does not indicate this is perfect, we are working on improvements and fixing bugs until its very stable. Thus we have adviced doctors to use this with CAUTION
It's almost as if the options were either "Do nothing" or "Do something", and they chose to "Do something." This does not sound at all like someone just decided to build their own version of a software they already had access to. It sounds very much like they could either make this software or try to get along without it.
I understand the desire to "do something", but the certification and testing process for something like this is how you find out if this actually improves patient outcomes or not. Sometimes "doing nothing" in a certain area gives a better outcome.
I'm glad they're advising doctors to use caution.
I have some architectural concerns about how it's built (a browser's javascript engine is not reliable enough to be used as a safety-critical alerting engine!), but I can see how it's attractive.
Even if they're going to roll their own rapid uncertified solution, they could have still done so with more reliable and proven technology than what they appear to have chosen to assemble this system.
Many components for making safety-critical systems are commonly available in non-certified variants. These could be used to assemble a viable system, but would of course lack the process, practice, and evidence for certification that would normally be required. But, at least the technology choices would have, at a minimum, "proven in use" benefits if one is going to otherwise punt on the procedure & practice rigor that is typically applied.
Given that most of the components to build a safety-critical system are commonly available, or their non-qualified kin are, it doesn't make much sense to me why if you were tasked with putting this together quickly and cheaply, but were still taking serious the safety-critical nature of it, why you would cobble it together with the stack chosen here.
Even if the choice is "Do something" or "Do nothing", there's more than one way to "Do something", and this seems like a particularly odd way to have gone about it all things considered.
Can you elaborate on this?
That's not to say that the use of any of these things would necessarily result in a qualifiable system (it wouldn't), but you'd at least be in the same general vicinity of what is usually used/required to build such a system.
But, let's consider even the simplest aspect specific to the system this article is about. There are already multiple open source HL7 parsers that have been in-use, tested, and maintained for quite some time. What was the point of reinventing that wheel here, and if one was going to reinvent that wheel, then why not in a way that improves assurances rather than reduces them?
But, even if one concedes that the technology stack they chose was somehow the only one available to them, then what level of rigor should be applied to adapting a stack that would typically be considered unfit for this task under normal circumstances? I couldn't find any mention of their testing methodologies.
"Crisis" seems to be a scenario that your methods might not work well in. Groups of people were at risk of death. Would implementing a solution increase their chances of death? If you're continuing to monitor patients as you normally would without this tech, isn't worst case just "doesn't improve"? And isn't best case "We caught more problems than we would by normal monitoring" ?
The "worst case" that can come out of false positives & false negatives in safety-critical conditions has a bunch of dimensions depending on an enormous number of environmental and contextual conditions. There's no PFMEA nor DFMEA supplied in the OP to indicate how one should consider their choices.
2 orders of magnitude less means there is a lot of making do with what you have.
Similar concept with different data that we are deploying in US hospitals. Using React and Hasura.
Please everyone reconsider the uses to which you put your amazing abilities. Code could change the world if we built more useful things, less blogs, less shopping carts.
Closing point.... Once a project is dealing with hardware it's 20x harder. But the outcome is better. In code as in life, when faced with easy choice or hard, always take the hard one. It's how we grow as people and extend our skills.
I don't care that much if it has a display, just being able to access the information similarly to what's described in the article would be enough.
Channels are not lock free: https://github.com/golang/go/issues/8899
I'd like to read about those challenges as well.
They also have a solution for an ICU like monitoring system with several beds, but I have not used this functionality.
"Real-time" is now being used for streaming APIs to mean "small latency on average, but no guarantees".
Maybe you guys ought to call hard real-time "Guaranteed bounded latency" when needed.
Kind of how you'd say you have a dashboard for your business to see metrics in real time (as opposed to some kind of daily report).
I think that usage of real time is very common outside of a pure engineering sense, so you're being a bit of a stickler by being annoyed at this specific instance ;)
My Phoenix channels, your rails action cable, your web sockets: soft real-time.
Real-time was reserved for true real time guaranteed delivery times.
The non strict usage is far more common then the term of art. One would be well advised not to assume "real time" is being used as a term of art unless specifically called out, or clear from the context.
I'd say common usage of the phrase "real time," means with respect to human perception, and that is how its being used here.
So do you consider YouTube a real-time system? After all, images are moving on your screen "in real-time" from the user's perspective.
It seems to me that these are all streaming systems, and when the data is "live", then it's "live streaming".
Even colloquially, I hazard that "real-time" has some implication that there's some correspondance between your temporal reference frame and the original temporal reference frame. I think the engineering definition just made that correspondance precise.
Yes people do say "real time playback" with respect to video. Generally, however, that is only noteworthy to call out when its unusual. Such as "real time 16k playback."
It's been 5 years since I've worked in that industry, but I still view the use of "real-time" the same way and can understand some people's issue with the term in this case.
I think calling it a "streaming" or "live streaming" interface is a better descriptor than "real-time".
> I think that usage of real time is very common outside of a pure engineering sense
Maybe, but we're talking about the software engineering world here aren't we? I think we should seek to be precise with our terminology and try to avoid unnecessary ambiguity.
I think both uses of the term are acceptable in their different contexts. (But yes I also agree "streaming" is a very good term.)
I view real-time as "the clients reflect the true state of the world without taking action." So if something changed the state of the world, all clients paying attention to that should also be updated.
That's a perfect description of "live streaming".
Medical data in one case, moving images and audio in another. The properties are the same in either case, that's why the term fits.
Saying context doesn't matter is just ignoring how everyone else uses language and saying that your view of it is right.
So now we have two terms referring to exactly the same thing, which doesn't resolve the term conflation problem I initially wrote about. The "web world" context isn't that different from the larger software engineering world that they should repurpose engineering terms.
> Saying context doesn't matter is just ignoring how everyone else uses language and saying that your view of it is right.
I am saying that web programmers do tend to use professional language wrong, that it's unfortunate, and that we should try to correct it when it happens. I don't think the first part is controversial, but apparently trying to insist on precise engineering terminology when talking about engineering systems is controversial.
This page classifies a broad definition for real-time and then breaks into different categories (hard, soft, fail-safe, fail-operational).
Is the default definition of real-time "hard real-time", or does it always need to be defined by its classification?
My point around context is that no one in the web world is saying that soft real-time systems are hard real-time systems. That would be blatant misappropriation of a term. They are saying that "real-time" defaults to "soft" in the web world. Just like "real-time" defaults to "hard" in the embedded world.
I don't know if I would classify this project as the embedded world (despite it using hardware), because it's consuming a stream of packets and isn't at the actual physical device level. I may be misunderstanding what they're doing, but I believe this to be the case.
The broader reason why I commented in the first place is that your comment feels like gate-keeping of a term when most people are going to quickly pick up on the context of how the term is being used. I understand and agree with your point around hard vs soft, but I don't think it's as clear cut as you seem to think it is regarding what the default classification is in different contexts.
I'm not calling out this person specifically, I'm just hopeful that we can use more precise terms instead of overloading existing ones. I think "streaming API" or "live streaming" is a better fit for these purposes.
The earliest computers didn't even include an operating system, never mind one with real time guarantees as we think of them today. You submitted your program on paper tape (later, punch card.) and hours later, you'd get the output. This was the batch computing era. The idea of getting a (soft) real time response from the computer (which was the size of a room), was unheard of.
As computers evolved, real time systems came to refer to computers that operated on the opposite of batch computing. The user could interact with the system in "real time", rather than having to wait hours after submitting the job for the results.
Later on came the distinction between systems with hard real time (vs soft) guarantees. Computers didn't have the same CPU speed that we have access to today, so even audio processing required special real time guarantees.
Eventually, "hard real time" was shortened to "real time", which takes us to today.
Systems which have existed since the dawn of computing such as banks back in the 50's, or the IRS, still hold onto the batch vs real time nomenclature, possibly because batch processing systems still exist. Banks still process records nightly in batches, some even still run on newer versions of the same IBM mainframes, though some are starting to move to "soft" real time systems. (If you ever wondered why it can takes days for a check or bank wire transfer to clear, this is why.)
As someone who grew up with those systems, this is a major mis-statement of the terms.
In the early days the terms were "batch", "online", or online's near-synonym "interactive".
Real-time has always meant bounded-latency and guaranteed execution since the earliest days, to actual computer systems engineers.
It is a true statement that many people use the term differently today - much to the detriment of precise communication between humans. But there was never any confusion about the term until perhaps the 90's and widespread interactive screen-based applications.
Look, this is simple: literally all of our technical textbooks and engineering manuals agree on what "real-time" means.
If people were running around calling their smart phones "desktop computers", or "laptops", or "mainframes" you'd be very confused I think. Sure, smart phones and desktops these systems have a lot in common, even run a lot of the same code, and sure our phones have more computing power than mainframes back in the day, but overloading this term conveys literally no benefit and actually creates confusion.
Of course I'm pissing into the wind here, but whatever, if I've convinced even a couple of people to be more precise with their terminology, I'm fine what that.
There's a reason medical certifications are so hard to get, and medical software is so expensive.
You're storing patient information in postgres. What certifications do you have to assert that the patient data is stored securely, in line with your government guidelines on patient/medical data? There's a damn good reason this is the "holy grail" of information security certifications.
You've got critical alerting built into the browser window using JavaScript.
This "alerting" is the kind of critical thing that sometimes needs *immediate" intervention, or someone could die. What happens if your browser experiences a JavaScript error blocking processing? And your alerts don't fire?
What happens if they fire too often and you get "alert fatigue" because they're not tuned correctly or in line with the other alerts available at the bedside/nursing station?
How much testing have you done to correctly assert that you're interpreting the HL7 or other specs correctly? And aren't misinterpreting data for some conditions or types of individual?
The "throw things together quickly" startup mentality might (I stress might!) Be okay where it's the difference between nothing at all and something that can save lives, in a country like Sri Lanka, during a global pandemic, fine.
But afterwards, this is so much junk without serious thought and time put into certifying it.
Medical, Aerospace -- really, any safety critical industry where your code working or not could mean someone is seriously injured or dies as a result -- is an industry that needs disruption, but that disruption should happen slowly, carefully, and safely.
If this is some small town hospital in Srilanka, the choice is between an unaffordable certified solution and not having any monitoring. If Medical software didn’t bleed them dry, they wouldn’t go this route.
> disruption should happen slowly, carefully, and safely
Disruption always happens this way - same way Uber broke existing laws. Yes, few people will die. But this isn’t surprising when the alternative is even worse.
No, this isn't _necessarily_ the choice. Without a "false sense of security" that an imperfect monitoring system might instil, you have nurses and doctors actually doing rounds and checking their patients.
> Disruption always happens this way - same way Uber broke existing laws. Yes, few people will die. But this isn’t new when the alternative is even worse.
This is an absolutely horrible viewpoint to have. People dying because of "disruption" so a few companies can make a few more dollars is _never_ acceptable.
When it comes to solving technical problems? We are ever so happy to do exactly the same thing.
Any solution is better than no solution. Except when no solution causes people to stop trying to delegate an important responsibility. Which is quite frequently.
A crap solution crowds the problem space. If a better solution is possible, it now has to defend itself against the incumbent. Explain why it is more expensive, why people should be bothered to switch.
If you can't do something well, then for pity's sake let someone else try. Log away every cost of not doing it at all and then when you can justify doing it well, build your pitch.
This could likely be "good enough" for those that have no other options if open sourced.
“This thing has absolutely no evidence of reliability or safety in a critical environment” is not criticizing it for being less-than-perfect. It’s criticizing it for being possibly inferior to the status quo.
Here’s one simple example:
Staff gowning up for routine rounds are much more careful, and safe, than staff rushing into an emergency code. If this thing throws up even the occasional false alarm, its cost to staff (in exposure) could easily outweigh, massively, and reduced rounding requirements.
That’s not “oh, well that’s not perfect.” That’s “oh, that might be worse, masquerading as better.”
“Perfect is the enemy of the good” is a wildly irrelevant comment.
> The deadly virus can infect you with a very small mistake. As healthcare workers, our frontline has to wander around the isolation wards to check vital signs of a patient from time to time. This task involves disposing of the protective gear after a visit. All just to check some reading on a device.
> A request from health authorities reached us to develop a remote monitoring system for isolation wards. There are expensive softwares to remotely monitor them. But Sri Lanka might not be that rich to spend such amount of money.
I think you're wrong in this case.
edit: formatting
What makes you think the team covered enough edge-cases to be "good enough" software? Do you think the presentation in a single blog post is enough information about a system to determine its quality and reliability?
We have different interpretations. For me, TPITEOTG means:
Choose one: a solution that works well but is clearly not perfect, or no solution at all.
> Do you think the presentation in a single blog post is enough information about a system to determine its quality and reliability?
Epilogue FTA:
> We created this software on a request from healthcare staffs. It is not a commercial application. Even with this system, we strongly suggest doctors to visit their patients, take real measurements.
> As this software was developed fast due to prevailing pandemic situation, we released it with the most urgent feature monitoring. We tested this for long run, with multiple devices as well. So far it worked out well.
> It does not indicate this is perfect, we are working on improvements and fixing bugs until its very stable.
> Thus we have adviced doctors to use this with CAUTION
Many of the complaints in the OP were specious for the situation in play:
> You're storing patient information in postgres. What certifications do you have to assert that the patient data is stored securely, in line with your government guidelines on patient/medical data? There's a damn good reason this is the "holy grail" of information security certifications.
This is monitoring data from dying patients in a third world country. Do you really think that they should have spent a couple months making sure hackers couldn’t access patients’ vitals before putting into use?
> You've got critical alerting built into the browser window using JavaScript.
Yes, because that is the language of the UI toolkit they are using.
> This "alerting" is the kind of critical thing that sometimes needs immediate" intervention, or someone could die. What happens if your browser experiences a JavaScript error blocking processing? And your alerts don't fire?
The alternative appeared to be that those alerts might not be noticed anyway because they might not have the staff to gown up and go into each room frequently enough.
> What happens if they fire too often and you get "alert fatigue" because they're not tuned correctly or in line with the other alerts available at the bedside/nursing station?
What happens if the device in the room fires too often?
> How much testing have you done to correctly assert that you're interpreting the HL7 or other specs correctly? And aren't misinterpreting data for some conditions or types of individual?
They seemed to find that it was accurate enough for the crisis* at hand.
> The "throw things together quickly" startup mentality might (I stress might!) Be okay where it's the difference between nothing at all and something that can save lives, in a country like Sri Lanka, during a global pandemic, fine.
Whelp, here comes a “not perfect but good enough to use part
> <further hand wringing on future concerns irrelevant to the situation under discussion>
I really hope they 1. Open source it 2. continue to work on this throughout the crisis and get it to a state where its actually suitable for critical care, and then 3. Work on achieving the relevant certification.
It sounds (just guessing) like the device vendor sells their own software separately, and is unwilling to budge on price during this time, forcing an already stretched hospital to look for new solutions.
Always consider the alternative. This could be a hospital in a remote part of a third-world country. Maybe they're understaffed. How are they currently handling the task of gathering information from monitoring devices and reacting to alarms?
Maybe, their nursing staff has to run from bed to bed to check patient's vital signs and device alarms. Emergencies would frequently be missed because they are understaffed and checking is irregular. Now, you could introduce software which provides centralized monitoring. If it's introduced on top of the existing activities (i.e., running from bed to bed), it leads to a net benefit - you catch emergencies earlier and consequences of malfunctions are less severe. But if it's introduced to replace the existing activities, it may lead to patient harm.
Sure, it's self-coded, browser-based and buggy - but you always need to weigh risks with benefits, and those depend on usage context.
Of course, in most western countries, this would be completely illegal. But these are also the countries in which medical software looks like it's from the 90s, with catastrophic usability and missing features.
We need to ask ourselves: Right now, we heavily prioritize patient safety over innovation - but have we got that balance right? What are patients missing out on if we could just bring a few more of the latest advances in technology to their bedside?
You know, not machine learning, the blockchain or the internet of things. Rather things like browser-based applications which "just work" and have great usability.
Note: I'm a physician, software developer and consultant for medical software certification :)
So, why not use those to build the "something is better than nothing" solution?
Just availability.
This was a quick and dirty hack to improve access to patient data done with what was on-hand, for a constrained deployment using specific known devices. They didn't have anyone with knowledge on using any of the tech you mentioned, some of which requires spending months setting up unless you have practical experience in delivering on the platforms. Just getting a more safety-minded setup for a MCU using free software can be a harrowing experience.
And they don't have the money to just contract it out or pay for the commercial grade stuff.
They did what they could with what they had, with explicit mention that it's not good on safety and security - but it brings some benefit now.
Here in Poland, a few weeks into lockdown, nobody asked for certifications on volunteer made PPE parts anymore. Because a shoddy PPE with no certification was still better than none.
It feels to me like the management has misunderstood the cost of the software vs not having the software. It feels like they're saying "this software is expensive, and doing nothing is free" when they should be saying "having all these healthcare professionals spending time putting on and taking off PPE the check patients is costing us this much per year".
As you probably know, an ICU will go through 30 sets of PPE per patient per day. That's a lot of time putting stuff on and taking it off.
I agree with you on LiveView as well. I believe it would prove to be more reliable than client-side JavaScript.
Java would be ok but Go is much simpler to get started with and to deploy. In general it is more lightweight and easier to use (unless you need generics, though Java doesn't really do those right either).
The most valid reason for using a language is simply because the team is comfortable with it.
Heh, you should try a language that comes with a repl and allows you to evaluate code directly from your editor in any context and get the results back. Then you'll have experienced "instant feedback". Until then, I can agree that Golang compiles faster than other languages, but it's nowhere near instant. In some other languages, you don't really care how fast it compiles as you only do it for deployments anyways (like Clojure).
https://github.com/motemen/gore
Also you kinda don't need a repl as much with go (coming from a python background and constantly repl-dev-ing), especially with an IDE. Everything is so clear and type-safe there is little divide between the architecture in my mind and the code's behavior.
I don't find that a REPL really gives me anything that I don't get from tests in Go. The only difference in practice is that I type the code into a text editor rather than into a REPL prompt (which is actually a better experience). On top of that, it's usually good to have that kind of code saved somewhere rather than existing ephemerally in a REPL session.
I don't know if it has a different word, but a repl for Clojure using a repl-workflow is very different from a repl for Golang with the normal "save -> compile -> run" workflow you have with Go.
For example, using a repl-workflow in Clojure doesn't mean you don't write code in your editor. I use vim + vim-fireplace to write my code, then highlight the parts I want to evaluate. So I get both in one. The repl is running in the background, and vim-fireplace communicates with it. Combine this with a SPA and I can have something like `(.alert js/window @username)`, highlight it and evaluate, then have an alert prompt in my browser window, which is next to my editor. Saves me a ton of time.
>I use vim + vim-fireplace to write my code, then highlight the parts I want to evaluate. So I get both in one.
Sure, and with Go in VSCode, I just click 'run' above the test I just wrote. It isn't any more difficult.
Statements like this makes it seem like while you probably have dabbled in Clojure and other lisps, you haven't used them substantially, as if you're using Clojure, you still make your changes in your favorite editor and code is still tracked in version control. It's not REPL _or_ code editor, it's code editor + repl backing.
Redefining functions on the fly is not a hack to lessen the pain, it's a defined workflow that makes you be able to work faster. If you're using it and it feels like something to lessen the pain, you're using it wrong.
> Sure, and with Go in VSCode, I just click 'run' above the test I just wrote. It isn't any more difficult.
This assumes a lot of things around the code that you're executing. First, you've already setup a test and it's already isolated enough. With a powerful repl + idioms, you don't need to do that, as you can execute parts of a function just to introspect that part, kind of like a debugger but works anywhere automatically.
In the end, I'm not gonna try to convince you to try something if you don't want to try it. But if you're anything like me, finding more productive ways of working is always interesting, even if they end up not being as productive as we first thought. With that in mind, since you seem to not have super much experience with a powerful repl, if you try it I'm sure you'll find what I say to be helping you be a better programmer.
Version control does not track which sequence of code snippets you have (re)-evaluated to get to your current repl state. I'm fully aware of how editor integration with a Lisp repl works in Emacs and (to a lesser extent) Vim.
>have dabbled...In the end, I'm not gonna try to convince you to try something...seem not to have super much experience with a powerful repl...etc.
As with most Lisp advocates, you seem unable to a accept that I have tried it and didn't like it. Please stop throwing shade.
> This assumes a lot of things around the code that you're executing. First, you've already setup a test and it's already isolated enough
A test in Go is just a function, so no set up is required. A test is more isolated than an expression evaluated in a repl.
It seems to me that you don't have much experience of writing tests in a modern development environment, and this is why manually recompiling parts of your codebase using Vim hotkeys strikes you as a good idea :)
> we worked on a custom layout which is similar to a bedside monitor interface.
It's great they went that route and did their best to stick to what doctors know by heart, instead of going creative.
Regardless of the parser, I agree. Writing your own is dangerous. HL7 is deceptively complex. It leaves a lot of room for vendors and your hospital to customize. If you write your own, you can very easily end up with a system that is boxed in to your hospital's own, or even your department's own, implementation.