Hacker News Reader

Comment by skybrian | Hacker News Reader

You can't test your way to safety. Here's a thread from a real expert:

"All a sample can tell you is that one sample, from that place, on that day, was clean."

"A lot of water contamination events are very temporary: a rainstorm washes manure into a well or canal, and then it's gone within a day or two. Testing only catches that if you target your testing program for it. Guarantee you most farms aren't doing that; they test on schedule."

https://mobile.twitter.com/SarahTaber_bww/status/10670960981...

View on HN

> You can't test your way to safety.

> "Testing only catches that if you target your testing program for it."

Seems like a contradiction to me.

That said, just because testing alone isn't good enough doesn't mean it's better than nothing.

The political argument always comes in the form of "regulation bad!" vs "safety good!" without much attention to the actual policies. Thank you for bringing our attention to some of the problems with the proposed regulations, but the real issue is that we're politicizing something that we should clearly do _something_ about.

Very interesting points in this thread:

> Testing your irrigation water won't keep the feedlot next door from blowing dried-up cow shit dust all over your lettuce. (Or, you know, into your irrigation water on a day when you're not testing). [1]

> It won't keep harvest crew co's from putting so much time pressure on crews that they can't clean their harvest rigs properly between shifts. [2]

> And it sure won't solve our traceability problem. There's no way in hell we're having this much trouble tracing the lettuce to its source, unless there's massive supply chain fraud going on. We should be worried about THAT. [3]

This one reminds me of a situation I ran into once:

> (You also have to ask, WHY would a farm or any other business target their testing towards when they're likely to get red flags? They don't want that. Any time you do testing, there are lots of ways to game it to lean towards clean results. Which is what farms & companies want.) [4]

We were installing a drinking water monitoring system at a rural retirement home (on well water), probably a couple hundred residents.

Up to that point, one of the tests they did was a manual daily chlorine level check. Essentially you need to have a certain amount of residual chlorine in the water at the point of use, which lets you know you you are putting enough in (if you put too little in, it gets used up as it's breaking down organics in the water, and you end up with a zero). If the level is too low, you have to report it and explain what happened and how you fixed it, and if it happens too much, inspectors will start looking closely at your practices and forcing fixes and basically nobody likes that.

The new system had continuous monitoring of several parameters, one of them being chlorine, and would record everything as well as send alerts to people if anything dangerous happened. The chlorine monitor was installed at the opposite end of the building from the water supply, trying be representative of the worst case 'point of use' level.

As soon as it was online, it started sending alerts every night in the middle of the night. The free chlorine level would drop starting late in the evening (as the usage dropped), and then somewhere between 2 and 5 am would drop below the alarm threshold, and around 6-7 am jump back to normal. The owner was pissed that we had screwed the system up, because they didn't have this problem before the monitoring system was in.

Of course what was happening was not new, it was just that as usage dropped off as the residents went to bed, the water in the pipes just sat stagnant, and residual chlorine was slowly used up to the point there was none left. The daily checks were usually done mid-morning, after there was a lot of use as everyone woke up, and so all the water in the lines was freshly chlorinated when the check was done.

For years, anyone getting water in the middle of the night was drinking technically unsafe water (it still had been treated and likely posed no real health risk, so long as there was no contamination in any of the pipes or fixtures), and all that happened was the situation was exposed.

We eventually added a solenoid to the end of the line, and programmed it so it would open and dump water when the chlorine level got low but before the alarm threshold, at least getting fresh water into the main distribution lines of the building and avoiding the alarm condition.

It always stuck with me that the owners reaction was not "Oh, thank you for finding and fixing a problem I didn't even know about before it got someone really sick" but more like "your stupid system made things WORSE, we should rip it out and go back to how it was before."

[1] https://mobile.twitter.com/SarahTaber_bww/status/10670961003...

[2] https://mobile.twitter.com/SarahTaber_bww/status/10670961010...

[3] https://mobile.twitter.com/SarahTaber_bww/status/10670961017...

[4] https://mobile.twitter.com/SarahTaber_bww/status/10670961060...

Thank you for sharing your story. I hope you don't mind if I turn it into a thought experiment, because I find it a good example of how tricky finding and fixing a problem can be. I know nothing about water quality standards, so they might be naive, but the following questions stick out to me:

* Is there a reasonable contamination hazard from water sitting in pipes that are flushed daily with chlorinated water? In other words, is it vital that there always be residual chlorine 100% of the time, or were the standards designed to accomodate this sort of periodic lapse?

* If the dips were harmless, would the inspectors be willing to accept that?

* Was the dump solenoid effective at flushing the entire network of pipes, or just the branch containing the sensor?

Depending on the answers, the end result could range all the way from "the new system let us eliminate a serious hazard", through "we were probably fine before but now we can be sure at a little extra cost", down to "now we have to waste water to avoid tripping a sensor so we don't get fined, with no actual improvement to the water quality". It's a great example of how what we want, what we test, and what we enforce can get just a little out of alignment, and make a hinderance out of what ought to be a definite improvement. Thanks again for sharing.

Caveat: I come from the monitoring and control system side, so I've only learned about the actual water treatment part as a side effect from working on monitoring and control systems for it.

My understanding is there is very little actual hazard here, provided the pipes are in reasonable shape and there isn't a source of contamination.

One example of a source of contamination: dead ends in plumbing (eg, an old branch that's been capped off). The stagnant water can grow bacterial colonies which can then contaminate everything downstream from where the dead end branches off. The residual chlorine can fight this but it's of course better to not have the dead end at all.

The problem with the dips is it's not possible to distinguish between the "expected" nightly dips and real problems.

For example: do you just ignore all alarms between 3am and 6am? or should that be 7am? Is there another check (+alarm) to be sure the clock is correct (and do you now need a secondary clock source for that) and that DST is respected?

Or do you build something very complex that checks against recent flow rates before raising the alarm -- in which case, how do you test that code and ensure it never breaks (keeping in mind it's safety-related and it's very hard to unit test real-world flow meters, chlorine sensors etc). This gets difficult because you might have different plumbers / maintenance people doing things (adding branches, fixing leaks, etc) that might change the physical layout and not even realize there is control software that might be affected by changes. You can add more sensors to try to check for some of these potential conditions but each sensor costs even more money and adds more complexity.

A lot of this is really "CYA". If something ever did happen and a resident got sick (or worse) from the water, and it came to light that not only were there alarms every night but they were specifically suppressed, even though that might be a rational decision given the facts, at best, that decision would still be faced with a lot of scrutiny and at worse it could be considered criminally negligent.

It's basically not possible to flush the ENTIRE building, because you'd have to open every fixture (every sink, shower, and flush every toilet). In this case, the solenoid dumped water from what was effectively the main line through the building which everything branched from, so from any apartment running the water for maybe a minute would get you fresh water from that main line.

So I'd rank this in the middle of what you said: "quality was probably (usually) fine before, definitely more likely to be fine now, and if an alarm goes off at any time it's real".

Is there an engineering solution for this, or is it something beyond that?

For instance, could we have almost everything that pushes water to plants contain some kind of low-cost sensor that sends data about the water going through it back to some base station for analysis?

We would need to spend some money to: * develop the technology * deploy the technology, including ensuring either the sensor can be self-powered (solar?) or we can hook up electricity to the water outputs easily (and electricity + water = not an easy solution!) * monitor the data

However... all of this seems do-able if we want to reduce health problems like this.

I imagine, like most everything, it comes down to tradeoffs: how frequently do outbreaks like these happen? What is the monetary cost when they happen? Would constant monitoring tech like this actually help us detect it? Who would pay?

I'd argue this is exactly what something like the FDA should be helping with - ensuring our food is safe, and that means helping deploy state-of-the-art sensors to reduce the chance of contamination. With government money going to some group that can make this a reality, the risk of developing this on their own and hoping that food producers would choose to pay for it goes away.

I don't know but I'd recommend reading Sarah Taber's Twitter feed to get a better feel for what's likely to work.

Link to thread: https://twitter.com/SarahTaber_bww/status/106709609717109965...

>gone within a day or two

Shouldn't people on hackernews be familiar with "continuous testing"? There is no reason to believe non-tech can't modernize their test infra too.

It's not a matter of whether they can, but of why they should. Without direct regulation, there isn't much incentive to engage in continuous testing.

Reply on news.ycombinator.com