Hacker News Reader

Comment by arkades | Hacker News Reader

Re Smart Bathroom:

Every time I hear about all-knowing toilets, I continue to feel it’s putting the cart way ahead of the horse.

There’s a rule in medicine: don’t order a test whose outcome doesn’t change your decision. That’s not about cost-savings, it’s about avoiding false positives that make no sense in clinical context but raise flags that must be chased down.

This sort of thing is literally promising to just continually run un-merited urinalysis left and right.

“More data” is not inherently good. Without a framework of information for interpretation of that data, it will do far more harm than good.

Honestly, the reason so little tech disruption keeps happening in the provision of medical services: no one trying to “disrupt” bothers asking what is or is not useful. There are just some ideas completely out of touch with what people need or want or have friction over, and they just refuse to die because they’re the most obvious ideas to have if you’re unfamiliar with the territory.

View on HN

> There’s a rule in medicine: don’t order a test whose outcome doesn’t change your decision. That’s not about cost-savings, it’s about avoiding false positives that make no sense in clinical context but raise flags that must be chased down.

There is also a rule in (system) operations: don't page unless immediate action (from humans) is required. This doesn't stop us from collecting data. If there is an anomaly, then we can look back at the data if necessary.

Alternatively, there ARE things you can alert on. My brother has type 1 diabetes. He got a free urinalysis in the form of ants crawling in the toilet. I expect – but since I don't have time series data from this toilet, I can't prove – that sensors would detect and increasing amount of glucose in the urine over time.

If anonymized, this sort of data would be useful for research as well.

I agree that giving any joe schmuck access to the raw data would result in unnecessary procedures.

There’s a legal context here. Once the data is available, physicians are obligated to chasenit down even if they think it’s unimportsnt. Ignoring data loses malpractice suits; unnecessary and ultimately harmful diagnostics prompted by data does not.

Not all medicine is practised within the legal framework that you labour under. If you are a doctor then you should be able to publish your real opinions here, within reason (prefaced with a short disclaimer if you deem it necessary, I would, along the lines of "I'm a MD, practising in <country> and this my personal opinion") and I, for one, would love to hear them.

I'm an MD, practicing in the US.

I don't know the actual legality. But here's what I do know from attempting to implement multiple projects within the US for clinical care and for clinical research: most providers, hospitals, and pharmaceutical companies we have worked with have balked at the ability to have high resolution data fed into their medical record or research data systems. In most cases, they are worried that if they had the data, they would have to act on it were something to be concerning. And they do not want that responsibility.

Example: we were in talks to feed in near-realtime blood glucose, activity, and blood pressure measurements from patients on a clinical trial. Surely that would be useful for better sussing out the effects of a research drug than the current process (patient comes in every few months to get measured or have their log data entered or meter data downloaded)? Nope, multiple top 5 pharma companies didn't want the responsibility and liability of seeing their data that often, because what if a patient had super low or super high glucose and they later died? You had the ability to act on it but didn't.

They'd rather just not have it at all. It sounds ridiculous. And again, I don't know the actual legal/liability issues, but based on my experience with malpractice cases, I could see a viable case to be made. And there are certainly institutions out there who believe the benefit is worth the cost of that responsibility for certain use cases. But my experiences have definitely made me reconsider some of these things.

All that said, I am definitely in favor of collecting more health data (somehow), because until we have more data we won't know its utility. So far, in general it has been shown that more testing and more information often doesn't lead to better outcomes, but the cost of data is getting lower -- aside from aforementioned liability cost/fear.

Sounds like a bit of a catch-22: you don't want to collect the data until you've collected enough data to know that you're not causing harm.

Not really. We have a system for this: clinical trials. I, as a lone doc, am not equipped to make heads or tails of un-randomized un-controlled piles of data. A clinical trial -is-.

That’s why we use them.

> I, as a lone doc, am not equipped to make heads or tails of un-randomized un-controlled piles of data.

Isn't that a description of literally what every practicing doctors, that isn't involved in research, actually does?

And clinical trials are extremely expensive. Surely there's some utility in gathering data between the extremes of doctor-patient and clinical trial.

No, it’s not.

We use properly conducted clinical science to give us a framework for interpreting the data we see, comparing data we have to studies with appropriate external validity to allow that comparison.

And no, there isn’t, if an RCT is at all feasible. There was a great paper a few years back that reviewed various case-control studies for conditions that were ultimately explored using RCTs (eg, knee arthroscopy, internal mammary artery ligation). What they found was that the retrospective study consensus was basically a coin-flip with respect to the RCT consensus. They just don’t provide useful data - too many confounders, known and unknown, go unaddressed.

This sounds like a really interesting paper -- do you happen to have a link to the reference?

I’m not ignoring your request; in trying to dig up the paper, I came across a couple papers with data claiming to show the opposite. I’m trying to digest it all before returning to this topic, so I can give a more useful response. And I haven’t found that paper yet.

> We use properly conducted clinical science to give us a framework for interpreting the data we see, comparing data we have to studies with appropriate external validity to allow that comparison.

I am extremely skeptical that this is in fact an even approximate description of how medicine actually works. Maybe not what you're doing! But what almost all of the doctors I've ever seen? They're not keeping up with all of clinical science. And even if they were, lots of clinical science is not being properly conducted. Just listen to or read the gripes that the statisticians have!

My previous claim still stands; you wrote:

> I, as a lone doc, am not equipped to make heads or tails of un-randomized un-controlled piles of data.

An individual patient, provides "un-randomized un-controlled piles of data" to their doctor – or, rather, not even piles of data. There's the visual evidence available from looking at and watching a patient when they're visiting your office (or whatever). There are sounds, smells, touch too. Then there's the verbal evidence. Then there's the results of whatever tests it is you order. If that's not un-randomized and un-controlled data, I don't know what is.

There's no framework for interpreting all possible medical evidence, even for the smallest professional specialties. There are no RCT for medical expert systems; none that anyone could legally use instead of a licensed doctor.

And there are lots and lots of interventions that doctors perform that have never been adequately studied in the form of RCTs. Doctors, in the real world, exercise an incredible degree of latitude with regard to what interventions they can perform based on their own personal judgement.

From the situation as you describe it, it'll likely require a government mandate or "Good Samaritan" type of law to resolve the issue since it derives from fear of litigation. Unfortunate.

I imagine getting something like this implemented at a state level in one or two key states could enable collection of large scale medical data for analysis on behalf of the public.

Edit: Question for you, given your experience do you think it might be possible to convince big pharmaceuticals to collect high fidelity data if said data was collected into wearable devices and only processed after the fact? As analogy to use a "black box" recorder approach rather than flight control systems. Though not sure how far to take that "black box" analogy.

@doctoring Do you think that those 5 pharma companies would be interested in the service that would take care of data collection and alerting patient/911? If it is possible I would like to talk to you about this and other ideas. Could you contact me at redant409[at}gmail.com?

Okay, I really don’t want the data. I don’t want the data because:

(1) if I see something, I have to act on it, despite not actually having the knowledge of whether it means anything in this information-rich context. I just love things that -look- like information but that don’t actually tell me anything.

(2) we have a LOT of data suggesting that we do more harm than good when we try to intervene or investigate everything. For Pete’s sake, hospice patients outlive those who are actually pursuing life extension! Turns out every treatment has harms, and when you’re applying those harms to no benefit (because the patient is asymptomatic, or the anomalous event was never destined to progress to clinical disease), you just accrue harm that -looks- like care.

3) It is not likely to result in useful diagnosis, because catching shit pre-clinically offers little indication of what will or won’t become clinical. We see this in helical CT: a lot of people became “cancer patients” when helical CT came out, before we realized a LOT of growths recede forever without intervention - we just hadn’t seen them before, on the less sensitive tech.

If I see the data I have to act on it to cover my (American) ass, and I know I will do net harm to my patients. Thanks but no thanks. Do your due diligence: collect the data, analyze it, stick it in a properly randomized prospective study to see if you just trawled your way into an overfitted model, and prove that the intervention for it does more good than harm. THEN I’ll be delighted to use that specific data to pursue that specific intervention. Don’t just measure everything, throw it at my wall like hot spaghetti, and expect me to turn it into dinner.

-my blunt anonymous opinion, since it was asked for

You've outlined a cogent rationale for refusing all medical treatment! And there is a lot of evidence that this is in fact not a terrible idea. A lot of medicine is, not just not-beneficial, but actively harmful.

So as someone that would be liable to act on this data, I understand and sympathize that receiving this data would be an unfair burden on you. But what about the rest of us? We're being treated by people that we are expected to trust that actively refuse to learn whether they're actually helping. That seems pretty unfair to the rest of us.

No, I haven’t. I’ve outlined a rationale for helping when there’s reasonable evidence that the benefit provided will outweigh the harm caused. Eg, antibiotics for the average upper respiratory infection vs antibiotics for sepsis. In the context of diagnostics: if the prior probability of disease is such that the outcome of the test doesn’t meaningfully change the likelihood, it’s bad for the patient to create a situation where for legal purposes I must now act as though the probability is much higher than it is.

And no, we aren’t actively refusing to learn if we are helping. We want the data collected in a meaningfully structured way that actually answers the question of whether it’s helping. Garbage In/Garbage Out doesn’t answer that question.

You say it’s unfair, but go reread my post. My concern is that my arm gets twisted into doing things that are -bad for you-, not bad for -me-. Whether you like the reasoning or it’s conclusion, nothing in it is about being unfair to you. It’s about trying to do my absolute best to live up to my ethical responsibility to optimize your health outcomes. The only part of it that’s defensive is the part where I don’t ignore test results, because it will get me sued out of medical practice in the rare occasion they’re true positives. And I would like for that not to be the case, but there we are.

> if the prior probability of disease is such that the outcome of the test doesn’t meaningfully change the likelihood, it’s bad for the patient to create a situation where for legal purposes I must now act as though the probability is much higher than it is.

I don't disagree. I was claiming that both patients and doctors are caught in this trap, i.e. doctors are legally liable for not offering, or recommending, treatment that isn't warranted.

Not sure about the systems you maintain, but mine page for false positives all the time. They're annoying, sometimes they wake me up, but in general they're not a big deal.

A false positive from a medical test is much more costly, both in money for a doctor (or doctor's staff member's) time, and anxiety for the patient.

That said, your brother's case makes some sense. It's targeted, and I imagine the action you want to take after getting an alert would be pretty well defined.

> “More data” is not inherently good. Without a framework of information for interpretation of that data, it will do far more harm than good.

I'm glad people are saying this. I'm in a constant fight at work with people who want "more data." What ends up happening most of the time is it's ignored or people do not interpret it correctly because they don't have the ability or context to do it (doing stats wrong for instance).

Have them read Chapter 5 of this book. https://www.cia.gov/library/center-for-the-study-of-intellig...

"Using experts in a variety of fields as test subjects, experimental psychologists have examined the relationship between the amount of information available to the experts, the accuracy of judgments they make based on this information, and the experts’ confidence in the accuracy of these judgments."

"Key findings from this research are: • Once an experienced analyst has the minimum information necessary to make an informed judgment, obtaining additional information generally does not improve the accuracy of his or her estimates. Additional information does, however, lead the analyst to become more confident in the judgment, to the point of overconfidence. • Experienced analysts have an imperfect understanding of what information they actually use in making judgments. They are unaware of the extent to which their judgments are determined by a few dominant factors, rather than by the systematic integration of all available information. Analysts actually use much less of the available information than they think they do."

But, the main issue is "once an experienced analyst has the minimum information necessary to make an informed judgement" is followed by "they are unaware of the extent to which their judgements are determined by a few dominant factors."

Which implies they do not know which part of the data is most important. No?

disclaimer: didn't click link/read chapter

The authors ultimate solution is to use the scientific method to compare competing hypothesis...otherwise you'll be blinded by all sorts of biases. If you are running an experiment and using competing hypothesis...in theory, you can identify which metrics are best for that test... but if you had much more data, you'll probably end up using every data point past the first 5 as confirmation bias. (I'm no expert though and haven't read the book in about 3 years.)

Depends on the way of consuming that data. I agree it may be counter-productive to feed more data to humans, who are able to hold on only to 7-9 object in immediate memory at a time, and driven by subconscious reflexes hardwired into us by billions of evolution.

AI is not going to "think" like humans. It is not going to be better or worse, just "different". And all experience so far show that the more data the better are the decisions made by AI.

Cart, horse.

People are proposing smart whatever now, and handeaving its value proposition as -eventually- feeding into ML which will -eventually- have benefits to patients.

I don’t think there is disagreement that data + ML is useful, but the value for increasing data because of AI assistance isn’t meaningful in the absence of that AI. Hospitals and insurers have huge data sets available: build a halfway decent AI first, and then argue for the value-add of the data.

Those data sets are not public, hipaa and other issues make it difficult to build any sort of AI for this.

That is technically correct, the best kind of correct!

Except any department in any hospital or insurer that welcomes you to do research can make that de-identified data available to you with a simple contract.

How do you think college freshmen get in the door to work on research with physicians?

No, it’s not public, but it’s trivially accessible.

It's not trivially accessible. It's not impossible, but far from trivial. Startups like Tempus are doing it but I doubt it's trivial for just any startup to get that data

If I hadn’t worked in precisely that field and seen how easily we gave out data sharing agreements to partners, ...

But I did. No, I use the word “trivial” knowingly.

To partners, sure. What sort of partners ? If I start a company and show up at the door, would they just give me the datasets ? Would those data sets be in useful formats ? if you are seriously telling me those data sets are easy to get, let's talk because I know what to do with them.

There’s usually a spot in every insurance company that considers itself the “innovation center.” Basically, if you incur no costs to them beyond getting you the data, and give them a free trial run on your product, you’ll get your data-sharing agreement.

The data for insurers is in the form of Billings (which come with Diagnosis codes!), which follows a federally-standardized format (check out the Medicare website) and you can get a sample of 5% of Medicare subscribers’ data off the Medicare site without any hoops to jump through.

I want to say, “if you actually know what to do with it, PM me and we can talk details,” but the fact that you don’t already know about this sort of low-hanging fruit suggests you’re from outside the hc sector looking in.

I am from outside the hc sector, but I know how to build models on large data sets. So if you seriously know how to get the data, I seriously know what to do with it.

j s at self driving medicine dot com

Reach out, we can talk about what you’re thinking.

And yet, anybody who actually works in machine learning has repeatedly said "figure out what problem you're solving first, then determine what data you might need - don't just throw a classifier at a data garbage heap"

And the experience doesn't show more data is better, it shows that an excess of features leads to overfitting.

Sure, having more features gives a model more opportunities to overfit. But having more data points has the opposite effect, since they reflect the underlying distribution better and therefore provide a better estimate of actual model performance.

Yup. More data is not the same as better information.

Put another way, at the extreme, you can have more and more data but unless you have the right data and are able to interpret it correctly, more data will only blind and confuse.

There is (pretty horrible) joke: "If violence does not solve the problem, you are not using enough of it". It may be the case with data. Put another way, we may be in the uncanny valley of data.

There is definitely an issue of under-diagnosis for many diseases. More data may not help initially but once we are over the critical mass in the amount of data and sophistication of ML algorithms trained on it, it will get over the hump.

We have a much bigger problem with overdiagnosis than underdiagnosis, and the latter is not likely to be solved with this sort of scattershot data collection. The former is very likely to be inflamed.

The difference is that you actually hear about the Diagnosis X That Was Missed, but you never hear about That Simple Blood Test That Should Never Have Been Drawn but that Showed Positive for a Marker Common In Disease But Also Found in the General Population so They Needed a Biopsy to Rule Out, and Spent Hundreds or Thousands if Dollars and Lost Several Days of Work For Nothing At All. The latter is much, much, more common.

The “miss” is usually due to missing something in the modern deluge of data coupled to inadequate appointment time/resources, not lack of data availability.

For a real life example: heard about a patient die the other day. Reviewing his records with a student afterwards I saw his potassium had been climbing in his last two panels (though it hadn’t yet gotten high enough to be a red flag), and he didn’t get the blood gas that should have been done as per protocol. Thing is, the data was there, the rules were there, but someone with enough time to meaningfully review it was not. Would more piles of detritus data to sift through made that more likely, or less?

I also remain an ML skeptic for the time being. ML analysis of patient data is a big sophisticated version of retrospective analysis, which for purely statistical reasons is highly, highly unreliable.

Not sure how this proves me wrong. What it does prove is that analysis of data should not be left (entirely) to humans, who are inherently flaky. Dark irony of this situation is that _more_ data from additional test had a chance of saving the patient.

The way it should have worked: based on previous labeled examples, computer system should have noticed the uptrend in potassium and flag this client for blood gas, and then make classification based on available and _additional_ data.

It’s not the blood gas that would have saved him; just seeing the potassium trending up would have done it. Thing is, there was a context where that mattered. In others, it wouldn’t have. That’s why you need someone capable of interpretation to -look- at it. EMRs have done more to harm patients than help them, now that everyone constantly copy-pastes old patient hx before adding their notes (so nothing can be said to have been missed), people are constantly shifting through piles of be and missing the pertinent information.

To rephrase: algos intelligent enough to be useful in the manner you imagine are a long way off from existing, but data deluge that harms patient care is -already here-. Don’t worsen the deluge until you improve the ability to digest and present it.

That EMR “decision aids” aren’t anywhere near sophisticated enough to tell the difference in the myriad of situations we have is why docs fight so hard against those automated “flags.” We get piles of them, all the time. QI people call it alarm fatigue, they know it’s a thing, and yet... they do it anyway, because it sounds (like in your post) like the obvious thing to flag.

It’s not that AI is inherently incapable of getting there. It’s that when it does is a long way off. Medicine looks a lot simpler from the outside.

I agree with you and are suspicious of the power of increasing raw data in the near term. I think there are a few dumb things we can do that have a chance at being useful. The usefulness of something like this would have to be extremely narrowly focused. For example, it can look at only potassium trends. We would first need to just monitor people for long periods of time and see what happens. We might see certain combinations of simple markers that make additional tests worth the risk and minimize false alarms. After this research, would we turn on the alarms and start worrying people.

Thanks for the insight. I wonder though about the difference between scattershot data collected in a hospital setting, vs personalized data collected over the normal, healthy course of a person's life. Establishing relevant baselines would go a long way towards eliminating noise and flagging real outliers.

I have nothing but agreement with all your points.

I just finished (the book) Everybody Lies. It seems to me using the doppelganger approach (as mentioned in the book) would at least be helpful.

A couple months ago I spent a good number of hours over a couple week at local hospital waiting for a close friend's mum to recover from a stroke. I could be mistaken but the only "data analysis" I noticed was taking place in the heads of the doctors and nurses.

The same goes for the physical therapy facility. That is, no one - not even the insurance company - was using similar cases as a guide. Yes, at a high general level they were. But using real data didn't seem to be on their radar.

This void made family decision making stressful and difficult.

That is correct: the analysis of large data sets is opaque to patients, and application of the evidence base to specific patients happens in the heads of physicians.

Patients are not part of the decision making process, and therefore not privy to the analysis, unless there is a decision to be made that requires their input (eg, two reasonable roads forward that depend on their preferences, or on their risk-tolerance.)

The process is made opaque to patients intentionally.

Yes. But at no point did I sense the options being offered were based on doppelganger type of data. Or even (the doctor saying) "here's the data...here's our interpretation...here are the recommendations and pros and cons..."

Nothing like that. No continuity between docs or floors/wards. Just everyone kinda making it up as they go.

It just struck me as another reason why outcomes aren't optimized, and costs, relatively, inflating.

I'm not suggesting doctors are replaceable. But the doppelganger approach to (big) data seems to be ideal.

> "here's the data...here's our interpretation...here are the recommendations and pros and cons..."

I understand. That’s part of what is intentionally opaque. We don’t speak like that to anyone but other medical professionals. There’s a lot of Dunning-Kruger in what people think of their ability to parse hc information. As someone who grew up with serious chronic disease, I’ve been on both sides of this fence. It’s not something you really “get” until you’ve had to care for patients.

> costs inflating

Several Health Econ studies have shown that about 70% of hc costs are driven by new technologies (implants, procedures, patented meds.) It has nothing to do with how docs communicate with one another.

And how we communicate is also opaque to patients. Not intentionally, it’s just not a conversation for patients, any more than any two technicians in a software company talking/arguing technical details are having a convo meant for the ears of customers.

I think it is good to remain skeptical about medical ML, but I don't think it should be because of the algorithms or that retrospective analysis is flawed. In most retrospective medical research, the data sizes are relatively small. Likewise current ML needs huge amounts of labeled data to be accurate, and patient data just isn't available to ML experts in the volumes necessary. The question I have is would dramatically increasing the size of available patient data improve ML analysis? Is it the case that some of the flaws of retrospective analysis are actually a result of too little data especially in the context of ML? I don't know the answer, but I do know we need to fix the data availability problem if we want to find out. How do we fix the data availability problem? Not sure either, it a complex problem.

> That’s not about cost-savings, it’s about avoiding false positives that make no sense in clinical context but raise flags that must be chased down.

Unfortunately the rule doesn't apply in this case. Your "smart" toilet is not trying to diagnose problems. It is trying to infer which ads you will click on the most, to make money for a surveillance company. A toilet that has a handful of true positives, a huge number of false ones, and doesn't kill too many people is pure win: surveillance and pharma both make money, and it's very hard for a consumer to tell if he never had X disease, or if buying the medicine prevented his X disease.

Moreover, the whole 'smart bathroom' thing has already been pretty much solved in Japan - and more or less keeping in mind the principle that you describe. You can get your temperature, get a warm seat, turn on some ambient music to mask your own, more personal music, flush, bidet, whatever all in most toilets in your average department store.

You can get more complex offerings that do perform some more complicated tests if you really want to.

The 'bath room' itself is also typically a single, self contained unit, often in some kind of mold, and including the shower, accoutrements, and bath in a full space designed to get wet and save/share the clean water in the bath.

The 'bath room' usually has a waterproof, electronic control panel which allows you to set timers for filling the bath, set an ambient temperature to keep it at, set the preferred level of water, set different default temperatures for the shower and bath, and typically also access all these features from other places like the kitchen where a companion panel is installed. This is typical of your average house - I just described my wife's parents' place, which was built 10 years ago.

The whole design thing is completely solved there, and why that approach has not been adopted across the U.S. and Europe by now is completely beyond me.

But the killer thing there would be to bring the Japanese approach to the US and Europe and plug in some better smart access features. Or go to Japan and plug them in there, where the infrastructure and demand is already in place [well, Google is probably on it already but...]

>“More data” is not inherently good. Without a framework of information for interpretation of that data, it will do far more harm than good.

There's an episode of Star Trek: Voyager about this[1]. Seven of Nine absorbs months worth of data (way more than a human could actually process) from the ship's computer, and "discovers" that both half of the ship plans on mutinying and that the Captain wasn't actually trying to get them home. Both of which were false.

1. http://memory-alpha.wikia.com/wiki/The_Voyager_Conspiracy_(e...

However she does also make the useful and correct deduction that photonic fleas have infested the plasma conduits.

Why would the data only be used for patology?

* Diet balancig? (Hey, eat more cucumbers)

* Calories? (IANAD)

* Also more serendipiteous opportunities we can't foresee now

Or:

(at the doctor's)

t = 0 - Doc - I would need a urine analysis

t = 60s - User - Here you are.

Cons:

* Huge privacy concerns (Hey, you are fired! btw, you also have cancer)

It's possible that a) diet isn't that hard to figure out and b) gamifying, quantifying, testing, and over-extrapolation from isolated scientific results make people confused and much less effective at making the right decisions consistently, which is the part they really need help with.

Generally speaking the topic falls into the desire to have real-time/always-on data feeds that magically produce value just because they’re always producing data. That said, beyond asking the obvious, why do you need the data real-time and how much value does this produce, I do not know a better filter or means to guide a startup covering this ground.

There is no easy guide. Speak to the people doing the work, and target your innovation to the existing revenue streams (that is, if your product benefits an insurer, why are you pitching it to the patients?).

As someone in the industry I do of course have a wishlist, but, you know.

As to clean: a self cleaning toilet and a roomba living in the corner is a huge step away from treating your toilet as an always-on medical lab.

It's also literally an Onion article:

https://www.theonion.com/new-e-toilet-to-revolutionize-onlin...

I'm guessing the big innovation here would be developing useful diagnostics that skew towards high specificity at the expense of lower sensitivity. Additionally, If you're collecting a bunch of datapoints much more frequently than current screening/testing regimes, You would need some smarter averaging to find trends and weed out many, if not all false positives.

I think Reliable, cheap, user friendly medical diagnostics would be a game changer, but of course the data analytics to make good use would be different than how current, one-off, medical testing is treated.

What if ypu use that data to train AI? How many lives saved or serious health issues avoided by early detection?

The applicability of that rule is a symptom of healthcare's overreliance on human judgement. To a rational decision-maker, more data is better. When the problem calls for an expert system, it's a problem well within the domain where algorithms outperform humans.

> don’t order a test whose outcome doesn’t change your decision

> raise flags that must be chased down

If a red flag must be chased down, doesn't that mean it changes your decision?

This reminded me of the urinal in "Benchwarmers"

Garbage in, garbage out.

Reply on news.ycombinator.com