AI tools have been built to catch Covid, but none of them helped
technologyreview.com
technologyreview.com
> Driggs’s group trained its own model using a data set that contained a mix of scans taken when patients were lying down and standing up. Because patients scanned while lying down were more likely to be seriously ill, the AI learned wrongly to predict serious covid risk from a person’s position.
> Errors like these seem obvious in hindsight. They can also be fixed by adjusting the models, if researchers are aware of them.
I wouldn't trust any method of fixing errors in data collection. You need to redo data collection and perform a proper case-control study. You need to select your cases and controls such that the demographics look similar.
In general, your goal is not to pick cases out against a background of the general population. It's to pick cases out from an at risk group.
And if there's any difference in procedure used for data collection between cases and controls, your data is compromised. Researchers often suggest correcting the dataset (cropping out artifacts etc.). In my opinion this doesn't really work... artifacts (for example in image data) can often cause subtile global differences in illumination which ML will pick up on but not be obvious to someone inspecting the images.
The only solution when you have such artifacts (and only one that would be acceptable to me if I was doing DD, or evaluating the research) is to redo data collection in the context of a more reasonable case-control study.
A client of mine wanted me to look into this dataset and help creating a model for detecting COVID. One of the doctors they where working with wanted to be able to submit pictures they took (from their point and shoot camera) of the CT scan. Good luck putting that variation in your dataset.
If all other parameters area randomized between case and control, this is also fine. I'd guess you can also add illumination and other artifacts to the datasets to make the training robust to this.
But ultimately in the extreme case (like wanting to submit images from a point and shoot camera) I suspect you'll have a hard time building a system that's robust to that... or robust enough to be used in a diagnostic context...
I personally don't think I'd be comfortable working without a standard configuration used by the radiologists and a controlled protocol.
Climate research would like to have a word with you.
Also I was mostly talking about correcting errors in existing data..
Sounds about right.
- 100s of models were done incorrectly, and none even remotely correctly
- there is no phenomenon to correctly model
Both seem extremely unlikely. Not sure what the alternatives are.
- Technology not suitable or adequate for this use case.
I mean, we've been to the "AI over-promises and under-delivers" rodeo before.
Yeah. Attempts at powered flight had a 100% failure rate in the 1800s.
It's kind of absurd to think that "AI" (especially in its current incarnation) must be able to solve any "hot" problem that it's thrown at.
And those older systems has more intelligence in them. Todays’s “AI” is a small number of tricks aimed at large amounts of data, with an unprecedented and enormous amount of marketing added.
If you're talking about deep neural networks, I can understand this viewpoint. But generally the last decade has proven widely successful for high quality vision recognition models that just weren't available before then. And with something like transfer learning, you can take a powerful off-the-shelf model and specialize it without a huge dataset.
The real barriers IME are around legal and privacy implications. There's also a strong argument about if these models creating enough value in the first place, but they can work on a technical level.
More or less. It's almost certain if "AI" makes any headway at all, it will fall far short of the hype.
> That seems like a fairly fringe view. Especially on a problem that has well defined data like medical imaging.
I doubt it.
Also on the wet side, there is very little progress on something like general antiviral drugs.
Some stuff is just really hard.
You can make all the models about phlogiston you want, none of them are going to accurately explain fire.
Another problem you have is when you do it right the results look worse than a biased dataset. So there's some incentive to continue working with a biased dataset.
COVID exists, it causes physical changes and affects the world outside the host. 100% it can be modelled somehow in theory.
That being said, there are harsh rules against collecting data (for example, it would be extremely illegal to create a sample set by taking a random group of people and infecting half of them with COVID on purpose). So I expect actually getting training data is a huge hill to climb. If most of the models used the same training data I assume there is a reason like that.
So in practice, I don't expect a model quickly.
The entire premise of modelling something rests on the fact that the behaviour of what you're trying to model in the future depends on its behaviour in the past.
This is not true for all systems, it's not even true for most systems. John Kay differentiates between 'resolvable uncertainty' and 'radical uncertainty'. Modelling a volcano eruption is resolvable because it's a reasonably predictable system and you can make confident stochastic claims. Complex human systems are chaotic, have 'unknown unknowns', are simply subject to chance or non-linear behaviour that completely throws any prediction off.
You cannot model something that is subject to forces you cannot even know at the point of making the model, and this is true for any complex human system pretty much. Modelling Covid as if we have access to Hari Seldon's psycho-history was always a terrible idea.
no, the correct behavior under uncertainty isn't modelling, it's something akin to Taleb's notion of antifragility or robustness. The correct response to a pandemic isn't some sort of Hari Seldon psycho-history which, as the article points out, is futile. The right response is building systems so responsive they can crush a pandemic before it gets to that stage.
For that we need better public health measures, those depend on modeling, vaccine development, etc.
There's no escaping understanding human health. It underpins and informs our responses. (Which kind of viruses are likely to pose a real threat, monitoring those threats, evaluating them, models, predictions, data, etc.)
I'm not talking about one big Seldon-esque equation, but about the public/global health system we already have, the whole of medical research, and so on.
These toy classifiers trained on COVID scans are in the same class as Tesla's dangerous "autopilot", because the system is too limited, it doesn't understand context, it lacks a world model. (Tesla's lack epistemic convergence and stability for object detection, ie. it lacks the understanding that cars, roads, obstacles don't flicker in and out of existence, nor do they change from frame to frame. Similarly a "medical AI" needs to understand that medical imaging artifacts are unlikely to be good proxies for real diagnosis, so it needs to be able to abstract away from the image, process away the artifacts and recording environment differences, etc.)
People with a background in methods like solving problems that are easy to solve with the methods they know. Those are rarely the problems people in biology/medicine want to solve. Some common pitfalls include:
* Asking the right questions.
* Framing the questions correctly.
* Finding computational problems that capture the essence of the questions and that can be solved efficiently.
* Interpreting the results.
* Avoiding systemic biases in rare but important edge cases.
Finding the right problems to solve takes time, and the first attempts will probably fail.
First hundreds of attempts? In your experience, is that a reasonable number?
If 10 people hike to Mt Everest's base camp, it's not the equivalent of one person climbing ~10 times further to the peak.
Given the fact that at that time there was still quite a lot of "if we all help we can beat the pandemic" positiveness going around. An easily run-able piece of small code that promised "results" was available and the fact that a lot of organizations wanted positive PR, you get these insultingly low quality "research" practices.
I looked at a lot of the initial papers and models and most were forks of COVID-Net or used an already established model like ImageNet and retrained it with the bad dataset. Presto! Paper out PR happy. Very few questioned the dataset or the approach in general. But they loved bragging about it on LinkedIn.
100s of wrong models doesn't sound extremely unlikely to me. It's really easy to train a model and test that it works on some dataset and then find it doesn't work on a different one. Setting up the inputs for training is difficult because of data privacy, there's intense pressure to publish, models have no expectation of explanability, lots of reasons for a wrong model that looks promising.
I've worked adjacent to people trying to solve problems with machine learning and it's rough going. When things work, ok nice, but when they don't, it's much harder to tweak than something based on heuristics (which, of course, don't always apply either).
It's selfishness and narcissism masquerading as philanthropy. Instead of doing an actual useful task, they want credit for the exciting/pioneering front line work.
It was the same thing with the ventilator shortage. How many people publicly declared they were "inventing" a new, easy to make ventilator? Design was never the problem, manufacturing was. But few were interested in manufacturing extant (and already FDA approved) designs.
With the right data and the right application, this technology can definitely be helpful.
However, it is very hard to sort out signal from noise since any reasonably competent undergrad can throw a convnet at some images and get an AUC.
Actually, I have even seen a high school science fair where they did a medical computer vision application.
I think this speaks more to the accessibility and ease of use of the technology that is out there than it does to the inherent limitations of the tools we have.
I am reasonably confident that there is a group out there that was more careful and has something that might have been useful, but it’s just impossible for clinicians to know who to trust.
I feel that both uncritical, un-nuanced hype (Radiologists will be replaced in 5 years!) and uncritical, un-nuanced closed mindedness (AI will never be clinically useful) is harmful for tackling the biggest problem the field faces right now: trust.
Who to trust. When to trust.
What is that confidence based on? It seems like the researchers here made a reasonably thorough effort to find "something that might have been useful" and only turned up two that were even worth further investigation:
"She and her colleagues have looked at 232 algorithms for diagnosing patients or predicting how sick those with the disease might get. They found that none of them were fit for clinical use. Just two have been singled out as being promising enough for future testing."
So are you reasonably confident that these researchers just didn't look hard enough to find the good stuff?
These processes take a lot of time and it’s not surprising that a year and a half after COVID-19 became widespread in the US, we still don’t have a fully validated model.
Problem is: how do you get enough buy in from clinicians to run a clinical trial with YOUR model, when there are hundreds of crappy models out there? It takes a lot of time to run these studies for this reason to start with and then depending on the size of the clinical trial, it takes time for the trial to run it’s course.
I just bought the COVID-19 NFT and burned it. Pandemic over.
If this algorithm was better it would absolutely save lives and make everyday life easier for clinicians.
Anyone who does understand the underlying mechanisms are probably still trying to clean the data sets.
https://news.mit.edu/2020/covid-19-cough-cellphone-detection...
“Across around 2,500 captured cough recordings of people confirmed to have COVID-19, the AI correctly identified 97.1 percent of them – and 100 percent of the asymptomatic cases.”
https://www.sciencealert.com/ai-cough-analysis-could-detect-...
I think it may be the case with AI researchers where if the only tool you have is a hammer then every problem starts looking like a nail /jk
It has some S shaped function to trigger
"Anti-intellectualism has been a constant thread winding its way through our political and cultural life, nurtured by the false notion that democracy means that 'my ignorance is just as good as your knowledge."
see also: the leaks to project veritas about Facebook's counter vaccine hesitancy program.
I think some on the left would refuse it because Trump.
I think some on the right would accept it because Trump.
I think some would still be hesitant because their concerns with the vaccines are not political.
The US FDA was among the first world-wide to approve (Europe was months later) which in my mind shows that there is not much to an argument that political forces delayed the announcement (and how could the opposition / democrats influence a government institution such as the FDA if the ruling party couldn't? Please no deep state bonkers).
> The vaccines they are pushing now are the very same ones that were developed and tested while Trump was in office.
I would challenge that Trump should receive any political credit for supporting the development of the vaccines, because when the development was happening at Moderna (US) and BionTech (Germany) in spring 2020 Trump was still rejecting SARS-COV-19 as irrelevant. Trump should receive credit for Warpspeed that made sure the production of the developed vaccines is happening and is happening in a way that benefits the American people first (to great frustration to many Europeans).
You can't claim they were just following the rules, because they weren't. They bended and broke whatever rules they saw fit to get the vaccine done, including doing multiple phases simultaneously. If the FDA were following the rules, we still wouldn't have a vaccine available, because they still don't have full FDA approval.
>Trump was still rejecting SARS-COV-19 as irrelevant
Trump was called racist by Congressional D's for wanting to shut down the international travel to prevent the spread in February.
UK MHRA: Dec 2. US FDA: Dec 11. European EMA/EC: Dec 21. Months later? Hyperbole doesn't help your argument.
Still: don't you think the argument holds: US FDA is one of the first to approve is a sign that there can't be much delay.
They [Pfizer] delayed the announcement to three days after the elections. I'm sure those three days were critical in testing and making the vaccine safe for the consumer./s
It was an obvious political move to deny back-then US president Trump a public success as a consequence from him enacting "Operation Warp Speed".[0][1][2]
Also, "Trumpista"? You must feel really clever and enlightened.
[0] https://www.theguardian.com/us-news/video/2020/nov/20/trump-...
[1] https://outline.com/bH8vBM
[2] https://jewishjournal.com/news/314659/biden-is-backed-by-bot...
https://en.wikipedia.org/wiki/Operation_Warp_Speed?wprov=sfl...
ETA: Here's Jezebel's pre election coverage of their statements, when everyone still claimed Trump was lying about the vaccine being close.
>On Sunday, during an interview on CNN’s State of the Union, Harris was asked whether she would trust a vaccine doled out by the Trump administration before Election Day.
>“I think that we have learned since this pandemic started, but really before that, that there’s very little that we can trust that comes out of Donald Trump’s mouth,” Harris said.
https://www.msn.com/en-us/news/politics/anti-vaxxer-accuses-...
https://www.gov.ca.gov/2020/10/27/western-states-join-califo...
So maybe the real mistake was making it free. IF a company could make a few bucks for every vaccine administered, we would have seen the ad industry being its full might to bear to harvest as many people getting it as possible.
I would agree that stupid is winning, but I probably agree more with Kant on what Enlightenment means.
Calling someone stupid just because they do not share your blind trust in a bunch of people is kind of malicious.
Here's an interesting read from 2010 about how they were having former execs put in power in Canada. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2875889/
But I guess I'm the stupid one.
ok, lets educate. And before educating others lets educate ourselves.
https://www.cnbc.com/2021/07/30/cdc-study-shows-74percent-of...
"About three-fourths of people infected in a Massachusetts Covid-19 outbreak were fully vaccinated, according to new data published Friday by the CDC.
The new data, published in the U.S. agency’s Morbidity and Mortality Weekly Report, also found that fully vaccinated people who get infected carry as much of the virus in their nose as unvaccinated people. "
And another point related to education. Couple days ago a White House official stated :"We're dealing with an evolving virus. There is no precedent, and there is no playbook". Note, it isn't the White House of the previous buffoon, it is supposedly educated current White House. No wonder that being led by the people like this the current pandemic response isn't that much better than the ones of the previous centuries.
lets that sink. Especially considering that according to the above mentioned CDC data infected vaccinated people spread virus as infected unvaccinated.
>It reduces the effects so you're less likely to need hospitalization.
So, as current vaccines don't stop infection and spread, why would the most of the population whose risk of hospitalization is very low need (and even forced) to vaccinate?
Good point.
https://www.masslive.com/capecod/2021/07/how-provincetown-ac...
Despite virtually everyone in the town being vaccinated, 25% of the cases in the outbreak were in the remaining handful of unvaccinated people! As more areas reach close to 100% vaccination, we will start to see weird quirks of statistics like this more often, but if your takeaway here is that it's really easy for vaccinated people to catch and spread COVID, you're very wrong. Breakthrough infections are prevented by vaccines like Pfizer with about 99% efficacy, and without a breakthrough infection there is no transmission. If pretty much everyone were vaccinated, the odds of transmission would go to zero very rapidly.
i don't think it was neglected, as it is more like obvious thing wasn't explicitly stated - as vaccine obviously not increasing infection chances, 3/4 of infected being vaccinated means that vaccination rate >= 3/4, and that pretty much means "essentially the whole town is vaccinated"
Even if all of Provincetown residents were vaccinated that would not imply that anywhere near the majority of the people currently in Provincetown were vaccinated. I'm not sure how we can usefully estimate the vaccination rate of the people who have been in Provincetown recently. Google says the vaccination rate for Massachusetts is about 72%. That seems really worrying to me because if that vaccination rate describes the people in Provincetown then the infection rate implies that the vaccine is offering no real protection - i.e. 75% of population vaccinated and 75% of infected were vaccinated.
First, you are missing that this is a Delta out-break. Delta has been found to be able to evade the vaccines more than other variants. This is something new and something we didn't know just 2-3 months ago.
Next, without stating how many percent of the population where vaccinated in MA what does three-fourths imply?
Governments world-wide are struggling with how to respond to Delta. Because it is more contagious and even for vaccinated people there is a risk of hospitalizations, there is a struggle what to do. Lift all restrictions (as England has done)? Stay rather strict (as in France/Germany where there are still public mask mandates)?
The reality is that there is no playbook.
of course. Posting CDC data and WH quotes. While somehow nobody mentioned trolling wrt. GGP calling opponents "stupid" and "anti-intellectuals" without any data to back his position. Not surprising. Just a typical for these days debate with the "intellectuals".
>This is something new and something we didn't know just 2-3 months ago.
You are repeating the politicians' talking point what i was talking about . The very high chances that the "improved" 2.0 version would come were obvious pretty much from the start for anybody who paid even slightest attention to the precedents. Yet politicians have as always been playing ideologically charged short game all along right into populace ignorance while completely disregarding the long game.
> of course. Posting CDC data and WH quotes.
And ignoring almost everything that was replied to you.
> The very high chances that the "better" version would come were obvious pretty much from the start
I haven't heard that this would be obvious. Could you share what you consider that long game to be?
I'm okay with handing Trump a win if it can help save lives and end this pandemic.
July 6 - 46.7% Biden, 35.0% Trump.
This is reported by the way, right?
Those numbers are vaccination rates in red vs. blue counties where the blue counties have higher rates than the red counties.
That interpretation is supported by the way the article is framing the data in the conclusion.
> Although there has been an overall significant slow-down in COVID-19 vaccination rates in the U.S., these findings show a widening divide of communities at risk for COVID-19 along partisan lines. A key component of any effort to boost vaccination rates among Republicans will be...
Your idea wouldn't work. MAGAs hate liberals, and whatever liberals say appears to them like the voices of the grown-ups in the Peanuts cartoons. Maybe they pull out a few phrases to parody.
We've had over a year of near-continuous propaganda from the Republicans. Masks threaten your freedom. COVID is just the flu. No one dies from it, they put it down on every death certificate. Vaccinations kill you, not COVID. Vaccinations spread COVID, that's the new one.
Giving Trump credit for it would change nothing, even if the MAGAs heard it, and they wouldn't.
> which I think he deserves
It astonishes me that after years of watching Trump lie and lie and lie and lie, rambling and incoherent, often contradicting himself from day to day, watched eighteen months of him lying about COVID including that first key year where he did everything he could as President to prevent an effective response to COVID, that people somehow believe that he's secretly a competent administrator who "gets things done".
What Trump "deserves" is to spend the rest of his life in jail. His deliberate mishandling of COVID resulted in hundreds of thousands of needless American deaths: https://www.thelancet.com/journals/lancet/article/PIIS0140-6...
Operation Warp speed is why, while everyone else is still struggling with supply, everyone the US who wants a vaccine can get a vaccine.
https://en.wikipedia.org/wiki/Operation_Warp_Speed
Unsurprisingly, the Biden administration did the exact opposite of what I suggested. Further entrenching COVID as a partisan issue.
> In January 2021, White House press secretary Jen Psaki announced that the program was expected to undergo a restructure and renaming under the Biden administration. Also in January 2021, Dr. Moncef Slaoui, former Operation Warp Speed lead, was told not to use the name Operation Warp Speed anymore.
Diversity is the way nature builds robust and balanced ecosystems - AI tools that reinforce ideological homogeneity could be catastrophic, for reasons I consider self-evident.
Returning to the topic at hand, and as a counterpoint to your claims, there are legitimate second and third order effects to consider regarding vaccines and the evolutionary dynamics of SARS-CoV-2. Don't take my word for it - here is peer-reviewed research from the brightest minds and top institutions in the world, raising serious concerns about approaching eradication of the virus purely through mRNA vaccines in their current form [1].
I cannot emphasize enough that the spirit of science is to question assumptions and rigorously prove hypothesis.
[1] Risk of rapid evolutionary escape from biomedical interventions targeting SARS-CoV-2 spike protein (April 2021) https://pubmed.ncbi.nlm.nih.gov/33909660/
The reasons almost don’t matter. If the leadership doesn’t have at least two plans to deal with this expected situation, then they have failed.
But that's not what we're getting. Instead, we're getting a large number of people who haven't spent ten minutes bothering to learn anything about the science posting obvious bullshit.
How is anyone supposed to get actual science done if nearly all the "questioning" is repetitive worthless lies by people who have absolutely no interest in science at all and are simply screaming obvious lies at the top of their voices?
A question I ask a lot of the vaccine deniers on my Facebook page is, "Can you give a quick explanation in your own words why COVID-19 is so good at getting past the human immune system?"
No one has even tried to answer.
>How is anyone supposed to get actual science done...
One would hope that scientists would not be diverted by questions by people they do not work with or know on Facebook and are able to get actual science done regardless of a minority thinks.
I wonder if it's rational or a good thing to defend scientists by potentially restricting the ignorance of lay people in public places?
I think many would say it's good to avoid Harm and protect lives. It's easy to be irrational to defend rationality.
Edits. Oh, it could be that people are extending "follow the science" and seeing how others are not and thinking this as a kind of lack of faith. Perhaps the idea that actual scientists are being stopped doing science because of no faith in them by small numbers of the public is actually a religious thought. That actual science needs people to believe in it for it to work, and if an other doesn't believe it, they are anethema. That could have explain my feelings about the irrationality of this.
There is an argument they weren’t as effective as they could have been. Is that what you mean?
Superspreaders (antimaskers, antilockdown, antidistancing, antivaxxers) are the people keeping COVID alive.
It's not as black and white as you try to paint it.
I'm not living in the US but I've encountered a sizable number of people who dropped all caution and safety measures as soon as they got vaccinated even though they still can catch the virus and spread the virus with the added bonus of favoring the formation of immune escaping mutations.