Sounds about right.
It's selfishness and narcissism masquerading as philanthropy. Instead of doing an actual useful task, they want credit for the exciting/pioneering front line work.
It was the same thing with the ventilator shortage. How many people publicly declared they were "inventing" a new, easy to make ventilator? Design was never the problem, manufacturing was. But few were interested in manufacturing extant (and already FDA approved) designs.
- 100s of models were done incorrectly, and none even remotely correctly
- there is no phenomenon to correctly model
Both seem extremely unlikely. Not sure what the alternatives are.
- Technology not suitable or adequate for this use case.
I mean, we've been to the "AI over-promises and under-delivers" rodeo before.
Yeah. Attempts at powered flight had a 100% failure rate in the 1800s.
It's kind of absurd to think that "AI" (especially in its current incarnation) must be able to solve any "hot" problem that it's thrown at.
And those older systems has more intelligence in them. Todays’s “AI” is a small number of tricks aimed at large amounts of data, with an unprecedented and enormous amount of marketing added.
If you're talking about deep neural networks, I can understand this viewpoint. But generally the last decade has proven widely successful for high quality vision recognition models that just weren't available before then. And with something like transfer learning, you can take a powerful off-the-shelf model and specialize it without a huge dataset.
The real barriers IME are around legal and privacy implications. There's also a strong argument about if these models creating enough value in the first place, but they can work on a technical level.
More or less. It's almost certain if "AI" makes any headway at all, it will fall far short of the hype.
> That seems like a fairly fringe view. Especially on a problem that has well defined data like medical imaging.
I doubt it.
Also on the wet side, there is very little progress on something like general antiviral drugs.
Some stuff is just really hard.
You can make all the models about phlogiston you want, none of them are going to accurately explain fire.
Another problem you have is when you do it right the results look worse than a biased dataset. So there's some incentive to continue working with a biased dataset.
COVID exists, it causes physical changes and affects the world outside the host. 100% it can be modelled somehow in theory.
That being said, there are harsh rules against collecting data (for example, it would be extremely illegal to create a sample set by taking a random group of people and infecting half of them with COVID on purpose). So I expect actually getting training data is a huge hill to climb. If most of the models used the same training data I assume there is a reason like that.
So in practice, I don't expect a model quickly.
The entire premise of modelling something rests on the fact that the behaviour of what you're trying to model in the future depends on its behaviour in the past.
This is not true for all systems, it's not even true for most systems. John Kay differentiates between 'resolvable uncertainty' and 'radical uncertainty'. Modelling a volcano eruption is resolvable because it's a reasonably predictable system and you can make confident stochastic claims. Complex human systems are chaotic, have 'unknown unknowns', are simply subject to chance or non-linear behaviour that completely throws any prediction off.
You cannot model something that is subject to forces you cannot even know at the point of making the model, and this is true for any complex human system pretty much. Modelling Covid as if we have access to Hari Seldon's psycho-history was always a terrible idea.
no, the correct behavior under uncertainty isn't modelling, it's something akin to Taleb's notion of antifragility or robustness. The correct response to a pandemic isn't some sort of Hari Seldon psycho-history which, as the article points out, is futile. The right response is building systems so responsive they can crush a pandemic before it gets to that stage.
For that we need better public health measures, those depend on modeling, vaccine development, etc.
There's no escaping understanding human health. It underpins and informs our responses. (Which kind of viruses are likely to pose a real threat, monitoring those threats, evaluating them, models, predictions, data, etc.)
I'm not talking about one big Seldon-esque equation, but about the public/global health system we already have, the whole of medical research, and so on.
These toy classifiers trained on COVID scans are in the same class as Tesla's dangerous "autopilot", because the system is too limited, it doesn't understand context, it lacks a world model. (Tesla's lack epistemic convergence and stability for object detection, ie. it lacks the understanding that cars, roads, obstacles don't flicker in and out of existence, nor do they change from frame to frame. Similarly a "medical AI" needs to understand that medical imaging artifacts are unlikely to be good proxies for real diagnosis, so it needs to be able to abstract away from the image, process away the artifacts and recording environment differences, etc.)
People with a background in methods like solving problems that are easy to solve with the methods they know. Those are rarely the problems people in biology/medicine want to solve. Some common pitfalls include:
* Asking the right questions.
* Framing the questions correctly.
* Finding computational problems that capture the essence of the questions and that can be solved efficiently.
* Interpreting the results.
* Avoiding systemic biases in rare but important edge cases.
Finding the right problems to solve takes time, and the first attempts will probably fail.
First hundreds of attempts? In your experience, is that a reasonable number?
If 10 people hike to Mt Everest's base camp, it's not the equivalent of one person climbing ~10 times further to the peak.
Given the fact that at that time there was still quite a lot of "if we all help we can beat the pandemic" positiveness going around. An easily run-able piece of small code that promised "results" was available and the fact that a lot of organizations wanted positive PR, you get these insultingly low quality "research" practices.
I looked at a lot of the initial papers and models and most were forks of COVID-Net or used an already established model like ImageNet and retrained it with the bad dataset. Presto! Paper out PR happy. Very few questioned the dataset or the approach in general. But they loved bragging about it on LinkedIn.
100s of wrong models doesn't sound extremely unlikely to me. It's really easy to train a model and test that it works on some dataset and then find it doesn't work on a different one. Setting up the inputs for training is difficult because of data privacy, there's intense pressure to publish, models have no expectation of explanability, lots of reasons for a wrong model that looks promising.
I've worked adjacent to people trying to solve problems with machine learning and it's rough going. When things work, ok nice, but when they don't, it's much harder to tweak than something based on heuristics (which, of course, don't always apply either).