Nvidia Scientists Take Top Spots in 2021 Brain Tumor Segmentation Challenge

He likes to say AI=avian intelligence...

Really not fake … ???

Actual article (2015): https://journals.plos.org/plosone/article?id=10.1371/journal...

Pretty incredible. I wonder if you forced a radiologist to make a decision within a second or two if it would improve their scores...allowing them to rely on the instinct they've developed vs the cognitive wet blanket that factors in all of the externalities of making a decision.

Intelligence to create innovative models, hardware to test/iterate models, etc.

AlphaFold is a clear example of what's possible with big money.

I dunno, their A100 results took about 20-30 minutes on 8 x A100s [1]. 8xA100s is like $24/hr on GCP at on-demand rates [2]. So like $12/run.

The efficiency was okay but not linear, so if you were more cost constrained you might go with 1xA100 for $3/hr and have ~2.5hr training times.

Getting that performance out of a GPU is more challenging than getting access to the GPUs. All the major cloud providers offer them.

(Nit: GCP deployed the 40 GiB cards rather than the later 80 GiB parts, but let's ignore that).

but it often doesn't matter

[1] https://github.com/NVIDIA/DeepLearningExamples/tree/master/P...

[2] https://cloud.google.com/compute/gpus-pricing

I shared the sentiment but I have began to think otherwise. First of all, we need solutions to big problems. I wouldn't want to tackle Brain tumor problems using DL but someone with the brainpower and resources such as NVIDIA, why not?

But besides that, the whole scientific field is pretty young. An average coder or DL researcher isn't useless because Google has all the machines. There are fields where DL isn't used much. Even in the DL field there are problems to solve. Algorithms to optimize.

The common sentiment in r/ML is that a) big labs need some accountability because they keep getting credit for work they published but never released source code and therefore third party audit is impossible *. Furthermore, Resnet-50 is back to sota because of up-to-date training protocols, and every paper that claimed superiority was using outdated training techniques (or even none at all) for the baseline.

Furthermore, there are many suspicions that the current generation of DL/ML has reached a local minimum because we have collectively pushed for models, architectures, functions, and optimizers that did well on existing hardware. For example, one hypothesis is that the Adam and AdamW optimizers are doing so well on average because we optimized model architectures for them.

* Before somebody claims that this is part of the research process and we should be able to validate and implement models on our own, I would like to point out that there have been multiple instances where the math appeared sound, but the implemented models were bugged, and outperformed the existing models *because* they were bugged (NB biased) and the test sets didn't capture that. When corrected, the models didn't perform any better than existing unbiased methods.

I attended the MICCAI workshops this year, and while I didn't go to the brain segmentation challenge, in the ones I did go to the winners were generally academic individuals or groups.

My MICCAI login is on my work computer, so I'm not going to go and check right now, but a quick google shows that the winners of FeTS2021 were all academics: https://twitter.com/FeTS_Challenge/status/144401094937199411... and HEKTOR2021: https://www.aicrowd.com/challenges/miccai-2021-hecktor/leade... looks similar.

EDIT: Academic groups are of course not the same as 'the average professional coder', but this is a fast-moving research field and if an individual is competing with the state-of-the-art that doesn't sound particularly 'average' to me.

EDIT 2: The DiSCO challenge appears to have been dominated by academics, both individuals and small teams : https://twitter.com/GabrielPGirard/status/144397287871477356...

- from the "data" section of http://www.braintumorsegmentation.org/

[1] https://ieeexplore.ieee.org/document/6975210

[2] https://arxiv.org/abs/1811.02629

The three tissue types of interest are fairly easy to identify in most cases. Edema is bright on the FLAIR sequence, enhancing tumor is bright on T1 post-contrast and dark on pre-contrast, and necrosis is relatively dark on T1 pre- and post-contrast while also being surrounded by enhancing tumor. These rules hold true in most cases, so it’s really just a matter of having the algorithm find these simple patterns. The challenge in doing this manually is the amount of time it takes to create a really high quality 3D segmentation. It’s painful and very tough to do with just a mouse and 3 orthogonal planes to work with.