An AI for AI: New Algorithm Poised to Fuel Scientific Discovery

This article is mis-titled. The ORNL project seems to be entirely about optimizing DNN hyperparameters (like Google's AutoML, I think), and not about inventing novel DNN architectures.

Worse, the work has absolutely nothing to do with Scientific Discovery, which is a difficult and hugely ambitious area of ML study which involves generating novel induction of necessary mechanisms, not just the mostly random generation and testing of plausible patterns against outcomes.

Unless a ML method proposes hypotheses that are based on the induction of novel mechanistic principles, it ain't discovery. It's mere trial and error.

[Edited for clarity.]

The nvidia article is recent, but the referenced scientific paper is from 2015:

https://www.researchgate.net/profile/Steven_Young11/publicat...

Presentation:

http://ornlcda.github.io/MLHPC2015/presentations/4-Steven.pd...

In the article:

> the team developed an algorithm that automatically generates neural networks

Title of the paper:

> Optimizing Deep Learning Hyper-Parameters Through an Evolutionary Algorithm.

I don't know how to feel when things like this happen.

I stopped reading at "Modeled loosely on the connections in the human brain, these do the “learning” in deep learning."

“Scaled across Titan’s 18,688 Tesla GPUs, ”. I am sure that’s all Nvidia really cared about, they’d write anything that made their customers look smart.

Like most automated machine learning, this looks incredibly intensive/expensive computationally. I look forward to the day it's within reach for more researchers. Until then, it's a weird kind of news that has little impact on how most NNs get tuned.

However, I disagree about the lack of a connection between ORNL's work on MENNDL and scientific discovery. I'm going off of what ORNL told me, but according to them, using DL has been hard for scientists and this could make it easier. I am a writer, not an engineer, but here is how ORNL explains it: "Because scientific data often looks much different from the data used for animal photos and speech, developing the right artificial neural network can feel like an impossible guessing game for nonexperts. To expand the benefits of deep learning for science, researchers need new tools to build high-performing neural networks that don’t require specialized knowledge."

Their news release has more technical detail than I could put in my blog. https://www.ornl.gov/news/scaling-deep-learning-science

The math behind Gene Regulatory Networks is basically the same as that used in Neural Networks. Gene Regulatory Networks control cell state and cellular development (including neurogenesis) and we can see neurogenesis as natures way of using neural network technology (Gene Regulatory Networks) to build a higher level neural network technology (Neural Networks).

You are right about the 2015 publication. I believe ORNL waited until it had results from a user (Fermilab) to release the information. There are also two related papers: -- one is about the Fermilab's neutrino research using MENNDL http://ieeexplore.ieee.org/document/7966131/ -- and the other is about evolving deep neural nets in HPC: https://dl.acm.org/citation.cfm?doid=3146347.3146355

So neural architecture search turbo charged into 24 hours with the help of a supercomputer?

How is this different from a parallel set of EA runs to tune hyperparameters and topology?

The ORNL next gen computer is Summit. Lots, and lots of Teslas in there. https://www.olcf.ornl.gov/summit-early-science/

Somewhat off topic, but I like that they named it "MENNDL". Kinda like a fellow named "Mendel".

Sounds like this is one of the cases where an "evolutionary algorithm" is essentially brute force search.

Folding AI against itself does seem like the theme of the next major AI frontier. Already we have GAN networks doing incredible things.