The Essential Tools of Scientific Machine Learning | Hacker News Reader

The Essential Tools of Scientific Machine Learning

stochasticlifestyle.com

This is an aside to the main point of the article, but I have to take issue with a couple of quotes:

> For a long time scientific computing has kept machine learning at an arm's length because its lack of interpretability and structure mean that, despite it’s tremendous predicitve success, it is almost useless for answering these kinds of central scientific questions.

> However, the recent trend has been to merge the two disciplines, allowing explainable models that are data-driven, require less data than traditional machine learning, and utilize the knowledge encapsulated in centuries of scientific literature.

Recent trend?? This is the norm in every scientific field I've even slightly worked in... This quote just described what we call "inverse theory" in the geosciences. It's the root of pretty much everything we do and has been for many decades. Those of in the scientific computing world have hardly been keeping machine learning at arm's length... A lot of machine learning methods come from scientific fields (e.g. gaussian processes).

There's more to machine learning than CNNs, and many fields have been using machine learning much longer terms like "data science" and "AI" have been around. We just trend towards parametric approaches or approaches that can be reasoned about more clearly. (E.g. lots of convex optimization problems and interpolation/"super-resolution" problems.) Those are machine learning methods as well.

All that said, most of what they're talking about in this article is about approximating solutions to differential equations with purely non-parametric approaches. That is somewhat new and is a much narrower topic, which this article does a nice job of describing. (I don't really take objection to the article as a whole, but a few sentences in it strike me as presumptuous.)

Solving inverse problems has been around forever, agreed. Not just in geosciences but also aerospace, biology, etc. But, some of the ways that people are beginning to use universal function approximators inside of scientific simulators is new. The techniques are very similar to traditional inverse problems though: use adjoint equations for gradients and update parameters with some optimizer. It's just now they models include more non-mechanistic terms to fit. There were hints of it before with latent force models, but I am not sure of anyone using this idea to successfully solve 100 dimensional PDEs until 2 years ago.

>There's more to machine learning than CNNs.

Most definitely agreed. I think there's a lot to do with utilizing all of the methods of machine learning neural networks and beyond, automatically within scientific simulation codes.

Yeah, using non-parametric methods to approximate PDEs is really what this article is about, and it is a recent and very fruitful trend. I got a bit unnecessarily up-in-arms about phrasing.

(edit: And I just realized you're the author -- excellent work by the way!)

Very much Julia oriented. Nice that Julia is catching up with other. This one however is missing comparison with R tools. I guess R would also be quite green on the "Quick Summary Table".

Do you have an example of an R automatic differentiation package which has an ecosystem of compatible tools like PDE solvers, automatic sparsity detection, convolutional neural networks, and global sensitivity analysis? If you have one I'll add it to the list. But the ones that I know are things like madness [1], radx [2], and TMB [3], all of which are good AD systems in their own right (the latter most having some good parallelism), but are missing most of the relevant features for a discussion of scientific ML (but would be great for standard ML). I could also point to autodiffr [4], but that's using Julia to autodiff R.

But if there's one I missed I'd love to hear about it and start tracking the project!

[1] https://cran.r-project.org/web/packages/madness/vignettes/in...

[2] https://rdrr.io/github/quantumelixir/radx/man/radx-package.h...

[3] https://www.jstatsoft.org/article/view/v070i05/v70i05.pdf

[4] https://non-contradiction.github.io/autodiffr/index.html

I don't, but maybe a disclaimer on R in this article could bring those who could share their experience with R?

I would be curious to know whether I have been the only one using TMB in production in a tech company. A tricky piece of software, but very powerful.

Good thing I did not ever "Give Up On Julia"™! So happy to see numerical programming breaking new grounds like this

You and me both ^_^

How do I, with a very nice AMD GPU, and running on Windows, play around with any sort of GPU accelerated learning for anything? Last I knew, pretty much nothing worked with either of those caveats.

Running Windows is perfectly fine; the major libraries for GPU-accelerated autodiff and networks (CUDNN with Pytorch or Tensorflow) have great support nowadays. It's the AMD GPU that remains essentially useless, as of 2019. If you want to get into the game, I'd recommend buying a middle-of-the-road NVIDIA GPU like the RTX2060.

For toying with autodiff and basic CNNs, CPU works just fine by the way...

>It's the AMD GPU that remains essentially useless, as of 2019

I guess more important question... Whyyyyyyyyyyyyy

Probably support for CUDA over OpenCL.

CuBLAS and CuDNN are just better libraries than what exist on OpenCL right now. Until that changes it'll be hard to switch for numerical work.

The point is that and AMD GPU is far from useless. The only thing that it DOES lack is the out of the box from major Python/R/whatever libraries. Why? Not because AMD GPU does not work, but because most (perhaps all) of these high-level libraries rely of underlying performance libraries provided by Nvidia.

Despite all the talk about autodiff this or that, the stuff that matters is implemented by hand by Nvidia and Intel engineers and then high level libraries build on top. AMD is simply lagging in providing low-level C libraries and GPU kernels for that.

For example, let me chip in with the libraries I develop, in Clojure, no less. They support BOTH Nvidia GPU AND AMD GPU backends. Most of the stuff is equally good on AMD GPU and Nvidia GPU. With less fuss than in Julia and Python, I'd argue.

Check out Neanderthal, for example: https:neanderthal.uncomplicate.org

Top performance on Intel CPU, Nvidia GPU, AND AMD GPU, from Clojure, with no overhead, faster than Numpy etc. You can even mix all three in the same thread with the same code.

Lots of tutorials are available at https://dragan.rocks

I'm writing two books about that:

Deep Learning for Programmers: An Interactive Tutorial with CUDA, OpenCL, MKL-DNN, Java, and Clojure [1]

and

Numerical Linear Algebra for Programmers: An Interactive Tutorial with GPU, CUDA, OpenCL, MKL, Java, and Clojure [2]

Drafts are available right now at https://aiprobook.com

[1] https://aiprobook.com/deep-learning-for-programmers [2] https://aiprobook.com/numerical-linear-algebra-for-programme...

>It's the AMD GPU that remains essentially useless, as of 2019.

This appears to finally be starting to change. See:

https://github.com/RadeonOpenCompute/ROCm

https://github.com/ROCmSoftwarePlatform/tensorflow-upstream/

Yeah, this deplorable state of the software on GPU side is the reason that GPGPU is staying a niche technology for the foreseeable future.

For me, the easiest thing has been to set up a google cloud compute server with a good gpu. They start you off with a $300 credit good for one year and charge like $0.50/hour to run. You can get pretty solid models running in the scale of minutes. Spend your time learning, not setting up.

I was hoping 'scientific' meant publishing the training data and resulting AI model in order that others could either use it or reproduce it. A lot of neural net A.I. isnt really scientific in this sense.

Reply on news.ycombinator.com