U-Net CNN in APL: Exploring Zero-Framework, Zero-Library Machine Learning | Hacker News Reader

U-Net CNN in APL: Exploring Zero-Framework, Zero-Library Machine Learning

dl.acm.org

94 pointstosh14 comments

It's neat to see ongoing Co-dfns work from Aaron and others! There are a number of YouTube videos online if anyone is interested in very cool and esoteric yet serious programming: https://www.youtube.com/playlist?list=PLDU0iEj6f8duXzmgnlGX4....

Impressively concise implementation, really interesting paper! Benchmark looks quite questionable though -- e.g. they use fp64 (while any sane person would use at least f32, if not f16), batch 1 (while normally one would try to get the max batch size which fits into memory, and it would reach 1 only for much bigger models or inputs), and also measure the time including transfer to/from GPU (while it would normally be interleaved with GPU operations). Not sure what results would look like in a more realistic setup, but still getting within 2x of PyTorch even in such a setting looks impressive!

Interesting Futhark mention as well in the Related work section:

> Another approach to GPU-based array programming with an APL focus is the TAIL/Futhark system [8], which is a compiler chain taking APL to the TAIL (Typed Array Inter- mediate Language) and then compiling TAIL code using the Futhark GPU compiler backend.

I wonder what it would look like on kdb+?

Nobody should write backward pass by hand.

Maybe grad students...

Yes, every ML grad student should write autodiff lib (ideally using dual numbers). But nobody should ever write backward pass for whole net. Researchers did that in 80s and 90s. Once we had autodiff, it became obvious that deriving backward pass by hand is no-go.

Why? As a learning experience implementing backpropagation is extremely helpful, implementing an entire FNN/CNN from scratch is, to be honest.

Also Implementing some basic automatic differentiation is something you should probably have done once in your life you are interested in Machine learning or numerical mathematics.

Agreed. Implementing backprop myself—even if it was a crappy, slow version in MatLab—is what finally got me to understand it. I've worked as an ML researcher for 2 years since then and I'm still routinely happy that I have that deeper understanding of what's going on under the hood of the models I'm training.

I agree that implementing autodiff is nice exercise to grasp the whole stack. But its a ladder that has to be thrown away after you climbed it. In other words, write that autodiff for yourself, and then use production grade ML solution where you won't ever write backward pass by hand.

The APL paper doesn't have implementation of autodiff, rather 90s style hand derivations.

I did it and it wasn't worth it.

Tell that to any ML Compiler Engineer

Standard ML? I’m getting confused

Machine Learning. There are compilers like XLA, MLIR, TVM and others.

Reply on news.ycombinator.com