Hacker News Reader

Comment by fxtentacle | Hacker News Reader

I feel like this misses the point.

Python is used as the language to describe things, but nobody is executing Python on the GPU. Instead, things are transpiled into highly optimized byte code, C++ source, or chains of optimized GPU sharers. See TensorFlow XLA and AOT.

So when I set up a training pipeline with TensorFlow, then it'll barely run any python while training. Instead, the TF C++ core will execute a network of TF CUDA operators for me, all inside one python function call. That's why pythons global interpreter lock is pretty much a non-issue by now.

Since only 1% of my training time is even running inside python, I don't expect any performance benefits from replacing it. And if both are equally fast, I prefer the familiar option, meaning python.

View on HN

You can be sure that they are both well aware of this, Jeremy is one of the creator of the popular python library, book, and online course fast.ai built on top of PyTorch.

The relevant part of the their discussion is this one:

> Like Python is not the future of machine learning. It can't be. You know, it's so nicely hackable, but it's so frustrating to work with a language where you can't do anything fast enough unless you call out to some external code or C code, and you can't run anything in parallel unless you put in a whole other process. Like I find working with Python, there's just so much overhead in my brain to try to get it to work fast enough. It's obviously fine for a lot of things, but not really in the deep learning world or not really in the machine learning world. So, like, I really hope that Julia is really successful because there's a language with a nicely designed type system and a nicely designed dispatch system and most importantly, it's Julia all the way down so you can get in and write your GPU colonel in Julia, or all the basic stuff is implemented in Júlia all the way down until you hit the LLVM.

Here the point is that when using other languages (such as Julia) you actually can use the same language for your GPU and for the glue that keep your scripts together.

I feel like you (and them) are still missing the point. 99% of users don't write GPU kernels. They use simple function calls and that's all they see. 99% of users don't care if they call CUDA code from Python, or Julia code from Julia, as long as both are equally fast and API is clean (e.g. Pytorch).

I watched an interview with him, his end goal is no code machine learning so when he is saying python isn’t the future it is because he prefers a type system that can’t be achieved with python.

I agree that users don’t care, I use Spark a lot and don’t care if I write scala, python or even c# now.

Have a listen: https://www.wandb.com/podcast/jeremy-howard

(I think I have paraphrased the podcast correctly :) )

end goal is no code machine learning

Now that I agree on. I suspect we will see something like GPT-5 reliably converting plain English into code sooner than Julia displacing Python as king of ML.

I don’t think they are missing the point. These abstractions are far from perfect and trying to hunt down why something is not performing and tracing it back to how syntax is being interpreted as data is stored (GPU memory vs system memory vs sent over the pcie bus) or passed between other glue code is a source of frustration I’m sure many people have experienced.

It will take a lot for another language eco system to catch-up to something as broad and deep as python but these issues combined with the horrible experience that is dependency management with pure python vs compiled etc as well as the need for real-time/close to real-time systems, it seems very plausible that something like Julia or other llvm language will become the default over python for _these specific_ use cases, not in general though. Tooling on top of Julia can likely be far better and more easily made than living in python which is another aspect.

trying to hunt down why something is not performing

Again, 99% of the users won't be doing that no matter how good the tracing abilities are. They will post a question on SO/GitHub. Regardless of the language, modern DL frameworks are just too complex for regular users.

Perhaps the fact that 99% of users never write kernels is partial due to the fact that you need to switch languages to do so. Maybe a lot more users might want to write and modify GPU kernels if they had that as an easier option

Writing a fast CUDA kernel requires quite a bit of low level GPU knowledge. You typically do that when you really need it and it's not already available - like when you invented a new sparse GEMM algorithm, or want to speed up convolutions with binary weights/inputs. If you do need that, congrats, you're the 1%, and you're likely to not be afraid of mixing languages and getting your hands dirty with C++.

One of the reasons CUDA wipedout OpenCL is due to its polyglot support, no need to get hands dirty with C++.

Fortran, Java, .NET. Julia, D, Rust and anything else able to either generate PTX or integrate with CUDA LLVM backend can be used.

>Writing a fast CUDA kernel requires quite a bit of low level GPU knowledge

False. Not in Julia: https://juliagpu.gitlab.io/KernelAbstractions.jl/ https://github.com/mcabbott/Tullio.jl

To share the details, here's a post demonstrating Tullio.jl, a Julia-based higher order tensor library, is faster than OpenBLAS for gemm kernels:

https://discourse.julialang.org/t/realistically-how-close-is...

And PaddedMatrices.jl is another example:

https://discourse.julialang.org/t/ann-paddedmatrices-jl-juli...

So there isn't just one example but multiples at this point. RecursiveFactorization's `lu` tends to perform better than MKL for small matrices and this is what we use in the differential equation solvers when there are <100 ODEs.

That's all on CPUs though. KernelAbstractions.jl is the GPU version and that's still being constructed but moving strong.

For the person who downvoted me, care to refute my examples?

I haven't downvoted you, but from KernelAbstractions.jl docs it's not clear you will be able to write a fast kernel (emphasis on fast) without having as much GPU/CUDA knowledge as what's required to write the same fast kernel in CUDA. I might be wrong, and if so that would be great.

Here's an example: not long ago I needed a fast GPU implementation of numpy.unpackbits. I can do it in Pytorch, but it's slow and it wastes a ton of memory. So, my question is: can I write it in Julia as efficiently as I can do it in CUDA? If yes, will it be easier to do so?

KernelAbstractions.jl has some simple examples in its repo [^0]. The API looks simple and elegant. Code looks like regular Julia code. Documentation well behind CUDA, of course.

[^0]: https://github.com/JuliaGPU/KernelAbstractions.jl/blob/maste...

The code is essentially just Julia code:

https://juliagpu.gitlab.io/KernelAbstractions.jl/examples/na...

and you get ~60% of the best known CUDA implementation on that test:

https://juliagpu.gitlab.io/KernelAbstractions.jl/examples/pe...

Not perfect but not awful.

I’m sorry, what does it have to do with my question? I gave a specific example where I couldn’t get what I needed from Pytorch. Would Julia be any different in this situation?

It answers your question because it's a specific case of KA's compile structured loops with annotations to low level CUDA code.

If you could express the problem in generalized index notation, tullio.jl is an even higher level abstraction that uses KA

I don't know about the numpy function you mentioned, but here's a KA matmul. Definitely doesn't look like it has any CUDA specific things. Just a regular loop with some restrictions

@kernel function matmul_kernel!(a, b, c) i, j = @index(Global, NTuple)

    # creating a temporary sum variable for matrix multiplication
    tmp_sum = zero(eltype(c))
    for k = 1:size(a)[2]
        tmp_sum += a[i,k] * b[k, j]
    end

    c[i,j] = tmp_sum

end

Wish I had time to write the code and compare to pytorch for you though

Here's a Tulio example. Tulip also provides source to source derivatives. Both KA and Tulio work in the CPU with the same code

using Tullio, OffsetArrays

# A convolution with cyclic indices mat = zeros(10,10,1); mat[2,2] = 101; mat[10,10] = 1; @tullio kern[i,j] := 1/(1+i^2+j^2) (i in -3:3, j in -3:3)

@tullio out[x,y,c] := begin xi = mod(x+i, axes(mat,1)) # xi = ... means that it won't be summed, yj = mod(y+j, axes(mat,2)) @inbounds trunc(Int, mat[xi, yj, c] * kern[i,j]) # and disables automatic @inbounds, end (x in 1:10, y in 1:10) # and prevents range of x from being inferred.

It's mostly just math!

>it's not clear you will be able to write a fast kernel (emphasis on fast) without having as much GPU/CUDA knowledge as what's required to write the same fast kernel in CUDA. I might be wrong, and if so that would be great.

What I showed was the the essentially no-knowledge pure Julia code gets to ~60% of the optimized CUDA kernel that libraries tend to wrap. Take that as you will for whether that means that means that a user requires to write a "fast kernel": it's a data point and if you define that as fast enough then Julia does it, or if you require 100% as fast as the fastest CUDA kernel ever found :shrug: it doesn't get there quite yet because there's a lot of people working on getting it there.

This is because 99% of users know only Python, and C++ is too much for them. If the advanced stuff was available in the language they use, larger portion of users would write more advanced stuff.

I prefer Clojure for this. Even simpler than Python, fast, and enables me to match or even surpass the speed expected from mainstream tools.

See this for example: https://dragan.rocks/articles/20/Deep-Diamond-Deep-Learning-...

Did you write gpu kernels in clojure? Or did you make function calls to cudnn like pytorch et al does it?

Both. See https://github.com/uncomplicate/bayadera too.

Whatever the language, you have to make function calls to cuDNN, because cuDNN is optimized by Nvidia, which put serious resources into it. Even if you wrote everything in C++/CUDA, you probably can't match it for the standard stuff.

The custom things that I write, I prefer to write in CUDA kernels + Clojure management code.

Clojure kernels are technically possible in the same way as Julia or whatever kernels are possible, but the thing is that you have to match low-level hardware features anyway, so CUDA C/C++ just makes more sense for kernels, which are tiny part of code anyway.

So if you have to use cuda c/c++, then why not do it from python?

No. You don't use cuda/c++ as it is used from Python. You use cuda/c only in kernels. That is compiled dynamically, in process, from Clojure funcion calls, and 95% of related code is the management code, which is written in Clojure. There is no separate C++ build. You never touch any C++ tools such as CMake or anything. You work dynamically from the Clojure REPL.

As a Pytorch user, I never touch c/cuda code. I never touch cmake. I can work dynamically from python repl. So, again, why should I care how those c/cuda calls are made if I don’t see them?

Because Clojure comes with a platform that supports AOT/JIT compilation out of the box, has several state of the art GC algorithms and in the near future even value types, while offering the same dynamism as Python, with all the capabilities of a Lisp in code manipulation.

Because Python is an ugly, inexpressive, and incredibly verbose language.

How much of a factor is the JVM for supporting this? In other words, I'm curious about how other lisps would fare?

And notably, CUDA.jl or whatever Julia package you use for gpu acceleration still has to incorporate CUDA, the same way Pytorch does.

So to the end user who doesn't want to write CUDA, I don't see why using a Julia package for gpu acceleration is any better than using PyTorch in Python.

Wrong. Julia can call out to Cuda kernels, but it also does codegen for custom ops and types. Technically with enough engineering time, someone could rewrite all those C cuda kernels in julia, and julia would just generate low level machine GPU code on the fly.

Pytorch on the other hand HAS to glue together c and c++ cuda code.

You mean like PyCuda?

Nope. PyCuda is writing inline C cuda and hardly integrated with python.

Julia has four layers of GPU abstractions, each being successively easier to use ( bit a bit less control. ), all in pure Julia and reusing the compiler and normal Julia abstractions and integrated with AD etc

CUDA.JL, GPUArrays.jl, kernelabstracions.jl, tullio.jl.

At the highest levels you just write normal loops or index notation and the code and backward pass generated. Also generic to backend-simd and multithreading cpu and soon AMD rock, intel accelerator and distributed

This all composed with deep learning libs etc

Indeed.

The main questionable implicit assumption that is made is that Python can't "the future" (which is severely underspecified without a time horizon) because of what is bad about it today. It even is literally there "Python isn't the future... It can't be" <because something in Python is bad>.

That's highly questionable, because it leaves out - that people appear to have reasons to adopt Python, likely because of something that is good about it, (this complements the observation that not that many people might be impacted by the disadvantages). - that Python and the ML tools in Python might not be static and actually evolve in "the future".

It seems to me a curious lack of reflection substituted for by being particularly assertive about it.

To my mind Python not being the future is more likely to happen if Python becomes less likable or the attempts to improve the shortcomings (which are there, HPy as a medium-term way to get out of the "C extension vs. parallelism/speedup/GIL" corner, mypyc/numba/TorchScript/JAX pushing the "what can be efficiently expressed in Python" boundary) fail.

Julia, Swift, Dex-Lang, HaskTorch all get things right that Python doesn't. But do they get all the things right that Python does get right?

Disclaimer: I'm biased because I like Python and have some involvement in Python ML tools.

Jeremy's literal statement is:

"Like Python is not the future of machine learning. It can't be. You know, it's so nicely hackable, but it's so frustrating to work with a language where you can't do anything fast enough unless you call out to some external code or C code, and you can't run anything in parallel unless you put in a whole other process."

So exactly the point made here.

Can someone tell me why ml/dl projects don't include even a requirement.txt or list of dependencies often?

I have always wondered why. I see so many projects that miss those. Projects from big companies and teams. Are they intentionally adding hurdles to using the code?

Reply on news.ycombinator.com