Hacker News Reader

Comment by p1esk | Hacker News Reader

I feel like you (and them) are still missing the point. 99% of users don't write GPU kernels. They use simple function calls and that's all they see. 99% of users don't care if they call CUDA code from Python, or Julia code from Julia, as long as both are equally fast and API is clean (e.g. Pytorch).

View on HN

This is because 99% of users know only Python, and C++ is too much for them. If the advanced stuff was available in the language they use, larger portion of users would write more advanced stuff.

I prefer Clojure for this. Even simpler than Python, fast, and enables me to match or even surpass the speed expected from mainstream tools.

See this for example: https://dragan.rocks/articles/20/Deep-Diamond-Deep-Learning-...

Did you write gpu kernels in clojure? Or did you make function calls to cudnn like pytorch et al does it?

Both. See https://github.com/uncomplicate/bayadera too.

Whatever the language, you have to make function calls to cuDNN, because cuDNN is optimized by Nvidia, which put serious resources into it. Even if you wrote everything in C++/CUDA, you probably can't match it for the standard stuff.

The custom things that I write, I prefer to write in CUDA kernels + Clojure management code.

Clojure kernels are technically possible in the same way as Julia or whatever kernels are possible, but the thing is that you have to match low-level hardware features anyway, so CUDA C/C++ just makes more sense for kernels, which are tiny part of code anyway.

So if you have to use cuda c/c++, then why not do it from python?

No. You don't use cuda/c++ as it is used from Python. You use cuda/c only in kernels. That is compiled dynamically, in process, from Clojure funcion calls, and 95% of related code is the management code, which is written in Clojure. There is no separate C++ build. You never touch any C++ tools such as CMake or anything. You work dynamically from the Clojure REPL.

As a Pytorch user, I never touch c/cuda code. I never touch cmake. I can work dynamically from python repl. So, again, why should I care how those c/cuda calls are made if I don’t see them?

Because Clojure comes with a platform that supports AOT/JIT compilation out of the box, has several state of the art GC algorithms and in the near future even value types, while offering the same dynamism as Python, with all the capabilities of a Lisp in code manipulation.

Because Python is an ugly, inexpressive, and incredibly verbose language.

How much of a factor is the JVM for supporting this? In other words, I'm curious about how other lisps would fare?

I don’t think they are missing the point. These abstractions are far from perfect and trying to hunt down why something is not performing and tracing it back to how syntax is being interpreted as data is stored (GPU memory vs system memory vs sent over the pcie bus) or passed between other glue code is a source of frustration I’m sure many people have experienced.

It will take a lot for another language eco system to catch-up to something as broad and deep as python but these issues combined with the horrible experience that is dependency management with pure python vs compiled etc as well as the need for real-time/close to real-time systems, it seems very plausible that something like Julia or other llvm language will become the default over python for _these specific_ use cases, not in general though. Tooling on top of Julia can likely be far better and more easily made than living in python which is another aspect.

trying to hunt down why something is not performing

Again, 99% of the users won't be doing that no matter how good the tracing abilities are. They will post a question on SO/GitHub. Regardless of the language, modern DL frameworks are just too complex for regular users.

I watched an interview with him, his end goal is no code machine learning so when he is saying python isn’t the future it is because he prefers a type system that can’t be achieved with python.

I agree that users don’t care, I use Spark a lot and don’t care if I write scala, python or even c# now.

Have a listen: https://www.wandb.com/podcast/jeremy-howard

(I think I have paraphrased the podcast correctly :) )

end goal is no code machine learning

Now that I agree on. I suspect we will see something like GPT-5 reliably converting plain English into code sooner than Julia displacing Python as king of ML.

Perhaps the fact that 99% of users never write kernels is partial due to the fact that you need to switch languages to do so. Maybe a lot more users might want to write and modify GPU kernels if they had that as an easier option

Writing a fast CUDA kernel requires quite a bit of low level GPU knowledge. You typically do that when you really need it and it's not already available - like when you invented a new sparse GEMM algorithm, or want to speed up convolutions with binary weights/inputs. If you do need that, congrats, you're the 1%, and you're likely to not be afraid of mixing languages and getting your hands dirty with C++.

>Writing a fast CUDA kernel requires quite a bit of low level GPU knowledge

False. Not in Julia: https://juliagpu.gitlab.io/KernelAbstractions.jl/ https://github.com/mcabbott/Tullio.jl

To share the details, here's a post demonstrating Tullio.jl, a Julia-based higher order tensor library, is faster than OpenBLAS for gemm kernels:

https://discourse.julialang.org/t/realistically-how-close-is...

And PaddedMatrices.jl is another example:

https://discourse.julialang.org/t/ann-paddedmatrices-jl-juli...

So there isn't just one example but multiples at this point. RecursiveFactorization's `lu` tends to perform better than MKL for small matrices and this is what we use in the differential equation solvers when there are <100 ODEs.

That's all on CPUs though. KernelAbstractions.jl is the GPU version and that's still being constructed but moving strong.

For the person who downvoted me, care to refute my examples?

I haven't downvoted you, but from KernelAbstractions.jl docs it's not clear you will be able to write a fast kernel (emphasis on fast) without having as much GPU/CUDA knowledge as what's required to write the same fast kernel in CUDA. I might be wrong, and if so that would be great.

Here's an example: not long ago I needed a fast GPU implementation of numpy.unpackbits. I can do it in Pytorch, but it's slow and it wastes a ton of memory. So, my question is: can I write it in Julia as efficiently as I can do it in CUDA? If yes, will it be easier to do so?

KernelAbstractions.jl has some simple examples in its repo [^0]. The API looks simple and elegant. Code looks like regular Julia code. Documentation well behind CUDA, of course.

[^0]: https://github.com/JuliaGPU/KernelAbstractions.jl/blob/maste...

The code is essentially just Julia code:

https://juliagpu.gitlab.io/KernelAbstractions.jl/examples/na...

and you get ~60% of the best known CUDA implementation on that test:

https://juliagpu.gitlab.io/KernelAbstractions.jl/examples/pe...

Not perfect but not awful.

I’m sorry, what does it have to do with my question? I gave a specific example where I couldn’t get what I needed from Pytorch. Would Julia be any different in this situation?

>it's not clear you will be able to write a fast kernel (emphasis on fast) without having as much GPU/CUDA knowledge as what's required to write the same fast kernel in CUDA. I might be wrong, and if so that would be great.

What I showed was the the essentially no-knowledge pure Julia code gets to ~60% of the optimized CUDA kernel that libraries tend to wrap. Take that as you will for whether that means that means that a user requires to write a "fast kernel": it's a data point and if you define that as fast enough then Julia does it, or if you require 100% as fast as the fastest CUDA kernel ever found :shrug: it doesn't get there quite yet because there's a lot of people working on getting it there.

It answers your question because it's a specific case of KA's compile structured loops with annotations to low level CUDA code.

If you could express the problem in generalized index notation, tullio.jl is an even higher level abstraction that uses KA

I don't know about the numpy function you mentioned, but here's a KA matmul. Definitely doesn't look like it has any CUDA specific things. Just a regular loop with some restrictions

@kernel function matmul_kernel!(a, b, c) i, j = @index(Global, NTuple)

    # creating a temporary sum variable for matrix multiplication
    tmp_sum = zero(eltype(c))
    for k = 1:size(a)[2]
        tmp_sum += a[i,k] * b[k, j]
    end

    c[i,j] = tmp_sum

end

Wish I had time to write the code and compare to pytorch for you though

Here's a Tulio example. Tulip also provides source to source derivatives. Both KA and Tulio work in the CPU with the same code

using Tullio, OffsetArrays

# A convolution with cyclic indices mat = zeros(10,10,1); mat[2,2] = 101; mat[10,10] = 1; @tullio kern[i,j] := 1/(1+i^2+j^2) (i in -3:3, j in -3:3)

@tullio out[x,y,c] := begin xi = mod(x+i, axes(mat,1)) # xi = ... means that it won't be summed, yj = mod(y+j, axes(mat,2)) @inbounds trunc(Int, mat[xi, yj, c] * kern[i,j]) # and disables automatic @inbounds, end (x in 1:10, y in 1:10) # and prevents range of x from being inferred.

It's mostly just math!

One of the reasons CUDA wipedout OpenCL is due to its polyglot support, no need to get hands dirty with C++.

Fortran, Java, .NET. Julia, D, Rust and anything else able to either generate PTX or integrate with CUDA LLVM backend can be used.

And notably, CUDA.jl or whatever Julia package you use for gpu acceleration still has to incorporate CUDA, the same way Pytorch does.

So to the end user who doesn't want to write CUDA, I don't see why using a Julia package for gpu acceleration is any better than using PyTorch in Python.

Wrong. Julia can call out to Cuda kernels, but it also does codegen for custom ops and types. Technically with enough engineering time, someone could rewrite all those C cuda kernels in julia, and julia would just generate low level machine GPU code on the fly.

Pytorch on the other hand HAS to glue together c and c++ cuda code.

You mean like PyCuda?

Nope. PyCuda is writing inline C cuda and hardly integrated with python.

Julia has four layers of GPU abstractions, each being successively easier to use ( bit a bit less control. ), all in pure Julia and reusing the compiler and normal Julia abstractions and integrated with AD etc

CUDA.JL, GPUArrays.jl, kernelabstracions.jl, tullio.jl.

At the highest levels you just write normal loops or index notation and the code and backward pass generated. Also generic to backend-simd and multithreading cpu and soon AMD rock, intel accelerator and distributed

This all composed with deep learning libs etc

Reply on news.ycombinator.com