nathanielsimard

I think it will be cost effective at some point. Computers were limited to research institutes before the personal computer arrived.

CubeCL supports WebGPU and can be used with wasm!

I don't recall the reason why, point is a valid name.

Well we can agree to disagree, CubeCL also has the concept of instruction parallelism, which would be used to target simd instructions on CPU. Our algorithms are normally flexible on both the plane size and the line size, adapting to the hardware with comptime logique. You are free to dislike the naming, but imo a mix of multiple APIs is worse than something new.

Using the naming from one of the existing API would put too much bias towards that API. It started as a WebGPU project early on, but some features are not present so mixing terms wasn't ideal. We're also working on extending CubeCL to CPU, so we want terms not only tied to the GPU word.

One of the author here, don't hesitate if you have any question or comment!

We have safe and unsafe version for launching kernels where we can ensure that a kernel won't corrupt data elsewhere (and therefore won't create memory error or segfaults). But within a kernel ressources are mutable and shared between GPU cores, since that's how GPUs work.

The need to build CubeCL came from the Burn deep learning framework (https://github.com/tracel-ai/burn), where we want to easily build algorithms like in CUDA with a real programming language, while also being able to integrate those algorithms inside a compiler at runtime to fuse dynamic graphs.

Since we don't want to rewrite everything multiple times, it also has to be multi-platform and optimal, so the feature set must be per-device, not per-language. I'm not aware of a tool that does that, especially in Rust (which Burn is written in).

We support warp operations, barriers for Cuda, atomics for most backends, tensor cores instructions as well. It's just not well documented on the readme!

One of the main author here, the readme isn't really well up-to-date. We have our own gemm implementation based on CubeCL. It's still moving a lot, but we support tensor cores, use warp operations (Plane Operations in CubeCL), we even added TMA instructions for CUDA.

A lot of things happen at compile time, but you can execute arbitrary code in your kernel that executes at compile time, similar to generics, but with more flexibility. It's very natural to branch on a comptime config to select an algorithm.

During the last iteration of CubeCL, we refactored the matrix multiplication GPU kernel to work with many different configurations and element types.

The goal was to improve performance and flexibility by using Tensor cores when available, performing bounds checks when necessary, supporting any tensor layout without any new allocation to transpose the matrices beforehand, and implementing many improvements.

The performance is greatly improved, and now it works better with many different matrix shapes. However, I think we created an atrocity in terms of compilation speed. Simply compiling a few matmul kernels, using incremental compilation, took close to 2 minutes.

So we fixed it! I took the time to write a blog post with our solutions, since I believe this can be useful to Rust developers in general, even if the techniques might not be applicable to your projects.

Feel free to ask any questions here, about the techniques, the process, the algorithms, CubeCL, whatever you want!

Burn is now the first fully Rust-native deep learning framework. Do everything in Rust, from GPU kernels to model definition. No CUDA, C++ or WGSL needed thanks to CubeCL that we released last month.

We've introduced a new tensor data format that offers faster serialization/deserialization and supports Quantization (currently in Beta). Loading and saving can be up to 4X as fast.

As always, we've added numerous bug fixes, new tensor operations, and improved documentation. Thanks to all contributors, over 50 for this release.

Introducing CubeCL, a new project that modernizes GPU computing, making it easier to write optimal and portable kernels. CubeCL allows you to write GPU kernels using a subset of Rust syntax, with ongoing work to support more language features.

Why it Matters

CubeCL tackles three major challenges in GPU computing

- Portability: The same codebase can be used to program any GPU without a loss in performance.

- Usability: No need for a new shader language — simply add an attribute on top of your Rust code and voilà, it can now run on any GPU.

- Performance: We generate fine-grained kernel specialization via an innovative compile-time system to use the most efficient instructions available.

How it works

CubeCL leverages Rust's proc macro system in a unique two-step process:

1. Parsing: The proc macro parses the GPU kernel code using the syn crate.

2. Expansion: Instead of immediately generating an Intermediate Representation (IR), the macro generates a new Rust function.

The generated function, semantically similar to the original, is responsible for creating the IR when called. This approach differs from traditional compilers, which typically generate IR directly after parsing. Our method enables several key features:

- Comptime: CubeCL functions can contain sections marked as Comptime. These sections are executed during compilation rather than at runtime. This allows for the creation of highly specialized kernels by incorporating compile-time information directly into the generated code.

- Automatic Vectorization: By simply vectorizing the inputs of a CubeCL function, we can determine the vectorization factor of each intermediate variable during the expansion.

- Rust Integration: The generated code remains valid Rust code, allowing it to be bundled without any dependency on the specific runtime.

Our goal extends beyond providing an optimized compute language; we aim to develop an ecosystem of high-performance and scientific computing in Rust.For now we have highly optimized matrix multiplication kernels, leveraging Tensor Cores on NVIDIA's hardware when available. We are going to focus on adding more algorithms, but community contributions are more than welcome. There is still a lot of work to be done!

Don't hesitate to check the repo and ask any questions that come to mind.

Happy to share what we have been working on lately. The blog post explores Burn's tensor operation stream strategy, optimizing models through an eager API by creating custom kernels with fused operations. Our custom GELU experiment reveal a remarkable improvement of up to 78 times on our WGPU backend.

This release is packed with new features and improved performance, but the major focus was on enhancing the user API and the documentation.

We updated the API to remove instances where the device chosen was the default one, potentially causing bugs due to device mismatch. Now, our API is more explicit about where the device should be specified, aligning well with the Rust philosophy.

The book has seen various updates, including a new section on dataset manipulation requested by the community. We also plan to create a contributor guide, to help new contributors get familiar with the internals of the project.

A lot of work has been done to improve our JIT compiler, where we can fuse WebGPU tensor operations into a single kernel for impressive performance improvement. We added automatic vectorization of element-wise operations, as well as integration with autotune. Additionally, kernels created on-the-fly can now be executed in-place for reduced memory usage. We now support running multiple optimization streams independantly, which helps when metric updates and training run on the same device, but different threads. This feature isn't enabled by default yet, but you can enable it with a backend decorator. Future releases will add more optimizations to the compiler, and we will probably ship it by default. We also have plans to add other compilation targets in addition to WebGPU, namely Vulkan and CUDA.

One of the major quality of life improvements is the addition of the new PyTorch recorder that allows loading PyTorch weights into Burn modules. We also support specifying regex to dynamically map the weights to your Burn model if the structure isn't the same as the PyTorch implementation.

With this new release, we spent a lot of time solidifying our infrastructure, testing our framework on additional OS (Windows and MacOS). Overall, our CI is more mature and allows us to more easily ensure the quality and correctness of every code change across backends and operating systems.

Release Notes: https://github.com/tracel-ai/burn/releases/tag/v0.12.0 Burn Book: https://burn.dev/book/

When developing our WebGPU backend for the Burn deep learning framework, we faced numerous challenges in optimizing the execution speed. Autotune serves as our solution to the challenge of selecting the most efficient kernel for GPU operations, taking into account factors such as device specifications and input shapes. A kernel is an algorithm that accomplishes a task of relative simplicity, normally in the hot loop of an AI model.

In our most recent blog post, we explore two different operations commonly found in almost all deep learning models: Reduction and Matrix Multiplication (Matmul). The post highlights the difficulties of manually choosing the right kernel, given the uncertainties arising from the hardware and various input shapes and strides.

For instance, in some scenarios, our first reduction algorithm can be 3X as fast as our second one, but in other scenarios, it can be 19X slower, highlighting the importance of selecting the right kernel for the job. For Matmul, often we have the best kernel performing 3X as fast as the worse one, but the fastest one changes constantly across a spectrum of scenarios.

We hope the post highlights our flexible solution to this problem and how it can support our mission of creating the fastest framework on all hardware.

We've been working hard since the last release to craft a book for the community. It will help you get started with the framework while highlighting its special characteristics.

Let us know what you think of it and how we may improve it further.

Happy reading