Hacker News Reader

It's only ~1000 LoC, seems like a pretty good case study to port over to other runtimes and show they can stand up to CUDA.

As far as I can tell, its optional dependency is Open MP, not CUDA. Doesn't seem directly dependent on CUDA.

Yes, a quick skim of the code only shows openmp dependency. The C/CUDA reference might have meant to be C/OMP .

Although I wonder if it would work well with GCC PTX OMP offloading.

The plan is to eventually implement with CUDA:

"Currently, I am working on [...] direct CUDA implementation, which will be significantly faster and probably come close to PyTorch."

Are there any strong LLMs trained without CUDA?

Yes, there are several. See this blog post from Databricks describing the landscape of LLMs trained on AMD hardware for example: https://www.databricks.com/blog/training-llms-scale-amd-mi25...

The most interesting one IMO is OLMo from AI2, which is truly open. You can read their blog post about it (https://blog.allenai.org/hello-olmo-a-truly-open-llm-43f7e73...) but basically it is open everything - they released everything you need to reproduce their weights (training data, training code, evaluation code, and weights) with a friendly (Apache) license.

Gemini and Gemma were trained on Google's TPUs.

Looking forward to your patches!