Portable LLM apps in Docker [video]

Docker is the leading solution for packaging and deploying portable applications. However, for AI and LLM workloads, Docker containers are often not portable due to the lack of GPU abstraction -- you will need a different container image for each GPU / driver combination. In some cases, the GPU is simply not accessible from inside containers. For example, the "impossible triangle of LLM app, Docker, and Mac GPU" refers to the lack of Mac GPU access from containers.

Docker is supporting the WebGPU API for container apps. It will allow any underlying GPU or accelerator hardware to be accessed through WebGPU. That means container apps just need to write to the WebGPU API and they will automatically become portable across all GPUs supported by Docker. However, asking developers to rewrite existing LLM apps, which use the CUDA or Metal or other GPU APIs, to WebGPU is a challenge.

LlamaEdge provides an ecosystem of portable AI / LLM apps and components that can run on multiple inference backends including the WebGPU. It supports any programming language that can be compiled into Wasm, such as Rust. Furthermore, LlamaEdge apps are lightweight and binary portable across different CPUs and OSes, making it an ideal runtime to embed into container images.