The pip ecosystem (including uv) doesn't encourage this. Although you could and many have, the build tools don't support it as first class.

For why? Ask what happens if you want an updated version or specific version of CUDA? The library author must add support for runtime linking or users would have rebuild the entire conda package.