one case is that you could have a bazel buildpack that contains bazel and java and all that stuff requires in order to run the build.

Unfortunately the resulting image would be larger than just a single binary, but building it would be a lot easier and repeatable for other people working on the project. The base layers would be cached though so in practice it might not be that much larger on disk.

I looked into using bazel for doing similar things, and the biggest stumbling block is that bazel itself is a PITA to install on all platforms. May end up trying https://please.build/ at some point.