This is likely because you are running it in some random PWD that doesn't represent a bazel workspace. When running in a workspace the bazel daemon persists. Inside my workspace the bazelisk --help invocation needs just 30ms real time.
Running bazel outside of a bazel workspace is not a major use-case that needs to be fixed.