A "whole Docker container" is not very heavyweight. Other than having their own filesystem view and separate shared libraries, container processes are nearly as light as non-container processes. It's not like running a VM.
why does the smallest script take seconds to even start then?
Of course, if you're invoking it on a remote cluster, there are many reasons it can talk longer, especially if the cluster has to scale to satisfy the request. But all those reasons are nothing to do with containers specifically - it's things like cluster capacity, node provisioning time, container pull time, network latency, etc. If you architect things properly, you can get the number below + network latency even for a remote cluster invocation.
$ time docker run ubuntu echo hello world
hello world
real 0m0.328s
user 0m0.011s
sys 0m0.010s