If you want to produce a standalone Lisp executable, you have to embed the compiler with it, hence the big size.
However, if you just run the programs with your own compiler, you only need the actual code — that is a 44 bytes "Hello world!", shebang included.
On my machine, the SBCL package is 70 MiB, while for example the Python one is 125 MiB.