It's not that bad, there are other languages which have this problem - you can see exactly the same situation in JS-land, for example. It would be much worse if not for quicklisp. I think quicklisp and asdf are among the best package distribution tools: I feel it's better than pip and gem and similar to npm. Installing a library is literally a single line in your code. There's the problem with finding the package you need and it's harder than with JS and npm, but doable.
BTW: what do you do to learn CL? I'm using StumpWM as my window manager and I think it's one of the best ways of learning CL. With SLIME, then, you get almost Lisp-only environment, especially if you use Emacs for your other editing too. I wrote a couple of blog posts[1] about StumpWM and posted my config on GitHub[2].
[1] https://klibert.pl/posts/stumpwm.html https://klibert.pl/output/stumpwm-better-modeline.html