clisp is packaged for Ubuntu LTS, I haven't complied anything with it and your make file might need to be checked for compatibility, but it could be an option
Ubuntu is a nonstarter. Because there are many "clisp"s, what's the precise name of the package? This determines which codebase was used, i.e., GNU clisp. And, are you certain it works on arm64? (GNU clisp should by virtue of being a slower VM byte interpreter)
I spent time in gentoo myself, and know what you mean about packages, but the package name in Apt is ‘clisp’ and I double checked by pulling and firing up the repl in Ubuntu 24.04 on a Pi4.