I did some quick changes to your code (inlining, stack allocating) and got a further ~2x speedup which makes SBCL performance equivalent to Julia.
EDIT: I'm also curious how much using an optimized vector math library (e.g. sb-cga) would buy you instead of hand-rolling your own vector math. It would certainly be easier.