There are many Lisps, some of them rank as arguably the fastest dynamic programming languages there are. They can even compete with statically typed languages in performance. So you should expect Java level performance to be possible on many Lisps. In the 1950s, Lisps were considered too slow in comparison to Fortran and C. You have to think though, (as I explain later), Lisp is arguably the first higher level language. In the 1950s, Java, Python, C#, Ruby, JavaScript, Haskell, F#, would all have been considered too slow also if they existed. Lisps compete with Java, C#, Haskell, etc. in terms of performance.
There's now also more archaic Lisps that don't use a GC, and compile to very efficient machine code, that could compete with C speeds.
Most Lisps actually have too many functions. Not that its bad, but I think you got confused with they require very little primitive functions to mean there standard libs to be small.
On to why, it's hard to explain, but it's arguably the greatest syntax ever designed. In that it's simple and consistent, yet can express all things and easily be extended. You really have to try it to understand though, and give it a good 30 days. It's only once you're familiar with it and past the initial hump that you understand its appeal.
Finally, I'll explain the historical reasons people like Lisps. Its because they pioneered garbage collection, conditionals, closures, multiple inheritance, mixins, dynamic typing, macros, meta-programming, functional programming and REPLs (aka programming prompts). They were also the second OOP language, some people say to this day, only Smalltalk and some Lisps are truly OOP. So there's a lot of love due to its legacy of everything it gave us.
They're now being rediscovered, and because most software can now be successful even at Java's performance and memory usage, Lisps are making a come back. Since they no longer have any defeating drawback.