I like vi better than emacs in general (it probably has to do with my learning unix without X and basically vi was the only editor to use), but vi doesn't work as an IDE for Common Lisp.
Vial (http://common-lisp.net/project/vial/) was started with that goal, i.e., to provide SLIME-like functionality for vi but it's one of those still-in-progress (and may never actually release anything useful) projects.