It's even less, as you can see 18 of those 32MB are cached stuff, but this is without X.
That being said, I suspect 64MB could be enough for console-mode work, and if you recompiled the kernel without any unused drivers, perhaps you could make it under 32MB.
Thats my Linux VM w/o X:
MiB Mem : 3028.4 total, 2945.5 free, 19.7 used, 63.2 buff/cache
Xserver is running on my main OS.