I wonder if that memory usage is with X running. If yes, then Im very impressed. 32MB with X is very decent result.
That being said, I suspect 64MB could be enough for console-mode work, and if you recompiled the kernel without any unused drivers, perhaps you could make it under 32MB.
Thats my Linux VM w/o X:
MiB Mem : 3028.4 total, 2945.5 free, 19.7 used, 63.2 buff/cache
Xserver is running on my main OS.