I think once the IDE catches up to the level of Visual Studio or the Java stalwarts Eclipse/IntelliJ it will really take off. I usually do my stuff first in VS then bring it up in Monodevelop to finish it off and make sure I am mono safe.
I started with Visual Studio 6 (mostly using Visual C++, and VB6) and it was a speedy. Visual Studio 2003 was a little slower but not bad, 2005-2008 about the same. It was all bearable and acceptable.
Now Visual Studio 2010 is painful and awful. I've got a Core 2 Duo, 4 GB of RAM and the interface is slow, you can see the GUI repaint sometimes. Clicking on the interface causes frequent pauses that take close to 1 second to respond.
There's definitely a lot of work/polish that needs to go into MonoDevelop and it's something Xamarin knows about and is working hard to fix.