> I find it helpful also not just because of the startup speed, but because I usually have one or more org buffers open taking notes on what I'm doing.

As someone who lives in org, this is the way. I'm still shell bound for a lot of things, but M-x shell is far superior, there's just no contest. Only thing that is annoying is stupid things like git diff that think they know better than you and want to do their own paging resulting in "WARNING: terminal is not fully functional". Thankfully I use EGG, but when I forget (and other things do it too), it's annoying.