I wish I had a reference for you, but I recall reading some UI guidelines that discussed how long you have before you need to give feedback to your users. IIRC 2 seconds was the longest amount of time that didn't feel like you were waiting forever. Once you got past 6 seconds, basically people think it is broken.
I have the same experience as you do, and these numbers of 2 seconds and 6 seconds really seem about right. If the test runs in less than 2 seconds, I can stay focussed on what I'm doing. By 6 seconds, I'm already thinking about something else. My personal experience has been that if the tests takes longer than 2 seconds, it is virtually impossible to do TDD. Instead you have to write large chunks of code -- or at the very least write several tests in one go.
I'm fairly well known in my current group for the 2 second rule :-) 2 seconds is really a long time for a modern computer, so my feeling is that if you can't build and run a large section of tests in 2 seconds, you've got a serious problem somewhere. When pairing with people who don't mind longer test runs, I've noticed that they never actually TDD. Instead they right large pieces of code (entire functions at the very least), or they write 5-10 tests all in one go. When I show people the normal rhythm of TDD, they suddenly understand why you need to have incredibly fast tests.
I also preach a 2 minute limit for the entire test suite. Kent Beck has talked about that previously (although I can't remember what number he used -- something like 2-4 minutes). He has even advocated simply throwing away long tests. Test coverage is less important than being able to run your tests quickly. I have found that if the test suite creeps up to about 10 minutes, the frequency of running that test suite is reduced to about once per day -- meaning that you have totally lost the context of the problem when you discover it.
Whatever you need to do to keep those tests running quickly is what you should do. Whether that means only running a subset, hot loading modules, stubbing out slow moving pieces (even the database), etc. I could write for quite a long time on testing strategy... This needs to go on my queue for a blog post ;-)