The reason that Yosemite/Mavericks stall for a long time (although they inform users wrongly that there would be few minutes left, causing panic) is due to the copying and moving things in /usr/local/. If you have a huge /usr/local, it will take hours. Chances are you will need to remove /usr/local/Cellars and reinstall home-brew anyway. So, removing it first will save you some time.
Yup, ran into that experience myself. Viewing the log helped me see the install hadn't busted, because it also gave me the beach ball of doom around the same time. Everything was alright in the end.
I can confirm this. The installer is currently (very slowly) copying /Library/Ruby/Gems. "About a minute remaining" for ~30 mins so far.