I'd guess that they process Prime members shipping before the free shipping. And I'd suspect some of the waiting is time to ship from the manufacturers to Amazon's warehouses. I also wonder if they transfer long tail items between warehouses to consolidate orders. There is speculation that Amazon intentionally delays free shipping orders to push people towards Prime. Maybe they just have capacity problems?
By comparison with other vendors like Walmart I usually see shipping in less than 24 hours.
Amazon forum complaining about slow shipping, people getting Prime stuff a lot faster, no response from Amazon people:
http://www.amazon.com/Super-Slow-Shipping/forum/Fx4HQD057HT5...
Giving credit to Prime members to accept slower shipping:
http://www.geek.com/news/amazon-prime-customers-receiving-cr...
Another random example complaint:
http://www.pissedconsumer.com/reviews-by-company/amazon/very...