I'm torn both ways on the double issue. On the one hand, doubles are much more widely supported these days, and will save you from some common scenarios. Timestamps are a particular one, where a float will often degrade on a time scale that you care about, and doubles not. A double will also hold any int value without loss (on mainstream platforms), and has enough precision to allow staying in world coordinates for 3D geometry without introducing depth buffer problems.

OTOH, double precision is often just a panacea. If you don't know the precision requirements of your algorithm, how do you know that double precision will work either? Some types of errors will compound without anti-drifting protection in ways that are exponential, where the extra mantissa bits from a double will only get you a constant factor of additional time.

There are also current platforms where double will land you in very significant performance problems, not just a minor hit. GPUs are a particularly fun one -- there are currently popular GPUs where double precision math runs at 1/32 the rate of single precision.