The only downside would be time.Time structures grow by 8 bytes, but this is really minor because if you're storing times you're likely storing timestamps and not time.Time structures.
>On 64-bit systems, there is a 32-bit padding gap between nsec and loc in the current representation, which the new representation fills, keeping the overall struct size at 24 bytes. On 32-bit systems, there is no such gap, and the overall struct size grows from 16 to 20 bytes.