You are losing information. Knowing if you need that information or not is sometimes difficult. That's the whole lottery hypothesis (mentioned elsewhere).

>What are the drawbacks of quantizing though?

A 0.01% loss in quality for a 4x speed up and 4x less VRAM/RAM requirement.

90GB models now fit and run on a $600 consumer video card with quality so similar the difference is only detectable on hours long automated tests with tens of thousands of iterations.