There are some downsides of perfect pitch. For one, choral singers with perfect pitch sometimes have a hard time re-tuning if the rest of the choir shifts pitch. (This can happen naturally over a long piece that has no accompaniment.)
Another downside is that almost all people with perfect pitch lose it when they hit a certain age. Imagine being an accomplished painter. One day, you wake up and see leaves as blue-ish green instead of green. It might be difficult to adjust to no longer being able to see the world in color.