Yes, this isn’t a logical error at all. If you don’t have taste in one area — actually, it’s even worse, you’re not even aware of your own lack of taste — why would I trust your taste in another area?
It's like arguing you wouldn't trust a lawyer with a medical negligence case if they can't suture a wound.
Or you wouldn't trust a graphic designer with a restaurant logo if they can't make good scrambled eggs.