I agree with you. A good example of this is what we experience when driving. If someone cuts me off, my instinctive reaction is to think, "What a moron!" Which is unfair, of course: it may have been a one-time stupid mistake.
Even if you think you see a pattern, someone who seems stupid at something may just be outside of their comfort zone/talent area. In another field, they may be quite smart.