Sadly, your physical presence makes a difference in how others start perceiving you and will probably dictate what approach you take to establish yourself. A lot of self-help leadership books seem to be written by guys who are at or over 6ft tall.
It doesn't mean you cannot be perceived as good leader without a leading man presence, but you do have to work at it differently. Advice for a classically dashing 6'2" guys often doesn't work for 5'4" women. You need to impress the people making that decisions and they often have the look. It sucks, but we cannot see other people's souls. Sorry for your loss, it sucks and is a bummer but don't let it break you, its just their opinion and probably isn't close to the truth.