The one caveat this article doesn't mention is worse handling but they replied in the comments about it. And handling matters way more at higher speeds, plus they commented the extra weight matters more for steep climbs too.
Aerodynamics are the only real reason, I think.