American system feels more unfair when you're given points for extracurriculars like playing instruments or sports, like that's not going to hold poorer children even more (also how's that related to academic performance at all? Unis should not care about unrelated things)
I probably agree with that, but also acknowledge there's no good way to make that completely objective.
Even worse, rich kids have far more means to engage in extracurriculars than poor kids.
The large campus-style uni is fairly recent creation - many came out of the land grant system during/after the Civil War. And even as newer unis have been created, they've followed that general design (even though they aren't land grant institutions).
US universities do care about extracurriculars and GPA and other things because they aren’t optimizing for raw academic performance, they’re optimizing for various other things like an interesting student body (that attracts donors, professors, and future students), real-world networks, and so on.