I think this discussion is typically fuzzy because people do not separate weight and size.
As for weight, no sane person wants to carry around more weight than needed. So there's a wide consensus that less weight is better. The counter point that extra weight helps with balance is a bit of a stretch. People often mention long heavy lenses but the truly heavy lenses you will not really support from the camera body anyway. Plus, we're not really talking about cameras where the "light" bodies really are that light as to make handling them unstable.
As for size, if we were to ignore weight for a second, I strongly believe a largish camera body is ergonomically ideal for almost everyone. These cameras are often portrayed to be "huge", but they really aren't as it comes to holding them. A large deep grip allows you to hold it well. A large back plate allows for lots of buttons and room between buttons.
I extend that remark to people with small hands. Even with small hands you can't fully "grip" a compact mirrorless camera. Don't get me wrong, for sure you can accustom yourself to deal with this compromise in ergonomics, I'm just saying that a largish size is the ergonomic ideal, if weight were not a factor.
Of course, the problem is that size and weight do correlate. So you can pick between a large camera that is ergonomically ideal to handle yet heavy to carry or a lighter camera with poor ergonomics.
In the end it's best to just try it. My main point is to keep an open mind towards largish cameras, they have some very strong pros.