Honestly, the elites are categorically better, on average. Not flawless by any mean but in many ways superior.
The Victorian elites had better education, better diet, better lifestyles, better care, etc... So yes, it makes for smarter and healthier people.
If you mean in term of human qualities like morals, kindness, ethics, etc... This is complicated. Many of the elites also tend to be better in that regard too, with a very big caveat. The reason why they are better is simple: it is easier to be generous when you are not in the need, education can play a part too but it can go both ways.
The thing that make elites worse is mostly the consequence of their actions, the "with great power comes great responsibilities" thing. What is an inconsequential for the poor can have huge implication when the rich do it. If you decide to stop buying bread from one bakery will not impact that bakery too much, they loose and gain customers all the time. If you have a restaurant chain and decide to stop working with a baker, you may have ruined him.
That's when things become more complicated, sometimes the elites really are terrible to the people under them, sometimes, they are as good as humans can reasonably be, noblesse oblige. In history, it is not uncommon, when there is a revolution and power is given back to the people to end up very badly.
I don't know how Victorian elites sit on this scale, certainly not all bad, certainly not all good (especially to the colonies), but where exactly is hard to say.