A lot of social democrats start out as libertarians and then move left once they see the bigger picture of power dynamics beyond the individual.
Meritocracy is not as good an idea as it seems to be in practice.
Why not just consider that it is a label which may not apply to the reality you observe? Have we gotten to the point where considering that any X is not Y is considered a "no true Scotsman" fallacy?