You are right in a way, but you can't ignore the influence of powerful individuals on the society in general. Yes, there are many conservatives in Hungary, but it's a whole different level if the person representing them repeatedly breaks the law and pushes things much further.
An analogous situation is in Poland. The unofficial ruler of the country declared that Germany is the biggest enemy of Poland, playing on the feelings of the older generation, and is trying to build a narrative around that (the EU servers only Germany, hence EU = evil). That wasn't a narration among Polish conservatives a few years ago, it was constructed by a single person and spread by his minions and the media he is controlling. The biggest absurdity of the last days was when Germany proposed giving Poland Patriots, the Polish defense minister happily agreed as any normal person would, then the day after after the unofficial ruler got berserk he completely changed his opinion and refused to accept them (how can Germans protect Polish sky if we say Germany = Evil?). What a shitshow!