It's a bit more nuanced than that though. If someone breaks into your house and you ask the police to remove them you aren't a fascist. If Russia invades your country and you fight them for every inch, you aren't a fascist. If Romanis invade your city and threaten, assault, and rob your citizens and you remove them, you aren't a fascist.
The issue with the Romani is that they refuse to integrate. Or rather, the people we call the Romani are the people left after parts of their population DID integrate and now are just a part of society. It's selection really.
What do you do with a population like this that is actively belligerent? The Nazis solution was extermination, and I obviously think that's very immoral. Just letting them walk all over us is also immoral. Forced integration by breaking them up into individual families isolated from each other and performing cultural genocide? Also immoral.
This is a classic dilemma. There's no good option here. But pretending there isn't a dilemma is just fooling yourself.