The Nazis went from a fringe political movement to a solid minority voice (with 20% of the seats and about 33% of the vote). Then came the Reichstag Fire and heavy repression of opposition - getting them to 44% of votes in a “free” election. And then came the Enabling Act and there were no more votes.
What this saying means ultimately is that enabling fascism via silence makes people anti-democratic. It does not take many people passively sitting out of the way for authoritarian regimes to take power - and they are only removed by the bullet box not the ballot box.
I will note I can’t find the providence for this saying actually existing in post WW2 Germany and it is likely a modern invention. It is nonetheless a true sentiment of some of the soul searching German and other fascist ruled societies made after WW2.