I always thought of it more as a reference to the Gnomes of Zürich than to Jewish people.