Is it a breaking change if a button is moved by a few pixels in a graphical application?
On the UI front it really depends, you need to make decisions and let users know to avoid surprises.
At Sun the process was that every possible interface that something or someone can depend on is acknowledged to be an interface. Is GUI pixel positioning an interface? Well yes, clearly it is as someone could write tools that rely on that position.
But do you as the author want to give users an assurance that you won't break this except in major releases? Maybe not. So interfaces were classified on the axis of public/private and stable/unstable (it was a bit more complex but that's the gist). Most applications used to classify the GUI elements as an unstable interface. Then you document that so consumers of the application know they shouldn't rely on GUI elements not moving.
This comes up in the real world sometimes such as with automatic testing scripts that use absolute cursor positioning, as well as in some kinds of compliance where any graphic change requires a compliance verification of the new UI.
The point is that most patch upgrades can just be run through a test suite and accepted. That may still lead to breakages, something that seemed minor and non-breaking to the developer could instead be breaking for a given user, but the point of semver is to convey actionable meaning and intent, which it does.