You're right. We should stop calling anything a button which is not used to fasten cloth via a buttonhole, even if the entire English-speaking world uses the term euphemistically to refer to round things.
Perhaps my English is letting me down, but besides clothing buttons (and also the US word for what British English calls a "badge") I've never heard "button" used for anything that you can't communicate with by pushing it.
In the US, of course, there are other buttons than push-buttons and clothing-buttons. Belly buttons for example. "Pin-back buttons" are usually just called buttons. Cornflowers are sometimes known as "Bachelor's button." You can of course use the term as an adjective -- button mushrooms, a button nose -- and you can refer to that indented mat that stops you from falling in the shower as a "button mat." If I described a "button mouse" to you, as opposed to a mouse button, you would probably come up with a visual image of a trackpoint mouse -- though you might also imagine a normal mouse covered with too many interaction buttons. The word "button" can also be used for things which are just generally small and globular and attached to other things -- you can find many search results for "button of rubber" and somewhat fewer for "button of glass" which fit this description.
Or the knobs on amplifiers, or door knobs - well I suppose door handle works but door knobs are distinctly different from handles in my mind.
I suppose I could see myself calling the temperature setting of a cooker the "Temperature dial", but not the mode selector knob thing. Mode/Temperature selector is quite a good one actually. Still doesn't come to hand like knob does though.