<input type="number"> is just completely awful. To handle it properly in our React application, we have a <NumberInput> component which is roughly 200 lines (and shrinking, thankfully), involving more than a few browser-specific validity checks and keydown handlers.
The only way to keep non-numbers out of your number input is to cancel keydown events selectively, but it's still a crapshoot. And I've been bitten a few times by that bizarre change issue you mentioned.
If there's not already a good MIT-licensed project that encapsulates inputs and gets the weird out, I think it would be worth doing, whether in React or Web Components or both.