Frankly, it seems to me that if you're worrying about the intentions of your CPU manufacturer, you're screwed anyway. I guess that I also haven't put that much thought into it though.
This paper explains how to weaken the intel RNG by modifying only one logic gate and still make it passes the integrity tests.
I think not trusting a hardware RNG is a valid concern. As a consumer, it might not be a big deal, but what if you're buying CPUs for use in top-secret government machines? Given Intel's interests, it may likely comply with a backdoor request depending on the target.
One of the areas I'd like to look into is hardware integrity checking. It's most likely impossible, but it's worth a shot.
Ideally it should be private and contain some amount of entropy.
I've had the exact same idea, derived from a machine to roll dice.
I was going to use a simple motor rotating with a cam, beneath a die in a small enclosed box. Stops after N cam rolls, snaps a picture of the die face, bit of OCR and continues. You could generate quite a few random numbers using say two 32 side dies. A simpler method with less moving parts would be what you've described. I remember playing this game with a dice enclosed with a spring. Press it down and the spring bent, popped back and the dice rolled.
Practically: lots of cryptographic protocols just ask for a source of randomness, but don't care where it's from.
Also quantum 'randomness': https://qrng.anu.edu.au