I do this, too. The first thing I do when I get a new keyboard is to remove the caps lock key. I never used it in my 20 years of PC usages and I don't get why it's still there.
Back in my CounterStrike gaming days I would also remove the Windows key because it would crash the game when accidentally pressed.