I've watched first hand how much QWERTY-labeled keys with Colemak layout confuses people. Dvorak people usually figure out things are weird in one or two keys, but Colemak intentionally has enough coincidences with QWERTY that it can take people a while to figure it out depending on what they are trying to type, and the frustration is possibly more rewarding to watch as they try to get grips with how "broken" your keyboard must be.

I have considered doing this to even further mess with my partner.

He gets annoyed with my unlabeled keys and adding Dvorak to the mix would remove any ability to use my computer (not that I care, I am just a troll)