The highs and lows, thinking that you've found the perfect layout, only to encounter an edge case which there's no solution for and giving up. The continual wishful thinking that "one more modification" and THEN I would have the perfect layout, without ever knowing if that's true. And then when do you stop? I couldn't find a point where I would ever stop, it was maddening. I used a Japanese keyboard for a while [1] to leverage the extra keys on either side of space bar, but then I realized that I probably wouldn't be able to use a laptop normally ever again, which was enough to snap me back to QWERTY.
In the end, I did end up going back to QWERTY, but with some modifications, which I really think improve the problems I have with the layout (sore pinkies) [2], most commonly from jupyter notebooks (hitting shift-enter). It also solves emacs pinky.
[1] - https://www.amazon.com/SolidTek-Keyboard-Languages-JAPANESE-...