But sometimes when I really torture my brain I can spend a few days of doing mathematical proofs etc to come up with a even simpler solution.

That extra effort is only sometimes necessary. (But can be lots of fun to develop.)

Why? It is so easy, just think of the work being done and pick the big parts, those are the bottlenecks.

Only reason I can see anyone fail at that is that they don't know how long things take in their programming language, but that takes very little time to learn, and once learned you will know the bottlenecks before writing.

In so many cases the bottleneck is using bad data structures everywhere, that often gets you 100x runtime and doesn't show up in a profiler because it is spread out all over the codebase, that is the real bottleneck that never gets fixed and is why programs are slow today. To fix that you have to learn to know the bottlenecks as you write the program and not rely on a profiler. Profilers helps you find how long things take, they are really bad at helping you understand how to make the program fast.

Some view simplicity more as minimizing lines of code, e.g., less moving parts.

I view simplicity more as increasing lines of code, the goal being to make the code very verbose. Sometimes this means more moving parts, but smaller "movement" at each step.

There are other views of simplicity as well.

But sometimes you already have something that works, is correct and fast, but you still want to simplify: for example, when understanding _why_ that code is correct is too annoyingly complicated to explain and understand.