Just remember that there can be hundreds of bottlenecks. Your second slowest matters too. And sometimes there are different requirements - the UI needs to respond within 1/10th second (sometimes much faster, sometimes much slower). Users can often live with your main calculation taking 10x longer than the optimized version so long as there is fast feedback that you are working. Eventually they will want the calculations faster too, but taking a minuter off of those is less valuable than a few ms off of your UI feedback.

it's more nuanced:

you're still releasing resources - so you might not become faster overall but you can compute more in the same time later if necessity arises (althougth that might be somewhat premature but can be good for library code - so it becomes more applicable in different environments)

and there are some rare but tricky scenarios like:

hardware is mobile phone: app seem to be bottlenecked on arithmetics according to the profiler, so it feels obvious to start optimization there

in reality what happens - hardware has limit on power, so it can't give full power to CPU, GPU and memory all at the same time

since app uses too much memory - it has to redirect power there, also memory emits heat, so CPU becomes throttled

By optimizing memory, everything runs colder -> CPU gets more power, can run sustained higher frequencies - app becomes faster

Translation: I'm "The Boss" so it's not a bottleneck unless I say it is.

It's not true, though. Speedups are speedups even if there are still slow parts.