Take for example a big and slow Python/Ruby/PHP/whatever web app, something that I'm sure a lot of people on HN have experience with. If you profile one of these apps you're probably going to find that it is slow because it's using an ORM that generates hundreds of SQL queries, the ORM creates tons of intermediate objects which put a lot of stress on the memory allocator and GC, and there will be an endless amount of code that is munging data to take it from one representation (e.g. whatever the ORM returned) to some slightly different representation (whatever the caller actually wanted). There might be a few queries that are particularly expensive, but once you've fixed those you're still going to be left with something big and slow with no obvious path forward to make the code faster. Furthermore the root cause of these problems may not be obvious when looking at profiles because things like memory allocation that are internal to the runtime of your interpreter are typically either not exposed by profiling tools, or if they are it's not clear what action can be taken to improve them.
Likewise if you have a C++ program that does a lot of unnecessary copying and memory allocation, the program is going to be slow because there's a lot of time spent everywhere and there's no one thing to fix. Look at a bottom up profile of a C++ program and see how much time is spent in string constructors, memcpy, etc. and unless you've been thoughtful about this stuff from the start what you find is probably going to be alarming. In fact, since a lot of copy constructors are inlined (including for STL types), with many profiling tools it might not even be obvious that copies are happening at all, since they won't show up in call stacks.
Not every program needs to be high performance, but if you start by writing a lot of code that is slightly inefficient everywhere then it's probably going to be impossible to make things fast if you change your mind down the line.