But if you do not take care the "more important things" will have to be refactored or even rewritten allo over because you build them on a poorly performing base.

Finding the bottlenecks should always be a priority. Fixing them can be done at a later point.