The most critical thing is avoiding unnecessary work — that usually means estimating algorithmic complexity and trying to reduce it where possible (e.g. by sorting + binary search, caching, using basic data structures like a priority queue, etc.). I'm meaning to make a long blog post about my approach to algorithms, but meanwhile you can check out my recent slide deck on the matter:

https://speakerdeck.com/mourner/fast-by-default-everyday-alg...