When I got my first real programming job at a games company in the early 2000s, the Technical Director of the project once gave me some advice: if the number of things you are working with is on the order of 10,000 then don't bother optimizing it. Given the increase in computer power in the last 20 years I believe bumping that to 100,000 is pretty appropriate.