Seeing what you made get to the point that that's an issue is a "a good problem". Yes, it's worth it to optimize at that point, but that's still an argument for what I consider premature optimization.