Indoor is always higher ppm (how much depends on many parameters) without proper ventilation. „Proper“ should include a „Heat exchanger“ thus you don’t need to reheat fresh air.
If your room has 2 times the open air concentration, and you are concerned if it's 2.0 times or 2.2 times, you should already be dealing with the problem.
So at 500 external you'd pretty much need continuous ERV but not necessarily scrubbers just yet.