You only need physics to run at a "fast enough" rate. Let me rephrase:

Why not just snap physics frames to graphics frames?