clamp the metal firmly between a couple chunks of 1in thick plank. supports the bit and fills the flutes of the bit with wood chips, which also helps prevent this problem.
if your need for precision is such that this much out of round is a problem, there's probably better ways of machining your hole anyhow. sometimes ya just gotta get it done tho.