Might not be as great as you seem to think, you'd be unflagging silence that you'd want cut anyway. I imagine you'd get false positive like with a speaker frowning and be silent for a few second because he's thinking.

Point is you'll still have to review manually the result anyway for under- and/or over-cuts.