You're actually right high public debt could also lead to disinflation as the economy crumbles. Which economists say high public debt doesn't matter? Its mostly politicians. I've never heard a serious economist says it doesn't matter. There is consensus that above 100% debt to GDP is no good.
If public debt doesn't matter btw, why does private debt matter? Simply publicly print away all private debt and viola problem solved. In fact why can't we all be billionaires?