The latter should be “out of scale” (think 3.6 Roentgen…) “out of band” is quite literally outside the prescribed channel.
It's a totally valid and common jargon phrase from the web services world, apologies though, I assumed that it was wider jargon than it turns out it is...