That's true in practice, but technically speaking it's false. You could imagine a contrived system which is not Turing complete but still contains a (rather useless) primitive that causes an infinite loop.
Non-Turing complete doesn't mean that you can't create too high complexity algorithms out of O(1) primitives.
Did you mean to reply to a different comment? You seem to be restating the correct part of orthoxerox's comment. I was merely refuting the incorrect part.
Arguably that would still terminate with an error message.
I don't think a Turing-incomplete language implies that programs always terminate. As a counter-example, imagine an obviously Turing-incomplete language that contains only a single instruction which simply does nothing in an infinite loop.