That's true in practice, but technically speaking it's false. You could imagine a contrived system which is not Turing complete but still contains a (rather useless) primitive that causes an infinite loop.
Non-Turing complete doesn't mean that you can't create too high complexity algorithms out of O(1) primitives.
Did you mean to reply to a different comment? You seem to be restating the correct part of orthoxerox's comment. I was merely refuting the incorrect part.
Arguably that would still terminate with an error message.