With businesses, investing borrowed money can help the company grow, and give returns greater than the interest on the debt.
With countries, the theory goes, a little debt means other countries have a vested interest in your success or failure. For example, why hasn't China invaded the USA yet? Perhaps they are waiting until they get their ~$4T back! :)