It's not just tax optimization. One of the key differences between the two are that buybacks reduce the supply of the stock. As there are now more buybacks than there are new stock issuance in dollars, this means that stocks are a depreciating asset. Any argument made of depreciating currencies like the during the gold standard or Bitcoin today is equally valid towards stocks when buybacks are allowed.
> doing investments with the money
They are induced into doing so. If their competitor is doing tax arbitrage, they have to in order to compete. It's actually much worse than it seems, as investors mostly invest through index funds now. If you buyback your stock, this increases your market cap and more index funds will buy your stock, pushing prices up further. If you don't buyback your own stock with debt, the index funds will sell your stock to buy the stock of companies that do. Virtuous and vicious cycles that end with only buyback companies surviving.
The volume of buybacks and indexing is such that smart money who calculate companies values through what they actually do has no chance to move the needle, and thus smart money is reduced into betting which company will do buybacks. There's no price discovery anymore.
> they have borrowed
This is just interest rate arbitrage. The Fed has set rates at an artificially low number, and as such there are benefits to these corporations taking out as much as they can to simply buyback their stock.