> If inflation hits 4% and your mortgage is 3.5% you would be making 0.5% a year off the mortgage interest
This ignores the rates-price and inflation-rates nexuses. When inflation goes up, ceteris paribus, rates rise. When rates rise, all else held equal, the price of leveraged assets like real estate falls.
TL; DR this trade is an arbitrage only if you've perfectly hedged the price of your underlying asset, the house.