Yes you can get bottled LPG gas, but it is a pain – you worry about running out, you have to call up and get them to deliver it. With piped gas you just set up a direct debit for the bill and never think about it ever again.
Also, some people prefer gas heaters because it is a different kind of heat. My father replaced his wood combustion heater with a gas one, even though he already had reverse cycle air-conditioning. Maybe just using the air conditioner would have been cheaper (he already has rooftop solar–I don't think he has a battery, but could easily put one in), but it wasn't what he wanted.