The biggest problem with airless tyres is that bumps on the road cause localised pressure changes whilst pneumatic tyres distribute those pressure changes around the wheel. In practise this means that you feel more of the bumps and for big potholes you can end up damaging the rim. There's also the problem of energy loss as part of the tyre is compressed - with pneumatic tyres you only really have the outside of the tyre that is deforming and so they have less rolling resistance.
When I tried airless tyres (a long time ago though) the ride felt very "dead" and I haven't bothered with them since.
Tubeless however - that's the best ride performance that I've tried and you very rarely get punctures that the sealant doesn't fix.
The actual weight of the tyres won't make a huge amount of difference (i.e. the rotational mass) as it only really affects accelerating which is much less common than simply maintaining your speed.