Yes - unfortunately we have a lot of leftover usages of asbestos.
Historically, it was used a lot. My father even remembers playing around in the bush as a kid, and using asbestos for chalk a bunch of time to mark stuff on trees. My friends dad also remembers coming home covered in asbestos after work a bunch of times too.
I'm not sure I agree with your assessment about seeing asbestos in nice neighbourhoods. It depends how you define nice.
Asbestos was banned in 2003,but hasn't been used in housing since the 80s, so it's only going to be in older developments that you'll really encounter it.
Your wording regarding "plenty of Aussies" is also unclear to me. The numbers as a percentage are going to be very low, but it undoubtedly is a thing.