Being a startup in Australia is hard, but being a startup is hard anywhere. I think there is a certain mentality amongst Australian startups that the US is the land of milk and honey for startups and Australia sucks. There is a certain element of truth in that, but its not as bad as some of the posts to HN make it out to be.
First Tax Breaks: The R&D tax incentive is amazing (http://www.ausindustry.gov.au/programs/innovation-rd/RD-TaxI...). We have recovered around 40% of our seed funding as a tax refund from the ATO. If you do something slightly innovative and can prove that you do some testing, you qualify. This tax incentive has directly allowed us to put on another developer.
GST: Yup GST is a pain but at least its uniformly applied to the whole country. The US you have to apply it on a state by state basis (depending on where you are located etc and the nature of the good/service you provide). Other countries have a consumption tax. Seriously meeting your tax obligations is a requirement of doing business anywhere, its not that hard to do: if(countryCode == "AU") {//apply GST... }.
Venture Capital: Nothing can compare to Silicon Valley in terms of money available for the tech space, but at the same time Australian VCs are often happy with a 10x exit as opposed to pushing you to be either a $1 billion company or go bust. The choice of where you raise will be a personal choice more likely influenced about who you know. Remember the higher the valuation you get, the bigger the exit you need to make.
Australia being expensive: Sure you pay a little bit more for your mac book pro, but the guy using it will be seriously cheaper than someone in the valley. You think Australia is competitive for developers who could get a cushy contracting rate. Try competing with facebook, google etc for the top talent.
Internet: Ok the current government hates in the NBN, it sucks, I love fast broadband as much as the next person, but unless your startup is about delivering high bandwidth content to Australians who cares? Your servers are going to be sitting on AWS in US_EAST anyway.
Mind you having said that I do agree with some points. Employee Share Schemes are a pain in the arse and getting a good payment provider is tough. Seriously spend the cash to get a Delaware C-Corp and open a US bank account... this will make your life so much easier. Even Braintree Australia is a shadow of using Braintree from the US.