You should definitely have a decent amount of savings when you move over though, to offset the lack of credit.
E3 is a nice idea but dealing with some of life's basics can be grueling in the US if you're off the system. It's not difficult to imagine how people on the street here can't get out of the hole.
Credit Unions are the way to go, no doubt, but took a while to figure that out.
The trick is that it's a non-resident-intent visa, which is to say, while it's indefinitely renewable on paper, at some point USCIS is going to point out that you've been here for an awfully long time and you risk the renewal not going through.
Not sure if the same applies to E-3.
I do know of an E-3 holder who had their residency fast tracked, but there have been so few of them issued that it's probably a bit hard to tell the lay of the land.