Yes that's correct. Although, if you're working remotely you're not likely to be making much over $100K, so it may still be worth it. Your take home may still be close to working in the Valley, and your cost of living, excluding food will be less than half.

I'm Canadian though, so I don't have to deal with the long-armed-fingers-in-my-pockets embrace of Uncle Sam. Every other country in the world is sensible enough to only tax residents.