If prior art doesn't matter, what stops anyone from patenting someone else's publicly known invention and then suing them?
Who is suggesting that prior art wouldn't/shouldn't matter?
"First to file" patents would still be invalidated by any published (or publicly available) prior art. It just means that you couldn't invalidate by demonstrating that you were privately working on the same idea.
http://news.ycombinator.com/item?id=2288824
edit: changed link to relevant comment