I don't think that's quite right. As I understand the law (and IANAL), you could still get their patent invalidated for lack of novelty — you just couldn't claim the patent yourself in that case.
And even if you didn't derive your work from theirs, you still have a problem with prior art.
First to file only means that, if you filed for a patent on some thing after someone else who also filed to patent it, you definitely don't get the patent (in a first-to-invent system, whichever of you can show the oldest documentation of invention gets the patent).