Only a few have been seen with the human eye. The majority are seen via instruments. They know they're the same type of aircraft because their flight styles (or rather, the lack thereof) are identical to the ones seen in person.
They were largely undetected until the Navy rolled out some more advanced sensors/instruments relatively recently, and now they see "flocks" of them flying in formation, or sometimes individuals stationary and unaffected by wind patterns.
Is it aliens? Birds? Foreign drones? Experimental drones from higher echelons of our own government? Nobody knows. That's why it's actually interesting. Because even the Navy can't say it has an answer. But almost nobody seems to be interested in finding out, for some reason, despite them routinely flying in our restricted airspace without permission.