The time in the device isn’t independent of the received signal. The device typically has a clock that is much less accurate than the timesignal that gets set based on the gps signal. There aren’t many devices that can rival a gps satellite for time accuracy. A spoofed gps signal would not differ enough in time from a real one to tip you off.

Further, a big part of the working of GPS is that your location is trilaterated from discrepancies in time from the various signals due to time of flight.

The only way to accurately detect a spoofed signal is if you already know your location. (You can infer spoofing using IMUs, or signal analysis, etc… but that is a rare capability at this point)