The test positivity in lightly-affected areas is very small, for example <0.1% in much of Australia. This includes both true and false positives, so it bounds the specificity of the test.

https://covidlive.com.au/report/positive-test-rate

A Lancet article cited below references a paper that estimates much worse specificity of PCR tests for SARS-CoV-2 based on the specificity of tests for other RNA viruses (original SARS, MERS, influenza, etc.):

https://www.medrxiv.org/content/10.1101/2020.04.26.20080911v...

But why would anyone do that, when we have direct experience with the actual test for the actual virus?

Of course the false positive rate in heavily-affected areas may be higher, due to sample contamination--with more true positive patients, there's more opportunity for one true-positive patient to contaminate multiple samples. But such cross-contamination can occur frequently only if there are actually lots of true positive patients.

For another sanity check, the sensitivity of antibody tests is established on a sample of patients who tested positive by PCR, who are assumed to all be true-positive. So any PCR false-positive among those patients will show up as an antibody false-negative, so that bounds the apparent sensitivity of those antibody tests. But antibody tests show sensitivity around 85%:

https://journals.plos.org/plosone/article?id=10.1371/journal...

So even if the antibody tests were perfect, at most 15% of their positive sample was false-positive. (Note that this is a different quantity from the PCR specificity, and can't be translated into such without knowing the true prevalence in the population that the sample was drawn from.) Of course the antibody tests also aren't perfect, and I'd guess that 15% is almost entirely antibody false-negatives rather than PCR false-positives.

You can also compare reported COVID deaths (which are generally established by PCR) with year-over-year excess mortality. The agreement isn't perfect, but it's pretty close.

The vast majority of patients who test positive by PCR are or were infected by the virus. That's a different question from whether a positive test indicates the presence of replication-competent virus (i.e., whether the patient could infect others right now), since RNA from dead virus can trigger the positive too; but that dead virus had to come from somewhere.