Hacker News Reader

Comment by gryson | Hacker News Reader

"We recruited participants by placing targeted advertisements on Facebook aimed at residents of Santa Clara County..."

I wonder if these ads mentioned the purpose of the study, or if that information was only given out after initial contact (I don't know enough about IRB requirements here).

A concern is that this could cause a selection bias for people who suspected they had the virus. People may respond to such an ad out of curiosity ("I think I had the virus, so it would be good to know for sure") or obligation ("I'm pretty sure I had the virus, so I should help out with this study").

It wouldn't have to be a large selection bias, either. Of the 3,330 people they tested, they found only 50 who tested positive.

I would like to see a bit of a better method of sample selection before drawing any conclusions.

View on HN

Yes, the ads accurately described the study. Several friends of mine participated (I live in Santa Clara County).

Of course there is selection bias, which is why they make adjustments to make the sample better represent the population (see the abstract for details on this, or read the paper for all the details, there's well-established science behind these adjustments and it's up to the reviewers to be sure that they did it correctly). The point is that "they only found 50" is quite a large number compared to the number of confirmed cases.

What adjustments were made to account for selection bias of the sort described? It's not covered in the paper, and "well-established science" certainly doesn't answer the question.

These are very common statistical methods when dealing with surveys and populations. It isn't this paper's job to describe the background on something like that, just like it isn't this papers job to explain blood testing.

They describe their limitations and adjustments exactly like most other studies of this sort do. For example:

"Those imbalances were partly addressed by weighting our sample population by zip code, race, and sex to match the county. "

They said that they collected prior clinical symptoms in the survey, but have not mentioned that they adjusted for those.

Given that, I think it's fair to say that they have not adjusted for the selection bias of people with prior clinical symptoms being more likely to click the ad.

On the contrary - detecting selection bias of this sort can be very difficult. Do you measure it using a questionnaire? You might ask your subjects "Do you think you previously had COVID-19?" But what do you do with that response data? Exclude everyone that says yes? Then you're skewing one way. There's obviously no population data to match that response with like there is for general demographics such as race and age.

Many studies are only as good as their sampling methods. A good sampling method will alleviate concerns of selection bias.

I think you'd call individuals(using a method with far less self selection bias) and get them to fill out the same questionnaire and then do the same types of adjustments you'd do with race or gender.

But I also agree they 100% did not do any of these adjustments if they did it would be listed right here

> We report the prevalence of antibodies to SARS-CoV-2 in a sample of 3,330 people, adjusting for zip code, sex, and race/ethnicity. We also adjust for test performance characteristics using 3 different estimates: (i) the test manufacturer's data, (ii) a sample of 37 positive and 30 negative controls tested at Stanford, and (iii) a combination of both.

Adjusting for selection bias is really hard and potentially first-order.

Correct. For some things you could use baseline data, but there is zero baseline data to use here.

That reweighting doesn't address the most troubling selection bias: people choosing to participate because they want to get tested (e.g., because they think they might have had the disease).

Why would this matter as long as they got a good geographical sampling?

"We used Facebook to quickly reach a large number of county residents and because it allows for granular targeting by zip code and sociodemographic characteristics. We used a combination of two targeting strategies: ads aimed at a representative population of the county by zip code, and specially targeted ads to balance our sample for under-represented zip codes. In addition, we capped registrations from overrepresented areas."

Update: Okay, "Other biases, such as bias favoring individuals in good health capable of attending our testing sites, or bias favoring those with prior COVID-like illnesses seeking antibody confirmation are also possible."

If part of the value for participants was to know whether they had antibodies, then those that had gone through a flu (or covid) would be more likely to respond. This means that the overall sample likely includes more people who had COVID than a true random sampling would.

It looks like there's an argument to be made it could overcount or undercount depending on importance of each potential bias.

"This study had several limitations. First, our sampling strategy selected for members of Santa Clara County with access to Facebook and a car to attend drive-through testing sites. This resulted in an overrepresentation of white women between the ages of 19 and 64, and an under-representation of Hispanic and Asian populations, relative to our community. Those imbalances were partly addressed by weighting our sample population by zip code, race, and sex to match the county. We did not account for age imbalance in our sample, and could not ascertain representativeness of SARS-CoV-2 antibodies in homeless populations. Other biases, such as bias favoring individuals in good health capable of attending our testing sites, or bias favoring those with prior COVID-like illnesses seeking antibody confirmation are also possible. The overall effect of such biases is hard to ascertain."

There is also selection bias in whom Facebook will show the ads to due to Facebook's CTR models, other bidders' bids, and who is browsing Facebook more to see more impressions in general. They should have worked with an ad platform to run the ad as a PSA so they could use the ad platform's own randomization instead of relying on having the auctions they win result in anywhere close to a usable sample.

In the data from Italy, ~14% of the >1M tests done for the virus (not antibodies) were positive. That prob includes ppl tested multiple times, or after recovery.

I wonder why this is so vastly different from the study above.

https://en.wikipedia.org/wiki/2020_coronavirus_pandemic_in_I...

Reply on news.ycombinator.com