I used Benford’s law to analyze Covid-19 in 113 countries (2020) | Hacker News Reader

I used Benford’s law to analyze Covid-19 in 113 countries (2020)

kevinbasset.medium.com

104 pointsjanandonly37 comments

This analysis doesn't make sense to me. China and other countries have had periods of double digit new cases reported. Even if that data is fake, you can't use Benford's law on numbers that small (it has to be numbers that span magnitudes... clearly states so in the wiki article he links). Also, cumulative stats aren't "naturally occurring random numbers."

In a country that hasn't done very much about the virus the cumulative stats will be distributed well enough. Note how all of the countries in the lower right part of the graph are the ones that managed to control it and thus pretty much freeze the cumulative number.

Interesting. This is something that I hoped someone would do. However, I think that for Benford's law to be meaningful, you probably need to be looking at the finest-grain stats that are available, not just the top-line national numbers.

For instance, to give the U.S. as an example, the national numbers are a sum of state numbers (plus territories), which are themselves in most cases probably a sum of numbers from counties, which are perhaps sums from individual hospitals.

For Benford's law to work well I think you have to be looking at the same level where the fake numbers were created. If you add up a whole lot of fake numbers or a combination of real and fake numbers, the frequency distribution of the most significant digit of the total will probably end up being about the same as real data. For example, if fake numbers were being generated at the state level in four states, and at the county level in 30 different counties, and at 20 hospitals, then the national numbers would probably still look fine.

If we don't know whether fake data might be injected at the national, state, or county level or from individual hospitals then we'd have to check all of them. And if the data at the granularity we need to be looking at just isn't available then we probably won't be able to detect it just by applying Benford's law.

But as you drill down in detail you run out of enough cases to measure properly.

Benford's law requires data that is distributed over a wide range--it's going to find problems with any data that is too granular. Hospital sizes aren't distributed over wide enough a range to apply Benford's law to.

This came up not too long ago in looking for voting irregularities--somebody who didn't understand what they were doing applied Benford's law to precinct-level voting data--and of course it failed horribly. Duh! Voting precincts have an optimum size and thus will be pretty clustered around that size. The percent going to each candidate generally won't vary too widely, either--thus giving a substantial clustering of the numbers voting for each candidate. There's nowhere near the range of data to apply Benford's law to.

Using Benford's law on cumulative numbers is a bit silly, misleading even as you are sort of implying that some of fraud is going on.

Suppose 5 people died of covid in the Vactican state before they were able to get it under control. The cummulative series is going to look something like this:

0 0 0 0 0 0 1 2 2 2 2 3 3 4 4 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 ...

5 being way overrepresented - without there being anything unnatural in it.

This is explicitly discussed in the article. Why are you calling it misleading, when they make the same point as you do, in almost the same way?

> For example, if like China, you eventually eradicate the virus when you have about 80,000 cases, you may end up with hundreds of days with values starting with 8.

The article concludes by pointing out that for this reason, cumulative numbers aren't as suspicious, but anomalous daily numbers are...curious.

You are right - it is actually mentioned in the article - sorry about that.

I would still say that it is somewhat silly to apply Benford's law here - also in the non-cumulative case.

The distribution of the daily number how much disease is circulating - where I am at the moment we have daily numbers stable at some 40-50 thousands, leading to 4 and 5 being overrepresented, without it being in any way being fraudulent or made up.

In several Countries, the tax office will use Benford's law to decide if you are "randomly selected" for an audit. And it's quite useful to detect scientific fraud, and has been used that way several times.

This is an interesting write-up, but would seem the diversity in both severity and duration of lock-downs, plus simple issues with quality of data collection mean here Benford's law anomalies, just help identify the lack of consistent approach between countries in handling the pandemic. Also the quality of their data collection process.

This more than fraud, and in any case the article also makes that case.

> While today’s cumulative deaths is highly dependent on yesterday’s (it should theoretically never be less, for example), today’s new deaths isn’t as directly correlated to yesterday’s.

Sort of. Assuming a wave pattern, values will be close to neighbouring values and vary in an gradual manner.

Except that the timing for reporting cases and deaths often varies considerably on weekends, which is why it is better to look at 7 day moving averages to smooth out that noise.

Benford’s law is something that’s not the most obvious why but I’ve tested it myself on even small sets of numbers like 1 years worth of transactions at a small company and it comes out really really close to shaping that curve. I’m was kind of surprised because it’s supposed to work on large datasets but I’m talking literally under a few hundred transactions where it’s often the same vendors repeating and the curve was still pretty close. So looking at this article I do believe something odd is likely going on where indicated.

Any time a random person invoked Benford's Law, be wary. It's often misapplied for fraud detection, and it's use in any new arena has to be justified. You can't simply take a bunch of data and assume it should follow Benford's Law.

Here's another law: if you multiply two random integers, then the most likely outcome for the last digit is "0". The most likely outcome for the first digit is "1", but by a smaller margin.

HOW? I just did a toy trial with two random integers between 0 and 1,000,000. In Excel. I'm getting between 24% and 31% of the products ending in a '0'.

Benford's law makes intuitive sense to me, in that as soon as you run through the leading digits 1-9, you then spend just as long as all those combined with a '1' in front of the "next set".

But I always figured even digits would be more probable, but uniform across the [0,2,4,6,8] set.

Well, if either number ends in a 0, the product will end in a 0, so that's almost 20% of the cases right there. And if you get any even number and a number ending in 5 you'll get a product ending with 0 also, so 24-31% total sounds about right.

Such a simple answer. Thanks. I just did a 10x10 times table and got the following prevalences:

    [27, 4, 12, 4, 12, 9, 12, 4, 12, 4]

So '0' has 27% probability, '5' has 9%, and the rest of the evens 12%, the rest of the odds 4%

Another way to explain it is consider the prime factorization of both numbers. The product will contain all of the prime factors from both, so if there is at least one 2 and at least one 5 in then the result will be divisible by 10.

if you draw out the 10x10 multiplication table, anything 0 * x or x * 0 ends in 0 (19 out of 100 cases), and any even number times 5 also ends in 0 (8 out of 100 cases), which lines up with the 27% you're seeing.

Since this is on the units digit, Benford's law doesn't apply and you can basically expect the distribution of units digits to be uniform.

That depends on the choice distribution. Your claim is true for uniform distribution over a large enough finite range of integers.

If the range is too small, you fail. If the choice is not uniform, you may fail.

There is no uniform distribution over all integers, only finite subsets.

Your claim fails for other distributions.

The reason this mostly works is multiplying by something ending in zero gives zero, so it is the trailing digit the most. The leading digit is pretty much Benford's Law.

Concise writeup and useful case of Benford's law. Thanks to OP

What I found interesting is that his analysis did not pick up the (now strongly evidenced) degree of inaccurate reporting from India. Indications are that India's covid deaths are between six and seven times higher than reported yet it looks pretty good on the Benford chart.

From my coworkers in india, I think that India's reporting has been less to do with inaccuracy and more to do with their entire healthcare system just crumbling under the pressure of covid. Far too many people getting (and dying from) covid to accurately measure and capture the true impact. It was real bad.

Yes, lots of people died in rural areas and were buried without ever getting near a doctor in the Delta wave, so they weren't recorded.

Not saying India did this, but I would imagine someone aware of Benford's law could tweak numbers to look valid with it taken into account, right? The more folks talk about "I used Benford's law to spot malfeasance", the less useful it becomes.

It depends how much the tweak implementer knows about Benford's law. If you look through the figures in the Appendix of https://econwpa.ub.uni-muenchen.de/econ-wp/othr/papers/0507/... , a lot of people get the first and sometimes second digit behavior right yet fail to properly falsify the less significant digits. Of course, this sort of analysis might fall apart once you've summed 20 or 40 or 300 days of data.

It really depends on how you're manipulating data.

If you're just making up numbers, Benford's law could pick it up (unless you're aware of Benford's law and create numbers to follow it).

However, if you're just counting a sixth of all deaths, then your numbers will follow whatever pattern they were following before, just lower across the board.

The general form of Benford's law also fails in situations where numbers are capped for some reason. You'll wind up with a strong grouping around that particular number.

All in all, Benford's law can never prove anything. At most, it can give a good reason to look into a situation.

This is an excellent point.

Benford's law is sort of like the Bechdel test. It's trivial to evade if you know what algorithm for which to optimize; it's value is in demonstrating the surprising number of places where it fails.

The CDC estimates that in the U.S., there are about 32 cases of undetected covid-19 for every 10 that are detected. That ratio (derived from testing of pregnant women, car crash victims, blood donors, etc.) was from before Omicron, when there was for a while a shortage of testing capacity. So, if the ratio in India is 60 to 10, while that is worse than in the US, it's not that much worse.

In any case, Benford's Law helps to detect fraud (made-up numbers), not necessarily just inaccuracy (a low detection percentage, for example).

They're talking about deaths, not cases. For a case to go unreported is one thing, for a death not to be is much worse.

Benford would likely detect manipulation, but not systemic underreporting. If they're just missing half the cases and accuracy is essentially arbitrary, then it probably wouldn't show up as violating Benford.

I'd you divided all your measurements by 7, I'm pretty sure both datasets would still align with Benford's curve.

I'd you divided all your measurements by 7, I'm pretty sure both datasets would still align with Benefords curve.

TLDR - Article Highlights https://www.kontxt.io/document/d/nV9ATOU9PbvlwnbPlzXtdkouxX-...

Would you please stop doing this? It's repetitive and therefore off topic here.

It's fine to occasionally post links to your own work when it's relevant. But it's not fine to do it routinely or generically—HN readers consider that spamming.

Sorry. I've checked HN several times a day every day for a decade and personally desired to see highlights of articles as a preview or summary since there's so much quality content here and so little time to look at it all. So I built Kontxt.io to save, organize, and share the best parts of the web. I thought others might be be interested and find it useful as well.

I totally get it, but posting generic links in the threads isn't the way to share the project. The more you do that, the more users (and eventually the software) are going to interpret it as spamming. I know it can be frustratingly difficult to get attention on HN, but if you're going to post a link in a thread, it needs to be in connection with something specifically relevant.

Since you haven't posted a Show HN in two years, you should try again. "I built Kontxt.io to save, organize, and share the best parts of the web" is a good line - you should rework it (shortened) into your title. See https://news.ycombinator.com/item?id=22336638 for tips and https://news.ycombinator.com/yli.html for advice about how to present your work to HN. Email hn@ycombinator.com with a link when it's up, in case we can help. Good luck!

Reply on news.ycombinator.com