Investigating and preventing scientific misconduct with Benford’s law | Hacker News Reader

Investigating and preventing scientific misconduct with Benford’s law

researchintegrityjournal.biomedcentral.com

28 pointsPaulHoule21 comments

It can be fooled, and genuine data can sometimes not follow Benford’s law for a variety of reasons, but it’s a good tool in the toolbox for quick examinations. There is another technique (which I can’t find a write-up of, unfortunately) that’s similarly cheap to apply (although limited to cruder fraud than Benford’s law testing can detect): you can calculate backwards from a given percentage to see if it’s actually possible to get that percentage with the number of data points, test subjects, etc. If you have 71 observations and a reported rate of 57%, the test immediately tells you that’s been faked: it’s possible to get 56.3% with 40/71 or 57.7% with 41/71. Neither rounds to 57%. Even tests this crude will detect fraudulent papers, gut level of sophistication of fraud is surprisingly low.

It won't. Many researchers can't round correctly. Because they're doing it by hand. On Excel.

Yeah, incorrect (or more likely misunderstood) rounding is not a smoking gun, and is a pretty silly thing to try and use to "catch" somebody. I've actually regularly seen administrators with nothing to add to a discussion call numbers into question because they can't understand how rounded values may not add up exactly to a column total. For that reason, it's relatively common practice in consulting reports to fudge the numbers so the rounded values do add to the colum totals (there are a few other fudges like this to make numbers easier to understand). It's only actually dishonest or material if you then try and explain your conclusions based on the 0.01 you tweaked.

The point more generally is figures that can't be reached with the denominator. Rounding errors can only account for some of that, but there's plenty of cases where it can't.

I wish I could have found the writeup, it did a much better job explaining it than I did. Including the rounding thing obfuscated things; they had cases where the paper rounded to one or even two decimal points and that number wasn’t achievable with the denominators they had. To re-work my original example, they had real cases where 71 observations would have reported percentages of 57.1%, which couldn’t be achieved with any kind of rounding.

The law says 1’s appear above average as a leading digit, while 9’s below average. https://en.wikipedia.org/wiki/Benford%27s_law

Thus if a data set doesn’t have the same distribution, it might be suspect.

An easy very easy way to fool this would be to introduce a random non zero distribution aka salt to every datum.

Wouldn’t that just shuffle the distribution of the digits rather than creating Benford’s (power) law distribution?

This would confound the trailing digit portion of a Benford's-law-style analysis, but not the leading digits.

> Furthermore, the law can be generalised to digits beyond the first

Reminds me of a fun use for second digit analysis:

"This study applies Benford’s law to detect anomalies in county-level vote data for the 2020 US presidential election. Most prominent distribution violations are observed with Republican vote counts in blue states, all vote counts in states won by the Democratic candidate, and Democratic vote counts in swing states. Distributions are anomalous in swing states won by the Democratic nominee and not anomalous in swing states won by the Republican nominee. The results are robust to two-digit analysis, Monte Carlo simulations of p-values, broad or narrow swing state definitions, and when compared to distributions observed in 2008, 2012, and 2016 elections." - ["Detecting Anomalies in the 2020 US Presidential Election Votes with Benford’s Law" (2020)](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3728626)

It's too bad they don't disaggregate the data by state, instead assuming states which voted particular ways are equivalent. I guess that the Benford Law analysis wouldn't be as easy if they did due to the fewer number of counties to analyze per state?

So I wish they had instead grouped counties by how they voted (since certifying votes is done on the county level, typically), and run the analysis that way.

I'd also like to see an analysis based on county size (by population density, and by absolute population). As you get non-random effects based on density.

It's also interesting that in certain cases they use a 10%-level for confidence.

Edit to add a question: Does the registered voting population of these districts follow Benford's Law?

I don't know about that particular analysis, but there've been a number of such claims that don't stand up (mostly because, as you ask, the districts themselves don't follow Benford's law). See, for example, "Inappropriate Applications of Benford’s Law Regularities to Some Data from the 2020 Presidential Election in the United States" by Walter R. Mebane, Jr. [0], and "Why do Biden's votes not follow Benford's Law?" by Matt Parker [1]. This fits the general pattern that there's been a lot of suspicion raised about fraud in the 2020 election, but none of it actually seems to pan out.

[0] http://www-personal.umich.edu/~wmebane/inapB.pdf

[1] https://www.youtube.com/watch?v=etx0k1nLn78

It seems like the Benford's Law distribution should be adjustable by the specifics of its input data.

Remember, according to Wikipedia, Benford's law applies to election data in every country except the United States, where the laws of statistics are totally different.

For elections, the test should be with the second digit, not the first.

See https://en.m.wikipedia.org/wiki/Benford's_law > Benford's law has also been misapplied to claim election fraud. When applying the law to Joe Biden's election returns for Chicago, Milwaukee, and other localities in the 2020 United States presidential election, the distribution of the first digit did not follow Benford's law. The misapplication was a result of looking at data that was tightly bound in range, which violates the assumption inherent in Benford's law that the range of the data be large. The first digit test was applied to precinct-level data, but because precincts rarely receive more than a few thousand votes or fewer than several dozen, Benford's law cannot be expected to apply. According to Mebane, "It is widely understood that the first digits of precinct vote counts are not useful for trying to diagnose election frauds."

The other examples on this page used the second digit for their election analysis.

Except the other digits failed too.

They are quoting from the Wikipedia article you yourself suggested. You are free to suggest another source.

Wikipedia cites this article:

https://physicsworld.com/a/benfords-law-and-the-2020-us-pres...

> In a working paper published on 10 November, Mebane looks deeper at the US election data using a 2BL test, based on the second digits and Benford’s law digit probabilities, along with other statistical tools.

> The bottom line: there are no signs of irregularity in the officially declared precinct vote counts data from Fulton County, GA, Allegheny County, PA, Milwaukee, WI, and Chicago, IL, as some have claimed.

That article cites this paper:

http://www-personal.umich.edu/~wmebane/inapB.pdf

> The vote counts from the four jurisdictions are not final, so one should treat them cautiously. Nonetheless preliminary analysis shows little that suggests there are problems.

Presumably a more up to date source exists now that the votes are finalized. The paper also has a link to the data they used on GitHub if you'd like to see for yourself (and both this & the paper below say they downloaded it from the Secretary of State websites of each state, so presumably you could do that too if you didn't trust this random GitHub).

This paper from MITRE is interesting but doesn't use 2BL. (They don't find any evidence of fraud. They do discuss 2BL in an appendix.)

https://apps.dtic.mil/sti/trecms/pdf/AD1148123.pdf

I remember from an election fraud class in college that the best digits to check on vote counts (particularly in places like Russia) are actually the trailing digits, which should be uniformly distributed. Apparently eastern European fraudsters at this point are sophisticated enough (and have enough leeway to fudge the vote counts) that they can get past checks based on Benford's law, but they are usually too lazy to whiten the trailing digits of their fake numbers.

I was curious about 2020 after the pop science emerged and checked all of these precincts' trailing digits (as well as a few other statistics), and they looked totally fine.

Matt Parker did a good video with lots of visuals to explain why Benford's Law works in general, but why it cannot always be simply applied to election results.

https://youtu.be/etx0k1nLn78

Could you please stop posting unsubstantive comments and flamebait? You've unfortunately been doing it repeatedly. It's not what this site is for, and destroys what it is for.

If you wouldn't mind reviewing https://news.ycombinator.com/newsguidelines.html and taking the intended spirit of the site more to heart, we'd be grateful.

No, I don't think I'm Going to Go out and change Everything about my Replies.

Reply on news.ycombinator.com