So it takes the data and draws it on a log scale to visualize it, but doesn't explain why the data naturally transposes to a log scale (we are talking about random data sets in nature). If I think of it as a village with 1,000 people is more likely than a village with 2,000 people is more likely than a village with 3,000 people (and this continues in both directions), the only real conclusion I can come to is that nature prefers smaller numbers of things. But that wouldn't extend to something like accounting books (or at least the jump is not immediately obvious to me).
EDIT: I guess I chose a bad example (although my point still is useful as a thought exercise), since this was used as an exception to the rule:
> Benford's law is violated by the populations of all places with population at least 2500 from five US states according to the 1960 and 1970 censuses, where only 19% began with digit 1 but 20% began with digit 2, for the simple reason that the truncation at 2500 introduces statistical bias.
Hopefully that gives you an intuitive sense why Benford's law gives a highest probability for 1 as the leading number but 2 is the next most common and so on. You can also see the weaker application by the same argument to the following digits.
https://digitalcommons.calpoly.edu/cgi/viewcontent.cgi?artic...
StackExchange math has a decent analogy: https://math.stackexchange.com/questions/781/why-does-benfor...
Random data sets in nature, measured in some arbitrary unit chosen by mankind. If one assumes expressing lengths in inches, feet, meters, parsecs, or any other arbitrary units will not change the distribution of initial digits, Benford’s law follows.
For example, when converting a set of lengths measured in yards to one measured in feet, the numbers originally beginning with a ‘1’, i.e. in the half-open interval [1,2) will start with a ‘3’, ‘4’, or ‘5’ i.e. lie in the half-open interval [3,6) in the new table. That can’t be true if initial digits are uniformly distributed.
No one really knows why it's true. It's also true across physical constants, not just artificial quantities made up by humans.
The same type of distribution pops up in language, in nature, in numbers, popularity of opening chess moves, everywhere.
Any distribution of numbers that is "scale free" follows Benford's Law. This is extremely well understood.
The distribution of the areas of lakes, heights of trees, the proportion of the answers that start with the digit "1" should be the same no matter what units you use. You can measure heights in barleycorns, metres, inches, or cubits, the proportion of answers that start with the digit "1" should remain constant across those different units. Grind through the sums and Benford's Law pops out.
And Benford's Law is not Zipf's Law. Zipf's Law is also well-understood when you look at coding theory, and how we would expect language (and mutable codes in general) to change so that more frequently used "atoms" are "shorter.
So I don't really know what you're claiming here, but on the surface of it, with things I've studied, your comment seems mostly wrong.
But then ..
https://terrytao.wordpress.com/2009/07/03/benfords-law-zipfs...
I have only skimmed the paper, but apparently Belevitch [0] showed (in 1959!) that Zipf's law and Mandelbrot's Law [1] are essentially just first-order and second-order approximations, respectively, for a large class of probability distributions, so no wonder it is so ubiquitous.
[0] Belevitch, V. (1959). On the statistical laws of linguistic distributions. Annales de la Societe Scientifique de Bruxelles, 73(3), 301–326.
[1] https://en.wikipedia.org/wiki/Zipf%E2%80%93Mandelbrot_law
basically the reason for this is that for any probability distribution over natural numbers that can reasonably be called the average probability distribution of natural numbers, I have to assign smaller probabilities to larger numbers. There's a lot of ways in which this is formally true, but it's certainly true practically. Among the k digit numbers, the ones starting with 2 are larger than the numbers starting with 1, so they have to be at most equally likely, but there are the same number of each, so the probability of picking any number starting with two has to be less than or equal to the probability of picking any number starting with 1.
Typing that is easy with a composing keyboard. AltGr+!,? outputs ‽ (Or AltGr+?,! is ⸘)
Apologies for the off-topic tangent.
One-off tangents like that are surprisingly fun, and informative.
A lot more small transactions than large transactions (how frequently do you buy a meal or groceries? vs how frequently do you buy a car or house?)
A lot more small companies (every independent bookstore, electrician, etc.) than large companies (how many >$1 billion market cap companies).
A lot more small salaries (entry level workers) vs large salaries (CEOs.)
A lot more small stock trades (minor portfolio adjustments) than large stock trades (liquidating your entire inventory as you enter/exit a market.)
Start with any number and repeatedly add any percentage, e.g. 10%. Repeated multiplication will get you through the 2000s quicker than it got you through the 1000s.
E.g. the boiling point of water starts with 1. That's not an aspect of Benford's law: it's intentionally set.
In binary, every number begins with 1!
To convert from yards to feet, you multiply by 3. Anything starting with 4 or 5 will always start with 1 after this multiplication, as will lots of things starting with 3 or 6. So starting with a 1 has to be as likely as starting with 3, 4, 5, 6.
The same sort of thing has to work if you multiply by any number. Benford's law gives you the distribution where this works.
The fact that this is scale-invariant means that it's not about preferring smaller numbers of things. It's just that the number of items in the range 10,000-19,999 is as large as the number of items in the range 0-9,999. Slightly bigger things also tend to start with ones.
This all just assumes that the distribution of leading digits doesn't change if you scale your data, which I find intuitive enough to be satisfying.
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
import seaborn as sns
sns.set_context('poster', font_scale=1.3)
n_values = 10000
lower_bound = 1
upper_bound = 100000
nums = np.array(
[int(str(x)[0]) for x in np.random.randint(lower_bound, upper_bound, size=n_values)]
)
fig, ax = plt.subplots(figsize=(12, 8))
sns.distplot(nums, kde=False, bins=list(range(1, 11)), ax=ax)
ax.set_xticks(list(range(1, 11)))
fig.tight_layout()
> Bedford's Law is apparently just a fancy name for the phenomenon where 1's appear much more frequently than 9's in any random sample of numbers.
because, as my example shows, it's false for the case of random numbers sampled uniformly across five orders of magnitude. You're also right that it's possible to construct a distribution (such as taking -log(rand())) where this behavior is observed.
To be more precise:
> Benford's law tends to apply most accurately to data that span several orders of magnitude. As a rule of thumb, the more orders of magnitude that the data evenly covers, the more accurately Benford's law applies. For instance, one can expect that Benford's law would apply to a list of numbers representing the populations of UK settlements. But if a "settlement" is defined as a village with population between 300 and 999, then Benford's law will not apply
There are many distributions which do lead to the distribution of leading digits that follow Benford's law, just not uniform. Nor "uniform over several orders of magnitude".
They mean uniformly on a log distribution. Ie, there are a uniform number of elements between 10^x and 10^(x+1), for many different values of x. This is best shown by the graph here:
So, in general, one cannot state that 1's appear much more frequently than 9's.
Also, that’s not in any random sample of numbers; you need to sample from a distribution with a certain property.
The distributions of many samples of real-world phenomena happen to have that property.
That's about leading digits only.