Benford's "Law" says that if you have a "natural" or "real-life" data set of numbers, 30.1% of the numbers will have 1 as their first digit, 17.6% will have 2 as their first digit, and the first digits of the numbers in the whole data set will be distributed like this:
| First Digit | Frequency |
|---|---|
| 1 | 30.1% |
| 2 | 17.6% |
| 3 | 12.5% |
| 4 | 9.7% |
| 5 | 7.9% |
| 6 | 6.7% |
| 7 | 5.8% |
| 8 | 5.1% |
| 9 | 4.6% |
These are very precise numbers, and that is because Benford found a formula that is a close match to the numbers he was looking at:
Where d is the digit we are interested in and frequency is how often we expect that to appear as the first digit in a number.
Benford's own results conform to this distribution rather loosely. Here they are:
| Title | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | Count |
|---|---|---|---|---|---|---|---|---|---|---|
| Areas of Rivers | 31.0 | 16.4 | 10.7 | 11.3 | 7.2 | 8.6 | 5.5 | 4.2 | 5.1 | 335 |
| U.S.A. Population | 33.9 | 20.4 | 14.2 | 8.1 | 7.2 | 6.2 | 4.1 | 3.7 | 2.2 | 3259 |
| Physical Constants | 41.3 | 14.4 | 4.8 | 8.6 | 10.6 | 5.8 | 1.0 | 2.9 | 10.6 | 104 |
| Newspaper Items | 30.0 | 18.0 | 12.0 | 10.0 | 8.0 | 6.0 | 6.0 | 5.0 | 5.0 | 100 |
| Specific Heats | 24.0 | 18.4 | 16.2 | 14.6 | 10.6 | 4.1 | 3.2 | 4.8 | 4.1 | 1389 |
| Pressure Lost in Air Flow | 29.6 | 18.3 | 12.8 | 9.8 | 8.3 | 6.4 | 5.7 | 4.4 | 4.7 | 703 |
| Horse Power Lost in Air FLow | 30.0 | 18.4 | 11.9 | 10.8 | 8.1 | 7.0 | 5.1 | 5.1 | 3.6 | 690 |
| Molecular Weights | 26.7 | 25.2 | 15.4 | 10.8 | 6.7 | 5.1 | 4.1 | 2.8 | 3.2 | 1800 |
| Drainage Rate of Rivers | 27.1 | 23.9 | 13.8 | 12.6 | 8.2 | 5.0 | 5.0 | 2.5 | 1.9 | 159 |
| Atomic Weights | 47.2 | 18.7 | 5.5 | 4.4 | 6.6 | 4.4 | 3.3 | 4.4 | 5.5 | 91 |
| n−1, √n ... | 25.7 | 20.3 | 9.7 | 6.8 | 6.6 | 6.8 | 7.2 | 8.0 | 8.9 | 5000 |
| Design Data Generators | 26.8 | 14.8 | 14.3 | 7.5 | 8.3 | 8.4 | 7.0 | 7.3 | 5.6 | 560 |
| Numbers In One Readers' Digest Issue | 33.4 | 18.5 | 12.4 | 7.5 | 7.1 | 6.5 | 5.5 | 4.9 | 4.2 | 308 |
| Cost Data for Concrete | 32.4 | 18.8 | 10.1 | 10.1 | 9.8 | 5.5 | 4.7 | 5.5 | 3.1 | 741 |
| X-Ray Voltages | 27.9 | 17.5 | 14.4 | 9.0 | 8.1 | 7.4 | 5.1 | 5.8 | 4.8 | 707 |
| American Baseball League 1936 | 32.7 | 17.6 | 12.6 | 9.8 | 7.4 | 6.4 | 4.9 | 5.6 | 3.0 | 1458 |
| Black Body Radiation | 31.0 | 17.3 | 14.1 | 8.7 | 6.6 | 7.0 | 5.2 | 4.7 | 5.4 | 1165 |
| Street Addresses | 28.9 | 19.2 | 12.6 | 8.8 | 8.5 | 6.4 | 5.6 | 5.0 | 5.0 | 342 |
| n, n2,...,n! | 25.3 | 16.0 | 12.0 | 10.0 | 8.5 | 8.8 | 6.8 | 7.1 | 5.5 | 900 |
| Death Rate | 27.0 | 18.6 | 15.7 | 9.4 | 6.7 | 6.5 | 7.2 | 4.8 | 4.1 | 418 |
Benford's paper is dated April 22, 1937. That was before the publication of Karl Popper's "The Logic of Scientific Discovery" (1959 in English; 1934 in German). It was also before Donald Knuth's "The Art of Computer Programming - Seminumerical Algorithms", which has 193 pages on random numbers, and was first published in 1969. It was also before electronic computers were invented (arguably in 1943 with the Colossus machine) and way before such machines were available to researchers such as Benford.
So it is not surprising that Benford describes a data set with 1,000 elements as "large". He also focuses on verification, seeking out data sets to confirm his "law", rather than following Popper's guidance to challenge his "law" with tests that would attempt to falsify it. Also it may explain his loose definition of "random".
Benford said that his "Law" works best with:
He said that there was "a distinct tendency for those [numbers] of a random nature to agree better with the logarithmic law than those of a formal or mathematical nature". I disagree with that statement; random numbers do
But I do agree with his emphasis on "used" numbers, because numbers that are driven by some human activity, are more likely to fall into a pattern than to be random. A sum of money, in which compound interest is reinvested, follows Benford's "Law" very closely. This extends to natural phenomena, such as the exponential increase in populations of living things that find themselves in favourable environments. Populations of many organisms from bacteria, to locusts, to human beings see exponential population growth from time to time.
Benford's observations are interesting and worth exploring. My reason for calling his "law" a fallacy, is perhaps harsh. It applies to more data sets than we might expect, but there are many more where it does not apply, so we cannot use it to predict with confidence the distribution of first digits in any data set that is produced by something that does not have a good reason to favour smaller digits in the first position. It is a circular "law"; data sets that follow Benford's "Law" are likely to follow Benford's "Law".
Benford takes five pages of his 22-page paper to describe why linear series of integers do
It can be interesting, and may be important, to work out why a particular data set follows Benford's "Law" or not.
There are those who claim that Benford's "Law" can be applied to:
The most enthusiastic supporters of Benford's "Law", such as those interviewed in the radio show mentioned below, suggest that failure to comply with Benford's "Law" is a strong indication of something being wrong with the data set, and that it is therefore fraudulent.
I strongly disagree with this proposition. Many data sets have erroneous data, and some of it is fraudulent, but I will argue that we cannot determine the validity of a data set simply by measuring patterns in the distribution of digits in the numbers. As Ben Goldacre says, "I think you'll find it's a bit more complicated than that".
The enthusiasts often refer to the "proof" of Benford's "Law". There are proofs that particular types of mathematical series of numbers do follow Benford's "Law", but there is no proof that failure to conform to Benford's "Law" is the result of something nefarious. The tax returns of the rich and famous do contain a lot of numbers, and some of them may be fraudulent, but Benford's "Law" will not tell us that. We would do better to look at what kinds of cars these people are buying.
I suggest that it would be extremely unreliable to draw conclusions using Benford's "Law", and the idea that it should be used in fraud detection, election data, or criminal trials is highly irresponsible.
Benford's "Law" is described in detail on Wikipedia, with many apparently-learned references. It is also described in the references listed at the end of this page, and far more places than I would have expected. I'm calling Benford's "Law" a fallacy because I think there are far too many exceptions, far too many spurious correlations, and some very muddled thinking about the causes.
There are many exceptions to Benford's "Law" and "many other cases where Benford's "Law" applies that resist simple explanations". How about
The Wikipedia article, and much of the other writing about Benford's "Law" does describe distributions that would or would not be expected to obey Benford's "Law".
Let's get one thing absolutely clear: a genuinely random data set of numbers does not obey Benford's "Law". Here is the analysis of 1 million random numbers between 1 and 999999:
| First Digit | Count | Frequency |
|---|---|---|
| 1 | 110,676 | 11.07% |
| 2 | 110,695 | 11.07% |
| 3 | 111,406 | 11.14% |
| 4 | 110,801 | 11.08% |
| 5 | 111,068 | 11.11% |
| 6 | 111,575 | 11.16% |
| 7 | 111,640 | 11.16% |
| 8 | 111,472 | 11.15% |
| 9 | 110,667 | 11.07% |
| Total | 1,000,000 | 100.00% |
The probability of any digit appearing at any particular position in the number is the same for all digits if the numbers in the data set are random. Such a list of numbers does not obey Benford's "Law" at all.
If we apply a function, say log10, to the random numbers we get a completely different distribution of fist digits:
| First Digit | Count | Frequency |
|---|---|---|
| 5 | 900,067 | 90.0067% |
| 4 | 89,905 | 8.9905% |
| 3 | 8,996 | 0.8996% |
| 2 | 933 | 0.0933% |
| 1 | 90 | 0.0090% |
| 0 | 9 | 0.0009% |
This is not a surprise. The log10 values for the numbers between 1 and 999999 range from 0 to 5.999999565705301.
Once you have applied any function to the numbers, the results will probably show a completely different distribution of first digits. It may be interesting to see how these change in a particular data set, but the distribution does not indicate any kind of fraud.
I suggest, therefore, that any data set that does obey Benford's "Law" has arisen by a process that is not random, in other words it has some direct or indirect human influence. The use of the terms "natural" and "real-life" are quite misleading.
Rather than applying a "Fourier transform of the logarithm of the probability density function", or other mathematical obfuscation, we should look for the actual factors that make data sets not random.
Here's one very common scenario in which you could see many different distributions of first digits, none of which are of any importance.
If you were the founder of a club expecting a few hundred members, you might (with or without the aid of a computer) allocate membership numbers. You want them to be unique because you want to identify each member separately. If two members had the same number, you might collect the membership fee only once for these two members and you might ask the wrong member to pay the other member's bar bill.
You would probably allocate a membership number to a new member by adding one to the last number you used. You might want all members to have a four-digit number so you might start at 1001 for the first member, 1002 for the next member and so on. That would give you a maximum of 8,998 4-digit membership numbers which would be fine for a small club. If you had 999 members, all their membership numbers would start with 1. That would not be a problem.
Most (if not all) modern database products have a feature called "sequence", or "serial" or "auto-increment" which does exactly the same as the club founder did with her membership numbers. All kinds of clubs, businesses, public bodies, charities, and any other organisation that has a computer system, will probably use the sequence feature of a database on that system to allocate membership numbers, account numbers, and any identifier (or "key") for any object.
We use sequences to generate integer keys because they are simple, reliable and efficient. Let's look at what happens as we generate a sequence.
This table supposes that we start our sequence from 1, and increment by one every time we ask for a new key. This is what usually happens.
Looking at the blue line (1 to 9) we get 9 different first digits. So this does not follow Benford's "Law".
Our next 89 numbers (the yellow section) come from the 2-digit decimal numbers. As we add the numbers 10 to 19 our complete list (1 to 19) looks like an extreme Benford distribution having 58% starting with 1. When we add the numbers from 20 to 29, we get 38% starting with 1, 38% starting with 2 and only 3% starting with 3. When we add numbers from 30 to 39, we get 28% starting with 1, 28% starting with 2 and 28% starting with 3. That leaves just 16% for numbers starting with 4, 5, 6, 7, 8 or 9.But when we add the numbers between 90 and 99, the starting digits have levelled out; every digit from 1 to 9 is represented in 11% of the first digits. So at this stage this does not look like a Benford distribution at all.
When we go to the next number (100) which takes us into the next order of magnitude, then we get 100 more numbers starting with 1, before we get any other digits in the first position. Then, gradually, through this order of magnitude, the starting digits get filled in until we get to 999, when the distribution is even again.
So if you are assessing the compliance with Benford's "Law", on a real-life sequence of numbers like this, then your conclusion will depend on how far that sequence has got through the allocation of numbers. If the total number of numbers allocated is in the first third of the way through the current order of magnitude, you would see that something similar to Benford's "Law" is being observed. If, on the other hand, they are close to the end of one order of magnitude you would see that their first digits are evenly distributed. You would therefore observe that they do not follow Benford's "Law". Should you therefore conclude that the person responsible for this system is guilty of fraud?
Clearly, we should never draw this conclusion.
And, to be fair, Benford explained what I have just tried to explain and says that such sequences will not obey his "Law". This seems to have been ignored by the Benford's "Law" enthusiasts.
The Wikipedia article includes a couple of examples: the heights of tall buildings and electricity bills. I am reminded of Tyler Vigen's Spurious Correlations. The examples are not convincing. Further down the Wikipedia page we see the use of Benford's "Law" in popular culture, which simply makes me more sceptical.
I did find the radio show referenced in the Wikipedia article and I listened to it. The speaker mentioned several cases of fraud that he had allegedly detected using Benford's "Law". He quoted several court cases where he "ran Benford's and 'boom'", the defendant was convicted. Hopefully there was some real evidence as well as Benford's "Law".
I also looked at an analysis of data for the 2020 US presidential election in the 259 precincts of Milwaukee County, Wisconsin. There was a deviation from the Benford's "Law" pattern.
The number of per-precinct vote counts for Trump follows the Benford's "Law" pattern fairly closely. Biden's votes diverge from the pattern. For Biden, the numbers of vote counts starting with 1 and 2 is lower; the numbers starting with 4, 5, 6 and 7 are higher.
Looking at the data more closely, the number of votes cast in a precinct varied from 186 to 2,528.
This chart shows that in the largest number of precincts (65% of them) between 700 and 1,100 votes were cast. Only 35% had fewer than 700 or more than 1,100 votes cast. Biden got about 68% of the vote overall, so it would not be surprising if, in 65% of the precincts, the first digit of Biden's vote count was 4, 5, 6 or 7. (68% of 700 is 476; 68% of 1,100 is 748). Of course, Biden did not get the same percentage in every precinct, but this does show that it was perfectly plausible that Biden would get more vote counts where the first digit was 4, 5, 6 or 7. The distribution of the first digits of the vote count is driven by the distribution of the number of votes cast in each precinct, and the fact that Biden won twice as many votes as Trump. Trump had more of the smaller numbers and Biden had more of the larger numbers. We have to look at what is actually happening, rather than expect a particular pattern to operate across all kinds of data.
Other more sophisticated analyses also concluded that there was no evidence of fraud in this election.
Sequences are just one non-random allocation of numbers, but they are common and they illustrate the point that Benford's "Law" does not apply to all data sets. Benford said as much himself.
As I noted above, data sets of random numbers do not exhibit compliance with Benford's "Law".
There are many data sets that have been found to comply with Benford's "Law" and many more that do not. Collecting data sets that do comply is not proof of Benford's "Law" it is simply cherry-picking.
Rather than suggesting that a data set which does not comply with Benford's "Law" should be regarded with suspicion, I think we should look very closely at any data set that
There are many practical methods for checking the validity of data sets, and Benford's "Law" is a very tenuous one. Realistic checking should be done more often, but that's another story (see Data Needs Testing Too for some of these).
The Law Of Anomalous Numbers. Frank Benford's original paper.
Base-Invariance Implies Benford's Law. By Theodore P. Hill. Reportedly the most robust "proof" of Benford's "Law"
Chapter 1 of Benford's Law, Theory and Applications. The book that this comes from is Benford's Law Theory and Applications, edited by Stephen J. Miller. Princeton University Press, 2015.
https://www.cambridge.org/core/journals/political-analysis/article/benfords-law-and-the-detection-of-election-fraud/3B1D64E822371C461AF3C61CE91AAF6D
https://rss.onlinelibrary.wiley.com/doi/pdf/10.1111/j.1740-9713.2016.00919.x