Look out for No. 1 | Hacker News Reader

Look out for No. 1

timharford.com

102 pointsyread26 comments

"Nobody seems sure why so much data has the Benford distribution. " : Have a look at http://terrytao.wordpress.com/2009/07/03/benfords-law-zipfs-... (cited in the previous discussion on HN) for an excellent exposition of the theory, and an explanation of the 'power' of the Benford Law.

That Terry Tao article is excellent, as usual (well, at least for math dorks - what's unusual about this one is that you don't need a very high level of dorkery to grok it 100%) and the "executive summary" reason that Benford's law applies so broadly is easy to find:

"More generally, it is not hard to show that if X obeys the continuous Benford’s law, and one multiplies X by some positive multiplier Y which is independent of the first digit of X (and, a fortiori, is independent of the fractional part of log_10(X)), one obtains another quantity X' = XY which also obeys the continuous Benford’s law."

In other words, multiplicative combinations of (independent) quantities will inherit Benford's law as long as any one of the quantities obeys it on its own, so it just takes one sub-factor that grows exponentially in order for an entire distribution to follow the law.

That line especially bothered me, it makes it seem like it's some divine or magical rule. If that were the case, I would hate for it to be used as if it were infallible with no actual understanding.

Good to be aware of this if you plan to falsify your tax returns, according to a '98 NYT article (from which this post takes its neat title)

http://www.nytimes.com/1998/08/04/science/following-benford-...

Previous discussion of Benford's Law on HN

http://news.ycombinator.com/item?id=2701342

I'm impressed that Madoff was smart enough to mimic Benford distribution in his monthly returns (that's what the article led me to infer).

I actually made an r/math post about Benford's Law that I thought was pretty interesting, although it didn't get much attention or many comments (admittedly bad signs). Any peer review (or su-peer-ior review) is appreciated.

Here's a snippet: "...generation of the geometric layout for Benford first digits is a similar process to creating a Sierpinski Triangle by drawing a triangle, and randomly plotting midpoint dots between vertices and sides recursively. In the Benford case the plotted dots are instead individual numbers, which share a geometric connection (e.g. two galaxies colliding will on average be a doubling, so galaxy size should possess Benford distribution)."

http://www.reddit.com/r/math/comments/gkj2i/benfords_law_exp...

Last month Khan and Vi Hart made some very interesting videos on Benford's Law:

http://www.youtube.com/watch?v=6KmeGpjeLZ0

http://www.youtube.com/watch?v=SZUDoEdjTzg

No link in the article so here is the report cited:

http://onlinelibrary.wiley.com/doi/10.1111/j.1468-0475.2011....

I don't think the issue at hand is that fraud detections are unsophisticated. The issue at hand is that the fraud detections are in cahoot with the fraud themselves. For example: Enron, Worldcom and Accenture (past) Greece, Italy, Spain and Goldman Sachs, Fitch, Moody, S&P (present) Chinese local banks, Chinese companies and SEC (future)

The problem with hackers is that they always try to find mathematical solutions to moral problems. What the world needs right now is a moral solution. A beheading for those that have fouled.

Unfortunately, the hackers, the workers - ones that actually have the power to demand changes - are willingly allowing themselves to be exploited by those that lie, and not demanding any moral judgements. They're not interested in changing the politics, they claim. They just want to keep producing interesting things, they exclaim. They don't realize they're the ones keeping the lie going. They're the ones propping up this decrepit shell of a society.

Until one day the shell collapses on their kids, anyways.

The problem is that revolutions tend to be more harmful than their predecessors: see China in the 60s, for example.

I think China in the 10's is a good counterexample.

That's definitely not a given. Revolutions are a time of instability and the outcome can be worse or better. It mostly depends on what precipitated the revolution in the first place, how educated the people in the country where the revolution takes place are and how involved they plan on being once the violence dies down. A revolution is a time of transition and instability without any guarantees towards the outcome.

What is a moral solution, can you give a historical example?

How about women's suffrage? I understand that there were many factors playing into the move toward equality of the sexes, such as wars that required society to use women's labour, but by-and-large it was an incredibly large shift accomplished through moral suasion rather than violence. And although it seemed very gradual - a couple of generations, and some would argue still not done - it was pretty rapid as societal shifts go.

> Unfortunately, the hackers, the workers - ones that actually have the power to demand changes - are willingly allowing themselves to be exploited by those that lie, and not demanding any moral judgements.

(Emphasis added)

I think this point of yours is frankly quite wrong. The productive part of society is not the part of society with political power. Gaining political power is in and of itself a non-productive activity, and accomplishing it takes so much time that none is left over for productive activities. So the non-productive part of society naturally has power over the productive part of society, just as teenagers who focus on being popular end up being more popular than teenagers who focus on programming computers and learning stuff.

Sure they do. Imagine all the doctors and programmers and air traffic controllers (and many others in different industries) going on strike tomorrow.

Spontaneously? No, someone is going to have to organize all of them and get all of them on the same page. In other words, someone is going to have to join the non-productive political class to organize all the programmers. Then it's the organizer who has the power, and then you have a central point that's vulnerable to corruption, which is exactly what has happened with labor unions in the past.

Facebook and Twitter works well.

> What the world needs right now is a moral solution. A beheading for those that have fouled.

I understand you're using a metaphor here - or at least, I hope you are - but you ought to choose your words more carefully. The literal interpretation of what you're advocating led to mass genocides, slavery, destroyed nations and cultures, and many places haven't recovered from it. Yeah, there's crooks in finance. Khmer Rouge, pogroms, "reeducation" slavery camps, and the like aren't the answer.

If you were just trying use a colorful metaphor, you can just use "prosecute" instead of behead without losing your impact.

I'm confused; which of those ills (which one could compare to the corruption that this is about) do you not think had a "moral campaign" (similar to what the grandparent suggests) fighting against it?

I'm not saying it's the best framing of the situation, but going from "we should fight this because it is wrong" to "slavery, genocide, and reeducation campus" is a massive jump.

> I'm confused

The original poster advocated "beheading" people for doing economic activities that he disapproves of.

It's not a new idea. It hasn't worked well in the past.

Have a look at 'the French revolution' for one instance in time where it worked reasonably well. The French royalty and its associated entourage had become such a drain on the country that the French decided to get rid of them in the most drastic fashion possible.

Yes it was gruesome and bloody. But it did move France forward in time ahead of some other European countries that even today have not managed to rid themselves of their God anointed overlords.

Of course there are plenty of counter examples but it doesn't always have to be bad.

That's curious, I'd have thought that the Reign of Terror would be a good example in support of what lionhearted is saying.

http://en.wikipedia.org/wiki/Reign_of_Terror

As I wrote elsewhere in this thread, the period of a revolution itself is extremely messy and it could go either way. But I think that it is impossible to argue that France was eventually worse off because of the revolution.

Ahhh, sorry, I guess I didn't take "A beheading for those that have fouled." as literal. And somehow I didn't notice your last paragraph either...

In my defense, I'm apparently stupid. ;)

There was a story a little while ago, about a corrupt official in China who allowed baby milk to be polluted by taking bribes to look the other way during safety inspections. The Chinese executed him.

Reply on news.ycombinator.com