Hacker News Reader

Comment by jpk | Hacker News Reader

Interesting article, but this stood out to me.

It frustrates me to see people in the smash community treat measures like elo as "the truth" because they "don't have any human input". This simply factually incorrect - these so-called objective measures have as much human input as anything else, codified into the constants and design choices of their algorithms. Designing these things is as much an art as it is a science, and the choice on how to weigh placements, upsets, losses, consistency, peaks, and the like are all just that - choices, made by a human sitting in a chair with Sublime Text 3 open.

I feel like this is applicable nigh everywhere. From social media timeline sorting, to industrial processes, to Melee rankings. Using an algorithm doesn't eliminate the human element from a system, it only abstracts it away.

View on HN

There is a large contingent of radical empiricists in machine learning who assume "big data + automation = truth", especially on HN, and this is a message they need to hear more of.

People have been advocating radical empiricism in some increasingly uncomfortable contexts recently, and I hope it's just that it's the only thing they were taught and the only way they know how to think about their craft. The alternative is that an increasing number of people really do want machines to triumph over human judgment and morality.

Gets me thinking of an Adam Curtis' documentary series:

https://en.wikipedia.org/wiki/All_Watched_Over_by_Machines_o...

Not exactly sure how relevant it is though.

This might be a case where classical econ ("social choice") can give some helpful perspective. Arrow's impossibility theorem is the most famous impossibility result in this area; it says that no algorithm can take in a set of rankings (e.g. match outcomes) and produce an aggregate ranking in a way that satisfies a small set of fairness criteria. This is classically interpreted as saying that any aggregation method must be "unfair" in one way or another.

Arrow's theorem is mathematics, but even citing it is greatly misleading. All Arrow proved was that, sometimes, voting becomes rock-paper-scissors between the three most popular candidates. That isn't "unfair".

It especially doesn't mean that all voting systems are equally bad, and yet the most common reason for people to cite Arrow's theorem is to try and dispute the idea that we can do better.

Unlike your parent comment, your comment is total nonsense. Arrow proved that, given a set of ordinal preference rankings held by several individuals, the concept of an aggregate preference ranking describing the overall "will of society" is not well defined; subject to four or five unobjectionable constraints, no function determining such an overall ranking exists.

Voting systems which obey the assumptions of the theorem frequently break down in ways that cannot be described as "rock-paper-scissors between the three most popular candidates", as when strategic voting caused the papal conclave of 1334 to unanimously elect the least popular candidate to the papacy.

Were you perhaps thinking of Condorcet's work when you referred to rock-paper-scissors? He, not Arrow, famously wrote about how voter preferences were not necessarily transitive.

That "rock-paper-scissors" cycle among the top preferences is the Smith set, and every sensible voting "violates" Arrows theorem by saying that sometimes it's ok to acknowledge that cycle.

One of Arrows assumptions, "Irrelevance of Independent Alternatives", is far too strong. If you relax it to "Local Independence of Irrelevant Alternatives", then Arrows theorem no longer applies. It's just not a relevant theorem.

Claiming that allowing a voting system to recognize a rock-paper-scissors cycle among the top candidates is the same as allowing dictators is obvious rubbish. But if you interpret Arrow's theorem naively, you might be tempted to make such a claim.

> Arrow's impossibility theorem is the most famous impossibility result in this area; it says that no algorithm can take in a set of rankings (e.g. match outcomes) and produce an aggregate ranking in a way that satisfies a small set of fairness criteria.

Ignoring the parenthetical "e.g. match outcomes", this is a correct description of Arrow's impossibility theorem. I don't see how match outcomes could possibly be an example of a set of rankings in the sense of the theorem, though.

Yeah, that was unclear. I'm thinking of a match outcome between teams A and B as a partial ordering on all the teams. Classically Arrow's deals with only total orderings as input -- I'd been thinking that it extends to partial orderings, but hmm, I'm not sure what research says about if we restrict the inputs to be just pairwise orderings/outcomes.

I always end up going back to Rump's example (https://link.springer.com/article/10.1023/A:1015569431383):

> Rump’s example is to compute the expression

> ƒ = 333.75 * b^6 + a^2 * (11 * a^2 * b^2 − b^6 − 121 * b^4 − 2) + 5.5 * b^8 + a/(2b)

> with a = 77617 and b = 33096. On an IBM S/370 main frame he computed ƒ in (1.1) using single, double, and extended-precision arithmetic, to produce the results:

> Single precision: ƒ = 1.172603...

> Double precision: ƒ = 1.1726039400531...

> Extended precision: ƒ = 1.172603940053178...

> This suggests a reliable result of approximately 1.172603 or even 1.1726039400532. In fact, however, the correct result (within one unit of the last digit) is

> ƒ = −0. 827396059946821368141165095479816... (1.2)

> Even the sign was wrong.

It's worth remembering that computers are very fast, but also very, very stupid. They do exactly what you tell them (or teach them, if we're talking about MCL), which isn't always what you want. Just because you've got a very scientific looking algorithm doesn't mean it's correct. Just because you're aggregating all your data doesn't mean the results are truly meaningful.

I'm in the middle of reading a pretty interesting book, where that is one of the core arguments - Weapons of Math Destruction, by Cathy O'Neil - I recommend checking it out if you are curious to learn more about the distinction that you made, and more of how we tend to abuse math through algorithms that are in some way designed by humans.

In modeling memory discrimination in psychology experiments, people often recommend d-prime from Signal Detection Theory (https://en.wikipedia.org/wiki/Sensitivity_index). Other recommend simple "theory-free" discrimination scores (Hits-False Alarms). This is frustrating because all measurement designs carry theoretical distributional and metric assumptions, just because one is explicit (as in signal detection theory: d' = Z(hit rate) − Z(false alarm rate), where function Z(p), p ∈ [0,1], is the inverse of the cumulative distribution function of the Gaussian distribution.) does not mean the simple (hit rate minus false alarm rate) is theory free an unadulterated. The theoretical assumptions are just different.

I'd take it even one step further and apply this to literally anything on the internet! Whether it's news (doesn't even have to be true these days), data (omitted/modified or not), opinions (externally-motivated), research (funded by god-knows-who), literally ANYTHING you view is in some way processed, designed, delivered, or created by a human being.

And even when AI becomes practical, no more than 1000 people can realistically be involved in its design and implementation. Our supposed 'objective machines' will in fact be designed to the ideals of those designing them: a generally non-diverse group of people. Food for thought!

It's "a truth", which is probably as good as you can get, given that "the truth" is always a subjective thing.

The benefit to algorithm isn't that it's infallible (it may well be), but rather than it's consistent. It's accurate, even if not correct. Considering how much of human judgement is inconsistent, there's value in quantifying it in a standard way.

Reasonable minds can argue about whether than quantification is correct or fair.

Why is accurate (by which you mean 'consistent') better than correct?

Isn't accuracy an aspect of correctness? Ie: can you be correct without being accuracy?

Primarily because "correctness" is likely to be subjective or difficult to define, while "accuracy" is not.

That said, the terms more often used here are "precise, but not accurate" (http://blog.minitab.com/blog/real-world-quality-improvement/...) but note that this uses a different meaning for accuracy than parent used. For example, say you use 22/7 to derive the value of pi. You can easily calculate it to 60 or more decimal places. You'll be very precise. However, you won't be accurate because your methodology of using 22/7 is flawed. It's also very easy to create more precision: just keep calculating. On the other hand, it can be very difficult to create more accuracy: How do we even measure pi to be able to confirm the ratio is correct?

It's not. But many "legacy" systems—using the term broadly to mean people and process—are inconsistent and not any more correct as a whole. A standardized system is easier to monitor and easier to correct, so I see this as an improvement overall for that reason.

I think my point is less concerned with a value judgement about algorithms, and more concerned with prompting critical thinking about how and who develops the algorithms, to what ends, the vectors by which subjectivity seeps into them, and how that affects their behaviors.

This is true for many statistical analyses as well. The worst is when people smuggle in their non-quantifiable base assumptions and pretend like the resulting inference is some objectively true reality.

Reply on news.ycombinator.com