Hacker News Reader

Comment by ddlatham | Hacker News Reader

I'm curious to see more about the distribution of questions and answers people had, and how the HN population may differ from the NYT's. There will certainly be self selection bias here, but if you're willing to share how you did with others, please enter it here: https://docs.google.com/forms/d/17e5BIL0lH8OHsGj89Zdtdl8GeCV...

The result summary is visible here: https://docs.google.com/forms/d/17e5BIL0lH8OHsGj89Zdtdl8GeCV...

The raw answers are visible here: https://docs.google.com/spreadsheets/d/1ZxR2_eOUtNLXwgKfLO1J...

View on HN

People familiar with unit testing and test driven development will feel at home with this kind of puzzle. That doesn't mean that they will be less biased in social/political decisions, it just means that this test will fail to prove a point.

Hah :)

Seriously, though, it seems a bit of leap from the existence of confirmation bias to explaining away the public outpourings of US Politicians about their financial crises and foreign policy disasters - in the absence of better data as to just why the given statements were made ascribing this to confirmation bias seems itself open to accusations of confirmation bias! :)

That's exactly what I was thinking. I (sometimes) follow TDD, and I applied it to this problem. I made sure to include negatives, 0, positives, and include primes here or there to help avoid issues with multiplication/exponentiation. After a few of these, I felt pretty confident that the rule was simple.

My process was:

At that point I could've done some more to be really certain, but felt confident enough and guessed (correctly).

[16 32 64] [0 0 0] [1 2 3] [3 2 1] [-1 0 1] [3 5 7] [-2 -1 0] [111 54231 9999999999999999999999]

At that point, I correctly answered the question.

Similar to mine, but we both actually failed. Real numbers are accepted and we did not test that.

It accepts floating point numbers. To approximately double precision.

But it accepts zero percent of real numbers.

I can nitpick better than that!

To talk about a certain fraction of real numbers you have to have a distribution over them. In general we take the uniform distribution if no distribution is explicitly given. That doesn't work for real numbers (it doesn't even work for natural numbers). (See https://math.stackexchange.com/questions/14777/why-isnt-ther...)

If there's no implicit default distribution, we have to pick on. I can pick one where they cover an arbitrary high percentage of real numbers..

Down the rabbit hole of pedantry: we don't need a distribution, just a measure, if we want to talk about how many reals it accepts. The Lebesgue measure is implied on the Reals if none is given, and the computable reals have measure zero.

We can't reasonably talk about a percent coverage, since the Lebesgue measure of the reals is infinite, but as a non-technical description, 'zero percent' is morally equivalent to saying it only covers a measure-zero set.

I did this plus also testing [1 1.5 2]

So you limited yourself to non-negative integers, and had no idea what negative numbers or fractions would do.

Based on the context of the question and the UI of the testing interface, fractions seem unlikely to be an intended part of the question. I likewise wouldn't bother testing unicode U+216x roman numerals.

It refused to accept both fractions and imaginary numbers. I did test negatives and zeros since that was really the only remaining set I could think of.

Most importantly I used about 6 tests (3 right 3 wrong) to come up with the answer and then did another 17 looking for the trick. After all, it couldn't just be that simple right?

Also he didn't know what numbers whose digits add up to 88 would do.

Their input on the phone wouldn't allow minus or period characters so I assumed positive integers were the set.

The presentation of the problem -- and I know that trusting the problem state is unwise sometimes in cognitive-bias tests, since many such tests are actually designed to be "we said we were asking X but actually meant Y" -- indicated a simple rule, rather than one which would behave differently on different classes of numbers.

So after the tests listed above I felt confident enough to guess.

Did you try floating point numbers? I didn't see anything in the text that said integers only.

I tried floating point numbers. Also, at 28 decimal places, the test breaks; it's not arbitrary precision. So, technically, the answer isn't simply "any ascending sequence of numbers".

I actually avoided going that far to avoid getting bad data. I was trying to answer the question, "What does the experimenter THINK his rule is?" rather than what will the computer do. Since the computer can't be infinite, it will inevitably fail with overflow, underflow, and such.

I was relieved, in fact, when it worked with negatives and floats in a "safe" range.

I also tested with 1,1,2 and 1,2,2 to make sure that the required increase applied to ALL of the values, not just a specific pair.

I too tried to test if it was only one pair that was significant. However I grew impatient and didn't try to come up with more tests when I thought I had a sufficient answer to explain my most vexing observation (negative, positive, positive out of combinations involving negative numbers).

The observation to brainstorm for ways of proving that a statement is in fact wrong, and exhausting them, is such an eloquent way of wording the hunt for a negative.

Hah! Nice catch.

Pentester. :)

The slightly-shorter 0.60000000000000000, 0.60000000000000001, 0.60000000000000002 will break it too, for what it's worth. If you punch those in to a Javascript console, you can see that the FP representation of all three is 0.6.

0.20000000000000000, 0.20000000000000001, 0.20000000000000002 also work.

Yes I did, though not to 28 places as tptacek had done. With 2 to 4 place floats, including negatives, the test worked as expected.

Like others here have said it wasn't a particularly hard "rule" to figure out. Easy to immediately rule out geometric relationship as in 2^y, which didn't leave a whole lot of possibilities to test. For the commenters here, I'd attribute ease of finding the solution to familiarity with the kinds of problems that programming presents.

Which leads to the idea there's value in learning even the rudiments of programming. Logically, it should encourage better problem-solving skills in general. We might think there this has important implications for our educational systems. But I know, that's probably not realistic at all.

I know the question isn't directed at me, but I thought you might want to know. I couldn't try negatives or floating point on the iPhone. The keypad didn't have the option.

I was a little less systematic, but still had a positive:negative ratio of 1:2 before I submitted my answer.

4,6,8 Y 1,1,1 N 1,2,3 Y 1,6666,8777 Y 1,0,1 N 3,2,1 N 3,2,3 N 5,6,4 N 7,5,6 N

I was just writing this out! I believe practice in unit testing is what let me get this question right. I almost submitted n^1, n^2, n^3 before I realized I didn't get any wrong answers and something didn't feel right. I credit this 'instinct' to having written thousands of tests, and trying to make them fail to make sure my tests weren't lying to me.

got tricked even if I tested 1, 1, 1

1, 1, 1 was actually the test case that clued me in that n^1, n^2, n^3 couldn't be the right solution.

n=1?

But I totally see why it would cause you to pause and rethink your original idea :).

The point is that it didn't work, so exponentiation was out the window.

My point was that 1,1,1 is 1^1,1^2,1^3 and actually fits the pattern :).

Right - by fitting the pattern but failing the actual criterion, it disproves the consecutive exponents hypothesis.

It's funny. Even though I got the right answer, looking back, I still see many holes in my tests. I only tested positive and negative integers. I didn't even think to try fractions, decimals, hex values, text, etc. and I was already expecting a confirmation bias test.

I mean, it also illustrates how training and systematic reasoning can improve these things. Whether explicitly or implicitly, I picked up certain skills and procedures for problem solving (from programming and math contests) that I now use by default. Trying a bunch of examples, coming up with a hypothesis, trying to disprove it, testing edge cases…

This doesn't mean I always use these—at the very least, I have to explicitly jump into "problem solving" mode—but it means they can be useful.

It's still a meaningful difference, and could very well apply to lots of things beyond this kind of puzzle.

Exactly. I'm just as 'No' averse as the next person but the 'No' I'm averse to is the one where you make wrong assumptions and it comes back to haunt you afterwards.

This is now the fourth time I've seen this exact rule. Another example: https://www.youtube.com/watch?v=vKA4w2O61Xo

It's probably not "fair" to say I got it in zero... but I did. :)

Now that the NYT has done it, this puzzle has probably attained enough popularity now that you really ought to change it up a bit now if you're going to run it yourself. Granted, the space of hypotheses as simple as "increasing/decreasing" is pretty small, but your ability to fool people with the first sample run is almost unbounded, so that helps.

Similar story here. I got it in zero because this problem shows up at the early part of HPMOR.

I suspect that the basic idea behind it is about right (people who insist on failures before committing to a theory will probably do "better"). But it seems to me that this test will be best at selecting people who've seen it before and can pretend they didn't (or even remember to ask negative questions when someone asks you to guess three numbers to get the job).

Yeah, I'd like to see other sorts of questions that you can only get right by looking for disconfirming evidence, but that don't have anything to do with choosing a sequence of numbers.

If you look at the raw answers, a lot of people who guessed the wrong answer according to: "What answer did you give?" said they got it right:

Right? What answer did you give?

Yes, double the previous number

Yes, Each number much larger than the previous

Yes, sequence must always increase by 1

Yes, Powers of 2

Yes, h = 2n, i = 2(n+1), j = 2(n+2) . n is an integer

Yes, Powers of 2

There's a selection bias - those of us who got it right are more likely to fill that out (:

As a result we can't really rely on overall accuracy, but we can break it out by yes/no to account for the selection bias to get a profile for how a HN correct and incorrect differ.

This crowd needs one more question:

How many "I know the answer, but can I find a flaw in their code" questions did you ask?

to be pedantic: "monotonically increasing" is incorrect. It should be "strictly increasing"

Cool idea, thanks!

Personally, I spent most of my answers playing with the inputs. The form was happy to report that "1e1, 15, 0x10" was a valid sequence. :D

This is a good idea. I submitted the survey but here are my tests: http://imgur.com/hYKudyl

FWIW, I've seen this kind of game before and I was expecting it to be something simple.

My answer was: "The sequence is of an increasing real variable, where each subsequent value is greater than the preceding value. It's monotonically varying."

When I clicked "I think I know it", nothing happened. I don't want to click their "I don't want to play; just tell me the answer". But it seems like the right answer. I can't answer your form question if it is the right anwer, since I haven't clicked on their link and don't know for a fact whether it is or not.

Although I used the wrong term, it's strictly increasing.

Your last question is "have you seen a test about confirmation bias before?" But the text of this test says that it's about "why no-one likes to be wrong", which means it's pretty obviously about confirmation bias (and therefore that the test-taker should be wary about just confirming their first intuition).

[ 456 , 42 , 48 , No ] [ 1 , 2 , 3 , Yes ] [ 3 , 2 , 1 , No ] [ 45 , 46 , 47 , Yes ] [ -1 , 999 ,1238978 , Yes ] [ 1+1 , 1+2 , 1+3 , Fail ] [ 7 , 654 , 653 , No ] [ 7 , 654 , 655 , Yes ] [ 1.1 , 1.2 , 1.3 , Yes ] [ 1 , e , pi , Fail ] [ 1 , e , 42 , Fail ] [ -1.1 , -1.0 , -0.9 , Yes ] [ 2i , 3i , 4i , Fail ] [ 1 , e , pi , Fail ] [ 0 , 1 , 2 , Yes ] [ -1 , 0 , 1 , Yes ]

If anyone's tallying

Intersting that the split of correct/wrong answers from the HN crowd is 78%/22% - the exact opposite of the general population : 22% / 78%! The HN community does think different :)

Or at least, the portion of the HN community who reported their results, reported that they think different. :-)

Yep, I'd guess this is a massive bias. The people who get it right are much more apt to report and show how smart they are

If you're taking that from the article, I think you misread it.

> Remarkably, 78 percent of people who have played this game so far have guessed the answer without first hearing a single no.

Some of those 78% probably got it right, and some of the remainder would have got it wrong.

Some of those 78% probably got it right...

In what sense? How did they know that e.g. (1, 1, 1) didn't work, unless they tried it?

In the sense that they guessed the rule correctly. The correct answer was the dominant hypothesis in my mind before I tried any sequences, and others here report the same.

The article doesn't actually specify that 78% guessed correctly or incorrectly, just that 78% guessed without ever entering an invalid sequence.

It just means that this test was described at least several times in different "computer" media over last few years and almost everyone has read about it. Same thing with all "logic" puzzles.

I wanted to answer "Probably" to the last question about having seen a similar question before.

I don't specifically recall seeing one, but it is likely that I have.

Guessed after just one "No"

  3  9 27  yes  (is it exponential series?)
  4 16 64  yes  (is it only odd numbers?)
  5  7  9  yes  (is it any numbers of the same parity?)
  6  7  8  yes  (is it any set of increasing numbers?)
  6  7  6  no   (just to confirm that it's x<y<z, and not something like x<=y<=z)

-2 -4 -8 (is it increasing distance from zero?)

You're right, I made the assumption that our domain was natural numbers- which seems to be an uncommon one so far among the HN crowd

6,7,7 or 1,1,1 satifies x<=y<=z not 6, 7, 6

It would be relevant to include a histogram of #yes_answers - #no_answers in the summary, to test whether people are biased towards positive rather than negative tests. I think the raw data suggests that it does although I totally failed to create a histogram in google spreadsheets within 5 minutes.

My fuzzing went:

8 4 2 (no) 1 2 3 (yes) 1 1 1 (no) 1 100 123 (yes) 1.0 1.1 1.2 (yes)

answer: incrementing numbers

Thanks for setting up the survey!

Could you perhaps move to a bar chart instead of pie charts?

Don't think that's an option for Google docs - but feel free to take the data and share one.

Reply on news.ycombinator.com