Yes Prime Minister, questionnaire design matters | Hacker News Reader

Yes Prime Minister, questionnaire design matters

ipsos.com

72 pointsadmp72 comments

I had a survey construction class during my MBA. This is mainly obvious stuff, but indeed should be studied and practiced because people screw this up all the time. Assuming your goal is to find out what people think (vs, have the survey tell you what you want it to tell you) - you have to realize that most people plop down their at-the-moment emotional state as the survey response, and that state is most recently influenced by the survey itself.

Eg, if I asked you "should Israel dismantle the Hamas terrorist organization responsible for the 10/7/2023 murder of 1500 people" - most people would say "yes" because the words "dismantle the terrorist organization" resonate with most people.

If I asked them "do you support a ceasefire in Gaza" - most people would also say yes, because the word ceasefire would resonate with them.

Obviously these would be contradictory results. What this survey wouldn't tell you is how many people actually think about this topic and which way they lean.

A better designed survey would be something like this:

Question 1: How aware are you of the current conflict in Gaza (1-5) Question 2: In your own words, how would you describe the conflict? Question 3: If 1 is "Ceasefire at all costs" and 10 is "eliminate terrorists at all costs", where would you place your opinion?

Having these 3 datapoints would enable you to filter by those who actually have an opinion (or, find out what percentage has an opinion), use question 2 to validate that and/or gain new insights and use 3 to identify how strongly people lean one way or another.

Do note that your two example questions aren't quite as diametrically opposed as you make it out to be. You can be in favor of both and still be consistent.

How you would dismantle Hamas while having a cease fire in place is left as an exercise to the reader.

Not only that, but "do people consider these options to be mutually incompatible" is itself interesting information.

I wouldn't expect the average survey to handle it too well, though.

If, like me, you can't read the questions in the results table, here is a legible version: https://datawrapper.dwcdn.net/jZ5sT/full.png

Wait there is barely any difference in the response to the last question in both surveys!

This is a problem with the way the data is presented and the questions asked, but there's a 10 point swing between "not opposed" and "in favour", and likewise between "not in favour" and "opposed".

There is a 15% difference, and that takes the overall results past the critical midway position that matters in a referendum.

I'm afraid they wasted their sample. It's a large company, so it's probably filler for them, but still. If they had asked the same question at the end, the effect would have been clearer (assuming no big differences in audience, because that really happens). Now you can ascribe the effect to e.g. people misreading the "oppose" question.

I had a similar thought. They could otherwise have asked the two end questions from a cohort without the lead-up to get the baseline.

The questions were flipped in the original TV programme.

I think it is not representative anyway because of how famous the sketch was.

This is a feature not a bug in questionnaire campaigns.

> "First of all, to be clear, we would never ask questions on such a topic in this way."

I love how they worded that. Having once worked for Ipsos, they DEFINITELY do it, but the sentence is correct, I don't recall any instances of it being done "on such a topic."

The art of pollsters is to produce results reflecting the hopes of whoever commissioned them, but with just enough even-handedness to look not-obviously-prepackaged.

This show and its predecessor “Yes Minister” are excellent. And as we see, just as relevant today as ever.

I wonder if people in politics or civil service feel the same about this show as the people in tech feel about Silicon Valley series.

These series are also scary because they somehow managed to remain relevant. Maybe the way the world works hasn't changed that much after all.

Us humans make the (human) world work and we haven't changed a lot in a very long time indeed.

If you like the series then this article[0] which was published in the LRB early in its run is well worth a read - it gives you a good idea of how the series came about and why it feels very closely observed. Reportedly many politicians at the time, including Margaret Thatcher, felt it to be very closely observed.

But the really clever thing about the writing is that you never know which party Jim Hacker is a member of and it is equally believable regardless of which one - it's so remarkable that it's a series about politics which manages to be really quite apolitical in that sense.

0. https://www.lrb.co.uk/the-paper/v02/n10/antony-jay/informed-...

And/or people have started behaving according to the standards set in the show because they assume that everyone is that cynical. This is kind of what worries be about these extremely cynical satires and depictions.

On the other hand, it's not like politics is better in the US which had The West Wing, which is very aspirational and not cynical at all so idk...

I mean if you want to talk about cynical, Veep really went off the deep end as a comedy, which was good for the show, but also unfortunately life is imitating art.

It's not my impression that Veep was/is all that widely known? I just heard about it because it's the US version of The Thick of It, but then again I'm not in the US. And just like the US version of House of Cards, it's also much newer. There isn't that long of a tradition of cynical satire as in Britain going back to at least the 80s (Yes Minster, Spitting Image, House of Cards, The Thick of It).

Based on anecdata from people in the UK and Australia, the Australian show Utopia [1] is the most realistic take on the civil service while The Thick of It is the most realistic take on party machinations.

Some civil servants find Utopia too painful to watch because it hits too close to home. The show has made several very prescient predictions [2].

[1] https://en.m.wikipedia.org/wiki/Utopia_(Australian_TV_series...

[2] https://www.news.com.au/sport/sports-life/abc-comedy-series-...

After working at the BBC, I can tell you that W1A is very true to life - though being film there also helped. https://en.m.wikipedia.org/wiki/W1A_(TV_series)

God bless Ian Fletcher and Siobhan Sharpe.

I wonder how Twenty-Twelve compares to reality

I don't agree with the conclusion; 3% difference is basically within the error margins for these types of things. I typically assume a ~5% error margin, which is nice round number I just pulled out of my ass, but it seems roughly accurate.

Re-read the question. The first one is 45-favor, 38-oppose, the second one is 48-oppose, 34-favor.

No it isn't. The first one is 45-favour, 38 don't favour. That's a totally different thing.

This proves the idea that you can get a result that points the way you want it to. But it doesn't quite prove that the leading questions have any effect.

To do that they would need to ask 2 more sets of questions with the "oppose"/"support" questions inverted. Otherwise the effect could just be due to the different wording of the final question.

You're misreading the answers. In sample A, 45% are in favor of reintroducing National Service and (38% oppose it). In sample B, the phrasing is reversed, meaning that 34% are in favor (and 48% oppose it). So, the difference is 11 percentage points.

> (38% oppose it)

No, you're misreading too - see my other comment.

Eh right, do'h >_<

But in that case you need to add a control for both questions without the prior four questions. How do you know the difference in phrasing doesn't matter?

Reply on news.ycombinator.com