Using Prediction Markets to Track Information Flows: Evidence from Google [pdf] | Hacker News Reader

Using Prediction Markets to Track Information Flows: Evidence from Google [pdf]

stat.berkeley.edu

139 pointsamasad19 comments

Having now run prediction markets inside companies for over 10 years, we can definitely backup the findings in the paper - especially the revelations of bias among different types of employees. We usually ask people to participate anonymously, and it's often the only outlet people have to express what they truly think.

It's always a surreal experience to sit in a meeting with people and their bosses and hear "we're on track, no issues, the new product launch is going to be great, we're going to sell a ton, etc. etc." and then have a private/informal meeting with the same people who will confide that the project/product/strategy is completely fucked and everyone thinks that. :)

Can you share any of the companies you've run prediction markets for? I'm curious.

The U.S. Intelligence Community does quite a bit of research in this space that we're involved in, we're also working with pharma, energy, large manufacturing. The profile (in the commercial space) tends to be very capital intensive orgs/projects, where if you can use a prediction market as an "early warning system" by surfacing what people actually know, you can potentially save them a lot of heartache (and money and pissed off/laid off employees, etc.)

Is this a consulting service? Or do you offer software? If you don’t mind me asking, what’s the average project size using a prediction market.

GP has http://cultivatelabs.com in their bio, they're working directly in this space. Very interesting stuff. (And, looking at previous comments they're hiring)

We have a prediction market platform, but we also end up consulting most of the time as part of the implementation. As you might imagine, prediction markets can be quite disruptive. They tend to make information radically more transparent and we learned the hard way that in order to not just churn through a bunch of companies, we needed to help companies deal with 1) what key questions to ask, and 2) what are they going to do with the new signals they have? How will they incorporate them in to their decision making processes and how will they acknowledge that the "crowd" has been heard?

The average project size we work with tends to be in the low hundreds of participants, with the biggest one being in the mid-thousands.

Thanks, this is helpful. I read through your site and have a hard time knowing what prices would be for an install or an engagement. I know it’s not clear to have a price sheet for consulting, but is there a way for a potential client to learn about high level prices? Or they have to just enter the sales cycle.

I want to to figure out if this would be a fit for one of my projects, but I don’t want to get a demo, talk to people, etc just to find out it’s an incompatible price.

I understand - I usually don't like to have to talk to people either. Since most of our projects are with large companies, we haven't shared pricing, as each one tends to be pretty different. But we've also worked with smaller teams who just want access to the software, and that's fine too. We start at 399/month for those types of projects and that's for unlimited everything for up to 100 people. You might consider playing with TinyCast which is our free, single question prediction market and see if you like it and people participate. If you/they do, that would be a good indication they'd likely do a full blown prediction market. https://tinycast.cultivatelabs.com

How do you deal with the cultural challenge of convincing people there's value and selfish benefits to having more transparency?

I've noticed that in large organizations certain information are held by a small number of people who may be reluctant to share them for political reasons. Another challenge with internal prediction markets is the possibility that knowing the "probability" of an project-related outcome reduces morale and hence creates a self-fulfilling prophecy that further reduces the chance of success. Appreciate your insights.

You might be right about the self-fulfilling prophecy. But this effect might already be in effect, as people might already have a gut feeling, while nobody cares (or knows) about it.

A trivial instance of this would be release planning in software. After our initial delivery plan my team gives a confidence measure, one release it was quite low, which was an important signal since we could discuss why that was and what we could cut to increase the confidence. Without that signal, we'd have overcommitted, and suspecting as much but without common knowledge might have even delivered less.

You're right. But I think the single-value, hyper-transparent nature of prediction market can create an institutional feedback loop that's more powerful than what arises from the diverse, individually held opinions that others don't know about.

YOu're right, this is indeed a challenge. But the benefit is often they're not getting screwed by bad planning, over promising, etc. Afterall, those are the kinds of decisions that really burn people out. Instead, they finally get a chance to have a say. Also, I'd argue all the information you're talking about that is potentially decreasing morale is already out there, it's just not being officially recognized/aggregated. At least this way, it's surfaced, and management understands what the perception is out there and can address it head on.

Bottom line, these aren't for everyone. One of the first questions I ask when someone expresses interest is what their culture is like and how humble their leadership is. If they're ready for it, prediction market can be transformative. But if they're not, they can fail - and we've surely had plenty of those projects too!

Conclusion (emphasis is mine):

In the past few years, many companies have experimented with prediction markets. In this paper, we analyze the largest such experiment we are aware of. We find that prices in Google’s markets closely approximated event probabilities, but did contain some biases, especially early in our sample. The most interesting of these was an optimism bias, which was more pronounced for subjects under the control of Google employees, such as would a project be completed on time or would a particular office be opened. Optimism was more present in the trading of newly hired employees, and was significantly more pronounced on and immediately following days with Google stock price appreciation. Our optimism results are interesting given the role that optimism is often thought to play in motivation and the success of entrepreneurial firms. They raise the possibility of a “stock price‐optimism‐performance‐stock price” feedback that may be worthy of further investigation.

We also examine how information and beliefs about prediction market topics move around an organization. We find a significant role for micro‐geography. The trading of physically proximate employees is correlated, and only becomes correlated after the employees begin to sit near each other, suggesting a causal relationship. Work and social connections play a detectable but significantly smaller role.

An important caveat to our results is that they tell us about information flows about prediction market subjects, many of which are ancillary to employees’ main jobs. This may explain why physical proximity matters so much more than work relationships – if prediction market topics are lower‐priority subjects on which to exchange information, then information exchange may require the opportunities for low‐opportunity‐cost communication created by physical proximity. Of course, introspection suggests that genuinely creative ideas often arise from such low‐opportunity‐cost communication. Google’s frequent office moves and emphasis on product innovation may provide an ideal testing ground in which to better understand the creative process.

Interesting, so if you want to cultivate a diverse set of opinions and ideas in your company, it's enough to place them on different floors – no need to move them to different cities.