Using computer vision to help win $1M in Mountain Dew's Super Bowl contest | Hacker News Reader

Using computer vision to help win $1M in Mountain Dew's Super Bowl contest

blog.roboflow.com

234 pointsrocauc92 comments

Alternate idea: create a bot to open new twitter accounts and tweet ever increasing numbers until you win.

EDIT: Bonus points if you use another bot to exclude all guesses that have already been submitted by others.

This was my first thought.

Also, I didn't read the fine print but do you even need to create a new Twitter account? Do the rules say one guess per customer? All I see is "Tweet us the correct number."

I'm 99% sure there will be some human input on the final selection. Probably will be biased towards an account that's active and a little popular.

Yeah I was imagining some kind of Pepsi Point Harrier Jet loophole.

Absolutely not. Could you imagine the backlash if it's found the first person who tweeted the correct number was passed over for an influencer, or a more popular account?

Most contests do this.

In most contests, the public can't independently verify the winner

No such thing as bad publicity

> Also, I didn't read the fine print but...

It's disqualified in the fine print.

"Always read the fine print, but don't bother with the EULA" - Rudy Giuliani

“The large print giveth and the small print taketh away.”

The reason for that fine print is:

http://hoaxes.org/archive/permalink/the_caltech_sweepstakes_...

This incident, as well as the earlier Frito Lay sweepstakes that was also hacked by Caltech students, was famously referenced in the movie Real Genius: https://filmschoolrejects.com/5-brilliant-things-you-should-...

RG has more than that. The one where they made an ice rink in the dorms actually happened. The water slide one was also done in Page House when I was there.

Caltech refused to allow them to film on campus, fearing it might put the university in a bad light. So the film makers found some other buildings with the same architecture as used at Caltech, and they hired techers as extras to wander around in the background (saying they were unable to duplicate the look of a techer, so the easy solution was just hire them).

Here's the fine print. One account per participant.

> NO PURCH. NEC. Begins when Sponsor Tweets this commercial on Sponsor’s timeline at https://Twitter.com/MountainDew (no earlier than 2/7/21) and ends when a verifiable entrant Tweets the correct number of Mtn Dew Major Melon bottles, regular or Zero Sugar, (including 3D, drawn, neon, metallic & any color) that appear in this commercial or 3/31/21, whichever comes first. Limit one Twitter account allowed per participant. Winner must be verified. Subject to complete Official Rules, at lifechangingDEW.com. Open to legal res of 50 US/DC, 18+ (19+ in NE&AL). Potential winner subject to background check per Official Rules. Void where prohibited. To Enter, visit Twitter.com/MountainDew. Sponsor: Pepsi-Cola Company, 1111 Westchester Avenue, White Plains, New York, NY 10604. The NFL Entities have not offered or sponsored this promotion in any way.

Maybe you could put together a syndicate of 1,000 people who coordinate to tweet increasing numbers over a short timeframe based on a rough estimate, and then split the prize? (Of course, that would be pretty logistically complicated, including the timing and getting all the people to trust each other.)

Known in game theory circles as Cena Equilibrium.

Hi, I just googled Cena Equilibrium to learn more and the result was this thread. Do you have the name right?

You can't see it

>"Winner must be verified."

Is that Twitter blue check verified, or some other form I wonder.

I assume it means they need to verify your ID and have you sign some legalese. They arent just going to mail a check to whatever address the winner DM's them.

Since this doesn't seem to have posted yet:

"Fans will have three attempts to give counting Major Melon bottles their best shot." [0]

The official rules confirm this:

"Limit one (1) number guess per Entry. Limit three (3) Entries (via Twitter)/per Twitter account during the Promotion Period. Limit one (1) Twitter account allowed per participant." [1]

[0] https://www.mountaindew.com/world/be-ready-to-count-this-sup...

[1] https://www.lifechangingdew.com/rules

You would be disqualified for using more than one account.

Only if you get caught.

Says they’ll do a background check. For a million bucks you bet they’ll figure it out.

I wrote the software for a competition 10 years ago. The first prize was about $5000. All the top ten entries were cheaters who spent hours clicking on Like links on social networks. The behavior was very apparent from the logs. Disqualified. Next top ten, cheaters again. One of them at least used curl :-) Disqualified.

Imagine if the prize was one million.

how far down the list did you have to go to find the first legit winner?

They started to show in the third iteration. People with a number of likes as if they had real friends, not bots or a carpal tunnel.

Each iteration took a few days because cheating manually takes some times to get into the high thousands of likes.

The lesson was that in this kind of competitions you don't test everybody for cheating. Only the winners. If the last one in the rankings cheats, who cares

It wouldn’t take much investigation to prove you broke the rules, and for a million dollars I bet you they will do the investigation.

This is a modern version of the 1975 Caltech Sweepstakes caper. I think that was the first use of a computer to enter a sweepstakes.

http://hoaxes.org/archive/permalink/the_caltech_sweepstakes_...

I would guess the best strategy for this is to simply make an educated guess, soon.

There likely will be way more competitors (at least a hundred thousand, I would guess) than possible halfway decent guesses (maybe a few thousand?), and it’s first past the post, so having the correct answer isn’t enough; you have to beat the other correct answers.

When I was 8 I won a 72 pound pumpkin by randomly guessing "72" in a "guess how much this pumpkin weighs" contest. I vaguely remember pumpkin things for a long time thereafter. Not sure how my mother felt about it.

Another trick would be to take the average (excluding outliers) of as many guesses as possible since they're all on twitter. The Wisdom of the Crowd should get very close.

https://en.wikipedia.org/wiki/Wisdom_of_the_crowd

It seems that would be weighted however towards a too-low count of bottles

The median is probably a better estimator.

You won't win if you pick the median since by definition someone else has already picked that number.

technically it could be an even set and you have to average the middle two

Prediction: Mountain dew will announce a winner. Then, they will have to demonstrate/prove it. Then, computer analysis will show that they miscounted/or the number will be disputed.

I was thinking about this also but I assume all but maybe a small countable number were CGI so it would be really easy for them to get right, especially if they know it’s important.

Unless there was a CGI mesh in the scene, but due to how everything laid out it never actually got any screentime. In which case there would be some debate as to whether it should be considered.

I believe that the way the rules lay out the definition of what counts as a bottle is very specific.

I'm sure that they have a custom ray tracer that rendered the scene. Every bottle has 2 parts that must be seen at some point in the video(not just at the same time), and reflections count again. This implies that they have a big grid, with all the bottles(top and bottom) on one axis, and all the reflective surfaces on the other. Every time a ray hits a bottle, check the appropriate boxes for the combo of bottle and surface. After rendering you'll have the ground truth answer they are looking for.

I tried applying the wisdom of crowds idea instead :D https://twitter.com/SmittyW62858649/status/13589265825139957...

Not sure if taking a median is the best idea. I figure a gamma distribution might work better!

Were there any gaps in the guesses? You have to be first to guess the correct answer so taking an un-picked lottery number in the vicinity of the median would be a decent strategy.

Yep looks like there are but are in the higher range of the distribution (>300). Updated the post with some interesting plots.

I would really like to use a z-score metric to remove the outliers and fit a gamma distribution, but unfortunately its a busy Monday!

I tried this a few years ago with a jar-of-coins problem on Twitter. I think it was Orange in the UK, the winner won the value of a tube full of pennies?

You were allowed X guesses per day, per account (so I roped a few friends in). Since the guesses were public, it seemed like a good opportunity to try the theory out. The closest guess would win. The strategy was pretty simple: use your guesses to bracket other people's answers. So if someone guesssed £100, you would guess £99.99 and £100.01. Of course someone might guess exactly, but I don't think that happened (or maybe it did?).

I wrote a script (I think in PHP of all things) to poll the hashtag and put all the entries into a database, with some code to figure out what ranges we needed to cover each day. There may also have been some heuristics to account for the fact that you couldn't possibly cover everyone's guesses, but you could try to maximise the range of values that you would win for. I think we also tried to weight the suggested guesses around the median value.

In the end it didn't win, but it was a fun exercise. I think the winning answer was very close to the median.

That’s pretty cool! I’m going along in the same vein. I know my median guess was taken first so I can’t actually win but I just wanted to see how close I could get out of curiosity.

I am surprised the median was an accurate estimator in your case but I suppose it makes sense if you are not close to the bound of zero (I.e. it start approximating a normal distribution)

https://www.lifechangingdew.com/rules

"The Twitter account must be set to Public/Unprotected"

"Go to your Twitter Account, Settings, Privacy and Safety, Direct Messages, and make sure “Allow message requests from anyone” is checked"

"the potential winner consents to such a background check"

"the prize will be divided evenly among all verified Winners"

"Proof of sending or submission will not be deemed to be proof of receipt by Sponsor"

"Winner grants permission for Sponsor and those acting under its authority to use their name, and address (city and state), photograph, voice and/or likeness, for advertising and/or publicity purposes, in any and all media"

Yeah. No.

I thought this was the best ad of the superbowl. Probably the only one I actually remember. I've heard a lot of people say the commercials have gotten worse over time, but I think it's actually just that we hate commercials a lot more than we used to.

One thing I really hate which is growing in superbowl commercial ethos is the advent of meta commercials, and expanded cinematic universe commercials.

On the other hand, the superbowl is the only event I watch for an entire year that has actual TV-style commercials anymore.

I liked the Edgar Scissorhands commercial the most, followed by this one. The Reddit commercial was good, too. I thought most of the commercials were pretty awful this year.

There’s no accounting for taste! I thought the scissor hands one was cringe worthy ! Weird us humans.

Nostalgia pushed the right buttons for me, I barely watched the commercial and just thought about Edward and his family

The paralympian swimmer ad made me tear up a little bit.

Ironically the commercial I enjoyed the most wasn't one of the official Superbowl ads but one of the regular tv ads that happened to be playing during the pre-game stuff; specifically "Becoming your parents" ad where they are threw away the "Live, Laugh, Love" sign

Their annotation methodology is flawed:

From the rules: > Correct Bottle Count will include bottles shown from the cap/top of the bottle to the bottom of Mountain Dew label

Also, I'm pretty sure it thinks John Cena's right ear is a bottle ¯\_(ツ)_/¯

You even caught the bottles reflected in the sunglasses, incredible!

Totally; the model does a good job finding what you teach it to find!

On the flip side, someone in the YouTube comments mentioned that we missed the ones on the wheels and license plates. So I added them to the dataset this morning and kicked off another train job.

The AWS bill is $2 milion

I also noticed some false positives. It is a good demonstration of the difficulty of the task: there is no easy, automated, out of the box solution that will arrive at the correct answer. You could make a decent approximation if you could correlate frames and track individual bottles but even that is beyond the scope of this demo.

Not to mention the 90% recall, meaning that the model is missing 10% of the bottles in the annotated data.

Cool blog post, cool tutorial, and great tool but its not going to win any contest where you need to guess an exact number.

Are those two seperate bottles being reflected once, or one bottle being reflected twice?

Good question. And do they count as separate bottles if the un-reflected version is shown in a different frame?

The official rules state that each reflected bottle is its OWN "counted bottle".

I was otherwise rather confused by the "count each bottle only once": there are a few in marquees and such where it's hard to tell if it's a repeat or a new bottle, frankly.

Not familiar with Roboflow. Is it something like OpenCV as a service?

We make developer tools for computer vision so engineers can create models without being machine learning experts: collect data, organize images, annotate, train, deploy, and improve.

That is a very bad pitch. I'm doing computer vision all day long but I still don't know what service you offer and/or if it might help me.

Maybe that's your experience because you're into the matter, for me, judging from the screenshots and explanation, it seems like Scratch-like/no-code CV which seems pretty cool.

I was referring purely to the text pitch here on HN.

Ah, yeah, that one liner resonates with teams establishing their vision infra. Teams build production vision models in the span of an afternoon with better tools, e.g. one dev built this Mountain Dew bottle model during Super Bowl half time.

Some ML engineers find value in things like automated annotation, testing model architectures (models.roboflow.com), and having one-click deploy for custom object detection APIs. Think of it like replacing all the one-off scripts so you can focus on your domain-specific problems instead of reinventing the wheel on vision infrastructure.

I’d imagine if it’s targeted at non ML experts then it wouldn’t be of much help to you,

It should still be UNDERSTANDABLE to ML experts even if they are not the target customer.

IMO Making your pitch non-understandable is just a bad idea ...

Or maybe they don't care to make a pitch as in sell you something, but just wanted to briefly explain it to that other dude?

A chance response to a comment on random third party thread with a quick explanation.

You: “very bad pitch”

Wouldn’t life be great if every comment was an ad?

Is roboflow a good way to do this kind of task? What is roboflow?

It sure is! Roboflow builds tools that enable any developer to use computer vision without having to become a machine learning expert. (Kind of like how you don't have to become a cryptography expert to add auth & accept payments in your app.)

For transparency, it might also be worth adding a disclaimer that the product you are advocating is produced by your current employer (even if that did not consciously play a roll in your decision to advocate for the product).

Its not clear to me from the walk-through how they ensured they were only counting each bottle once? Is that something they automated or is it left as an exercise for the reader?

How would you go about automating the unique count when you are sampling the video 3 times a second?

No, it just gives you bounding boxes. You still need to manually go frame by frame and count all unique boxes.

I saw their promoted tweet about this contest the night before and primed a tweet for when it aired live. I had a number just pop into my head: 206. Number of bones in the human body, felt good enough for me.

I just checked, and, to my best ability to determine, I believe I was the first legit 206 guess.

When I went and did a basic manual count, I came in closer to 216 or 220 or so, but it's really hard to tell, and when I did a painstaking frame by frame count, I came in closer to 221, but there are a few I'm _really_ confused about in terms of being a repeat or not.

And counting the bottles coming out of the trunk was hell... I used the promo tweet where Cena is counting out loud and up to 36 as a shorthand for one of my guesses. In my frame by frame analysis, I counted 45 but 3 are flying out when it first opens up, and then 42 more join in the next cut.

Similarly, on the final look back after the end marquee, there are 2 on the side that I don't know if they're recounts or not, and then the gearshift and flag itself are... two, which put me closer to 224/225. There's one other scene where I wasn't sure about as many as 5 possible dupes, though.

The "don't count the same bottle twice" gimmick is infuriating because it's really difficult to ascertain what's bedrock in the scene. They use the roller coaster example as 3 that persist throughout the entire commercial, so all bets are off.

Here's my notes if you want to insanely check my work:

Car: 4 wheels License plate: 5

36 opening frame

2nd scene: 47 total

whack a mole: 66 total

birds:..13 actual birds - 5 architectural = 18 = 84 total

overhead car: 1 in cupholder = 85 total

passenger side car: 7 = 92 total

dance hall dog: 13 color changing, 2 ticket booths, 1 necklace = 16 = 108

facing forward car marquee = 5 right neon = 2, roller coaster cars = 3

right parking neon = 3

left building neon = 3 left billboard = 1 left marquee = 3 = 20 = 128

facing car: hood = 5 right marquee = 2 pakace = 1 left box office = 4

passenger holding: 1 = 141

passenger bottle = 1 behind = 8 = 150

key chain: = 1 cena hold = 1 = 152

sunglasses: 2 = 154

cena talking = bg 2 = 156

back of car scene: 8? 5 possible dupes

trunk first open 3 fly out

trunk wide shot right side lit marquees: 7

left neon = 1

building neon = 2

flying out = 45 - 3 flying out already = 42 = 219

final marquee: 2

final look back: 2 on side?

flag post and flag content = 2

It missed ones on license plate

Good catch; someone on YouTube called this out this morning and we released an updated dataset (we had also missed the ones attached to the wheels): https://public.roboflow.com/object-detection/mountain-dew-co...

Really hoping someone uses this to win the contest!

It would be funny if the reflection in the sunglasses only count as 1,

The title is missing a key word "help". The prize doesn't appear to have been won yet- not sure if the original article changed their title.

Ok, we've put help in the title above.

I was character limited - thank you

Don't the rules of this content disqualify something like this?

Well if you've already annotated all the bottles in the video then why do you need to train a model to do it?

to catch the ones that were invariably missed due to optical illusions etc.

wouldn't it also count reflections, for example, which are not real bottles to be counted ?

That's where I think they are going to use their discretion to "pick" the winner, despite somebody else having been just as correct.

For instance, they say the one he is holding in one scene isn't counted again in a later scene.

What about the reflection in his sunglasses? Is that 1 or 2 (or 3 if you count the source of the reflection wherever that is).

Rules specifically say reflections are counted, even calling out an example of reflections in sunglasses would count as additional bottles.

And un-count the ones that appear in more than one scene, as mentioned in the article

He only hand annotated bottles in about 10% of the frames of the video. Those were used to train the model, which could then be used to annotate the rest of the frames.

Reply on news.ycombinator.com