How Coaches and the NYT 4th Down Bot Compare | Hacker News Reader

How Coaches and the NYT 4th Down Bot Compare

nytimes.com

50 pointstrevin44 comments

Tough to take everything into account, but I find it troubling that all non-success/failure outcomes are not part of the calculation (ex: 8-yard gain on a 4th and 10; a fumble on a 4th and 2 run, etc). It simplifies the formula to the layman, but I have to imagine that it adds enough additional variance to matter, especailly considering we're talking in variances of less than half a point in their stated example.

Also (and to their point), there is a ton of variation between offenses, defenses, punters, running backs, etc. Using this as a rubric is a nice idea for a rule of thumb for an armchair quarterback, but I'd strongly disagree with using it to accurately criticize any decision within a couple points of expected value.

This is especially true of 4th down plays. A surprise play-action pass on 4th and 1 can go for an especially long gain and thus should have higher expected value. There's also the fake-punt scenario, which I suspect tends to be more successful than a regular 4th down.

Absolutely. And I'm sure if coaches started trying to run more 4th down plays, that would dramatically change the algorithm as well.

These type of articles never take into consideration the reason why coaches don't go for it as often as a computer might. The coaches face a reverse moral hazard when taking risks because the negative outcomes are multiplied for them. Even if the statistics support their decisions, a coach will be fired for taking too many risks and failing. Is gaining an extra 2% (or whatever) in expected returns worth the risk of losing everything if the low(er) percentage outcome does actually happen? It is the same idea behind the whole "no one gets fired for buying IBM/Microsoft".

Until you get a coach's motivations more in line with the motivations of the team, you won't get a coach that is as aggressive as these computer simulations suggest they should be.

People always say this, but it is actually true? Has any coach ever been fired for missing too many fourth downs?

I can't think of one off the top of my head. Mike Smith was under a lot of pressure a little while back for his aggressive decisions on 4th down (that were statistically correct IIRC) but wasn't actually fired.

However, the obvious counter to your question is how many coaches really go for it frequently on fourth downs? Most coaches just follow the conventional wisdom unless they are already in a desperate situation like Ron Riviera or they have built up a lot of good will like Bill Belichick.

So it sounds like fans and the media need to grasp EV first before coaches will be allowed to take the appropriate risks.

Not even missing too many. Just missing more visibly, compared to coaches with equivalent records.

Ask Nick Saban how trying to kick a field goal with 1 second left in the Baylor game is working out for him. There are plenty of people saying that any other coach would be fired.

Nick Saban can call any number of schools right now and be the highest paid coach in NCAAF. He has a reputation for going for it on 4th down and taking risks. Also, it was Auburn, not Baylor.

That particular outcome had happened less than 5 other times in the history of the modern NCAA. 99/100 times that game ends with either an Alabama win or O.T.

He hardly made a controversial decision.

The bar that I was watching the game in exploded when that happened. You wouldn't believe the number of people calling him all sorts of names for making that call. And, I think it was a good call too. The point is that being risk averse is the right way for a coach to keep his job.

I think there's selection bias in the underlying data. The dependent variable (realized success on attempted 4th down conversions) is conditioned on 4th down attempts by coaches.

Since coaches are conservative, they're likely to go for fourth down conversions only in cases when they are likely to succeed (due to defensive mismatches, etc).

Hence the model may over-estimate the benefit of going for it on fourth down.

Moreoever, the probabilities are likely to change if behavior changes. For example fakes or trick plays become less successful if you do them often.

Why would you ever go for it on 4th and 1 inside your own 10 yard line? You would put yourself at a severe disadvantage if you failed.

Since the data from this is from previous plays in the last 10 or so seasons it is flawed. There probably isn't very little data of a team going for 4th and 1 and that deep in their own territory. The only time this has probably happened is in extraordinary circumstances - specifically in a game-winning drive scenario with time expiring, where the defense is in 'prevent'. In these cases it is highly successful in that situation but you don't have the time to get the score.

In place of 4th and 1 data AdvancedNFlStats would substitute 3rd and 1 stats. And I would agree that this substitution is suspect. Just a bit shy of bogus in fact.

Well, getting a yard in the NFL isn't typically a big problem and 3rd and 1 vs 4th and 1 are feasibly no different. But turning the ball over inside 30 yards is at least 3 points for the other team with only rare exceptions.

I think the biggest difference is that on 3rd and 1, the defense has already assumed that they're going to get a 1st down. The main job of the defense is to prevent a big gain and then get ready for stopping the offense on the next four downs. I wouldn't say that they're cheesedicking it, but they're not going all out.

Meanwhile, on fourth down, the defense is amped up and ready to roll. You'll usually see a strong blitz against a sputtering offense, and the percentage of plays that actually get a yard is going to be lower.

Not only that, going three-and-out means that the offense is unlikely to do any better going from the 10 to the 20 yard line. Better to punt the ball, trust your defense, and have your offensive coordinator fix whatever is going wrong.

I was thinking the same thing about 4th&short in your own red zone - generally the data points you'll get are from desperate late-game plays where the defence is also not playing to their standard playbook.

I think statistically, if you punt from the 9 yard line, the other team is going to get excellent field position (around the 50 yard line). The difference between failing on 4th and 1 from your own 9 and the other team getting the ball around the 50 yard line is 41 yards of field position.

That may sound like a lot, but I'd imagine the actual point difference is on the order of 2 points.

The more interesting thing to consider would be whether taking a safety there is ever a positive.

No. Because you cost yourself 2 points and you give the ball to them. Essentially you gave them 2 points at the cost of being able to "punt" the ball at the 35.

Ah, but sometimes two points is not so bad if the clock is a factor and you're about to turn over possession regardless. Down by one and facing a fourth down against the Broncos from his own one yard line with three minutes to go, Bill Belichick famously took an intentional safety[1]. The field position gain from the free kick was worth the two points, and the Patriots went on to win.

[1] http://sports.espn.go.com/espn/wire?id=1653659

Ah interesting, I have never heard about this game specifically.

It seems though he gave up some amount of EV by giving up the 2 points to the Broncos, thus a FG later on would only tie the game instead of winning it. But I assumed this was offset by the risk of having to punt and the Broncos end up getting a FG where then they'd be up by 4 points.

I don't know if Belichick was right to do this, but it paid off for him which is why everyone pats him on the back.

Contrast this with the 2009 Colts/Pats game where he went for it on 4th and 2 on his own 28 and didn't make it.

Exactly. Punt from the 1 and the Broncos will have the opportunity to burn clock and still likely kick a field goal, forcing you into a touchdown-required situation. As it was, the free kick netted even more yards than expected.

Going for it on 4th and 2 in the '09 game was the right decision, and few knowledgeable commentators disparaged Belichick for it. Manning was dissecting the Pats defense at that point in the game: He had a much greater chance of scoring a touchdown than the Colts defense had of stopping the conversion.

Now the playcall...

If you turn the ball over before reaching your own 30, you give the other team at least 3 points, because a field goal at that distance is trivial.

This bot also requires both coaches to follow it's recommendations, because otherwise the comparison between what they call "point values" of the two teams becomes meaningless.

This was discussed some in https://news.ycombinator.com/item?id=6728821

I've thought about it some since then and I'm also not convinced that expected points is a good metric. That will maximize the expected point differential per season but I am not convinced (although I could be wrong... I haven't put pencil to paper to make my thoughts rigorous) that there are enough possessions in the game on average to make this metric useful even in the first half.

The best defense is keeping the other team's offense off the field. Going for it on fourth down has a much higher probability of doing that than a punt does.

This does not take into account an important factor that is somewhat intangible, but real to the psychology of the game: point swings. If you go for it on 4th and 1 at your own 1 and lose you are definitely down 3 points and probably 7. This puts pressure on your offense--which just failed on a 4th and 1--to somehow make it all the way down the field next time. That can be demoralizing for that offense, paralyzing in fact. That is much harder to model, but an important part of a correct model, I believe.

This actually seems measurable. Do teams that are behind perform worse on average? Do teams perform worse than normal after failing a 4th down attempt?

The same statistics wonk has done a lot of work on "momentum" in games. His conclusion is that there is no such thing:

http://www.advancednflstats.com/2013/11/momentum-1.html

That jibes with my own anecdotal experience watching a lot of football, despite how often announcers like to cite it as a factor.

It's the same thing with penalty kicks in soccer. Statistically, you are more likely to score if you kick the ball down the middle. The goalie is going to go one way or another and there's a 50% chance he'll guess right. So why don't players kick it down the middle..because if they kick it down the middle and the goalie manages to block the kick, they'll come off looking like idiots and it would look real bad on the highlight shows.

There's an interesting hole in the data at 4 & 10 on the opponent's 36. In a sea of punting and kicking, this one data point has actual coaches going for it.

Also, looking at the overall pattern of the data, my gut tells me that the optimal rubric is somewhere in-between the 'bot and the meatcoach. There are some strange holes as well - at 4th&3, you should go for if you're on your own 8 or 9, but punt if you're on your own 7 or 10?

Without confidence levels, the differences are meaningless. Maybe that's the flip from 49% to 51%

That makes sense. At the 36, it is a 53 yard or so field goal. At the outer range of most kickers and pretty low percentage, so you give the ball to the opponent at the 40 something yardline with high probability. On the other hand, its also close enough to be very difficult for a punter to pin in inside the 5 or 10, so even if you fail you only give the opponent 16 yards over a punt.

But the holes at the your own end you mentioned, that makes no sense at all.

Hm, so the data points for likelihood of getting another first down is based on data from the fourth downs that coaches actually attempted? It may be that the fourth downs they didn't attempt were not as propitious for some reason - wind (which the authors mention), injuries on key players, lateral positioning which doesn't favor one formation or another, etc.

Agreed. The sample is very likely tainted by factors that made success much more likely than average if we believe that NFL coaches are both conservative (they forgo punts only when the odds are much in their favor) and rational (they are win maximizing).

Still, there is probably a case to be made that coaches should be a bit more aggressive, but that edge could be small enough that other factors (like perceived incompetence) dominate it.

The choice to go for a 4th down conversion is more likely than not primarily decided by the current score and time left in the game. However, the success of a 4th down conversion probably isn't heavily influenced by those factors (other than perhaps player fatigue), which would suggest that coaches are indeed too conservative.

is more likely than not primarily

This seems pretty loose. I'm not trying to be pedantic, but the theoretic flaws in the data are such that you would want a pretty tight logic in your model to have any level of comfort with it. The proportion of the state-space that is unobserved seems ~large and critically relevant.

No, it's based on third down data. The likelihood of converting 3rd and 1 is probably similar to that of converting 4th and 1.

Thanks! I read poorly the first time.

The fundamental flaw with this analysis is that statistics are descriptive, not predictive.

You can't reliably predict the future with statistics alone. If you could, then technical analysis of the stock market would be the most successful investing strategy. It is not.

Likewise, a lot of folks seem to think that taking an historical statistical trend and extending it into the future is a scientific approach to football coaching. It's not.

Scientists use statistics to understand evidence, but they formulate theories in terms of root causes. In other words it's not enough to know that dropping a hammer has a high correlation with acceleration of 10 meters per second squared. If you want to predict how fast other objects in the future will fall, you need a theory that describes why and how anything falls.

It doesn't look to me like the 4th down bot embodies or implements any cohesive theory of football success. It just finds trends and dots the line into the future.

Here is a high school coach from Little Rock that never punts, and always onside kicks. His reasoning is sound and it seems to work out for him.

http://www.youtube.com/watch?v=AGDaOJAYHfo

It is interesting what an emotional reaction people instinctively have to 4th down. Going for it on 4th down and failing feels like a catastrophe. But punting would have given the ball away anyway, so all you've done is give the opponent an extra 40 yards or so. Which is bad, sure, but it's a hell of a lot less bad than, say, throwing an interception on 1st and 10. But somehow giving up the ball on downs has come to be seen as a massive failure, on par with an regular turnover.

Turning it over on downs is a turnover, as sure as fumbling is. It's not accounted that way in the box score, but in terms of impact on the game it is the same.

An interception is better or worse than turning it over on downs by the field position delta. If it is returned past the line of scrimmage, then it's worse. If it is not returned, it's better. A deep interception with no return is like a punt in terms of impact on the game.

Reply on news.ycombinator.com