The berkeley paper completely writes this off without evidence or reason, even with his assumption that offenses will not become more conservative with play calling, and its obvious a defense and crowd would act differently on 3rd and 3 than 4th and 3:
"Since it seems unlikely that the defense has substantially more scope than the offense to affect the distribution of outcomes"
Primarily, the goal on defense is to get off the field, so other than special situations like goal-line or the edge of field goal range, its ok to give up 3 yards and even 4 is fine. You really just need one great play out of 3 - a blocked or missed pass, tackle for loss, etc. and they're 3-and-out.
The paper admits the entire basis and math would be wrong if his assumption was misplaced:
"Thus using third downs to gauge what would happen on fourth downs would lead to overestimates of the value of going for it."
On 4th down and 1-3 yards, everything changes. You try to play every play as hard as possible, but you bring something special when its 'us or them, right here, right now'. Thats partially why (the other reason is there is less room to pass) that you see teams march down the field and can't get the last yard 4 tries in a row. I know the offense also steps up, but in a short running play against relatively even teams, defensive players are usually shorter (leverage) and have an extra player (unless the offense does wildcat with a mobile quarterback).
The other missing piece is they didn't seem to look at time left in the half and timeouts left. Punting in the first quarter is not the same as punting with 37 seconds and the other team is out of timeouts.