I played again but this time I drove my car backwards and the model does much, much worse in this weird situation. It often doesn't do what I input at all and keeps straightening itself out. And on kick-off you always go for it, even if you don't.
Playing backward is actually a great idea for testing generalization