One possibility is that he didn't restart the program between retries, and the memory in question was already allocated. Another possibility is that he only ran handbrake and nothing else, and the OS was in more or less the same state both times. It could be that the problem was triggered by stack allocations rather than heap allocations and the video block in question caused a large-ish recursion that hit the problem, and would be likely to hit the problem no matter what was running since it's somewhat rare to have large stack allocations.
Chances are it was actually none of those things, but they're real possibilities anyway.