I know I said Game Dev, and games are a fun thing to program, but I honestly could be working on Foo or Bar and I'd have a good time with it.
You mentioned the MIT free courses and such - which believe it or not, I've done the majority of. I only bothered with MIT though because they had a lot of different courses and they were all reliable and in the same place. For a time I was interested in compilers and such, so I read the Dragon Book (Or a good helping of it. That's one ugly book). I've also read a few other technical programming books that I know you have to read in college. Despite this I think I'm leaning towards going to college now for many of the reasons you stated (background, networking, general knowledge). Thank you for your answer it is undoubtedly life changing.
Where did you go?