Yeah, it's fairly simple to compile the Android source. Plenty of tutorials out there on XDA, Rootzwiki and from Google themselves.
Basically, one just has to grab the drivers you mentioned above and pull the source using repo/git and compile using Linux (Debian/Debian forks are the easiest generally to do it with) or OSX.
I compiled the source the day after it came out for Android 4.1 for my Verizon Galaxy Nexus. Ran it all the way up until yesterday when I switched to Cyanogenmod 10.
I'm an admin (yarly) at Rootzwiki, so if anyone has questions on how to build the source or mod it, you can head over there and post questions in the development forum. There's usually a few of us that reply in our free time.