I'm no expert, but what do you want to write drivers for ? At the simplest level you can create a Linux device driver that has no corresponding hardware, in which case C will be perfectly adequate and no bus knowledge needed.

If you're just intellectually interested, I'd suggest starting with that and then create some toy hardware project on an Arduino and create a USB driver for it.