1) Nvidia and AMD both do per-game optimizations in the drivers themselves. Though my understand is that Nvidia has more resources dedicated to this. They will send 1 or 2 people to the studios of AAA game devs nearer release to help the company optimize their game plus to add "hacks" into their own drivers to work around inefficiencies in those games.
If they opened sourced their drivers, that work would be made free to AMD and other drivers.
2) "Benchmark Cheating". Graphics Drivers do lots of optimizations depending on the specific application. I'd wager that this includes popular benchmark apps.
3) Patent lawsuits. There are a massive number of patents around the graphics word. Obscuring how a driver works probably helps protect them somewhat from potential patent litigation.