Since you mention Nvidia, here's little case study about graphics card drivers. My direct experience is with Qualcomm's Adreno driver on Android circa 2.3, so it's kinda niche and a little dated, but I understand it roughly holds for the rest of the graphics world. Naturally, these points don't necessarily apply to all devices that require drivers, but they serve to illustrate things somewhat.
As background, a typical graphics stack looks like this, from bottom to top. All components are designed, implemented, and maintained by the GPU manufacturer:
1. Graphics card is presented to the kernel through some mechanism. On Android/ARM, this was as a device memory mapped into a hardware-specified set of addresses. This layer provides the computational component.
2. A kernel driver whose function is to accept GPU code and funnel it down to the device through the appropriate mechanism. This layer provides the input/output mechanisms necessary to communicate with the device, and contains knowledge about the communication protocol the card follows.
3. An OpenGL driver. This is basically a compiler than accepts sequences of OpenGL calls, and translates them down to the hardware-level implementation. Most knowledge of the computational details of the GPU's architecture lives here.
Of these layers, only the kernel driver, number 2, was open source when I worked with it. This is because it links into the Linux kernel, and so needs to be GPL (Not certain about this, please feel free to correct me.) In any case, this component does not contain much or any "secret sauce," so it's fairly safe to share.
The rest of the components, however, are closed source. The reasons for this are:
1. The GPU hardware's architectural design is a trade secret. Even the instruction set architecture is closed source. The reason for this is that much of the competitive advantage a GPU manufacturer might enjoy stems from the speed (and power efficiency) of its GPU architecture. Publishing the architecture would basically provide the GPU maker's competitors access to valuable intellectual property: a lot of research goes into designing a good ISA, and you could probably tell a lot about how the device functions if you had access to the ISA spec.
2. Following this rationale, keeping the ISA secret allows the manufacturer a great deal of flexibility. They have no reason to worry about backwards compatibility because they own the product, so they can do whatever they want with it: add instructions, remove instructions, change calling conventions, whatever.
For comparison, one of Intel's (many) strengths is its robust backward compatibility for its ISA. The basic outline of Intel's x86 ISA have been unchanged for decades, and when they do roll out changes, they're additive rather than subtractive, and they take years to fully roll out. Secrecy presumably allows GPU makers to turn things around as quickly as they feel necessary by freeing them of this backward compatibility constraint.
3. I'm less confident about this point, but I presume there are some tricks they do in implementing the OpenGL library driver. Wherever there are multiple competing implementations of something, you're guaranteed to have tricks the various implementors are proud of.
tl;dr: Keeping drivers secret allow manufacturers to obfuscate their architectural implementation, protecting their intellectual property and affording them a measure of flexibility when they want to make changes to it.