I don't think that's the reason.
The large majority of the "Linux kernel code" is drivers, but the large majority of the driver code is the GPU drivers. And the GPU doesn't have its own collection of drivers for every network chip or USB controller ever made.
The second biggest part of the Linux kernel is "arch/" which is the architecture-specific stuff, but GPUs don't really have that either -- a given vendor more or less corresponds to a platform architecture, but if you compare it to, say, "x86/" that's only ~11% of "arch/" and <1% of the kernel.
The reason the GPU drivers are so big is that they're gibberish. Instead of specifying an interface to interact with the GPU in a sane way, they're full of magic numbers that seem to map a (large, overly complex) API interface into the values you pass to the GPU to call the API functions implemented by the GPU's firmware, which is the actual GPU "operating system" but the vendors want to keep as a black box.
Which is obviously counter-productive because it keeps users from optimizing for their GPU, which would make things run faster on it (or have fewer stability bugs), which would make more people want to buy them over a competitor's, which was supposed to be the reason for the secrecy.