The OpenGL threading model is completely screwed up. You essentially have a global lock per GL context, which means you're restricted to 1 thread for issuing rendering. And in GL a context is for everything, including shaders, texture states, you can't even reasonable do uploads of new scene data in a separate thread.
Vulkan fixes this big time, by allowing apps to construct GPU workloads for a single GPU in parallel. Only the final submission step (which is supposed to be very low overhead if the driver design is decent) is single-threaded per GPU context. And even for that Vulkan is better: It allows you to allocate different contexts for separate engines (e.g. rendering vs. compute vs. copy engine for data up/download to/from the GPU vram).
The lower CPU overhead is just the icing on the cake, the real deal is that Vulkan fixed the threading/locking model.