Scheduling and allocation are done by the GPU driver, e.g. the CUDA runtime, with some hardware/firmware assistance from the GPU, which also contains one or more microcontrollers which may perform some of the tasks required for this.
Can you point me to reading material about how the CUDA runtime does this with hardware assistance? I looked but I have been unable to find any thing persuasive in this direction.