Can you share your script for toggling the video card?
Documentation for the nvidia module claims [0] runtime pm is supported even when the module _is_ loaded for GPUs since Turing, but I haven't been able to test that (mine is older than that).
[0] https://download.nvidia.com/XFree86/Linux-x86_64/435.17/READ...
That said, if I want to start a process with using the NVidia driver and card I use this script.
#!/bin/bash
__NV_PRIME_RENDER_OFFLOAD=1 \
__GLX_VENDOR_LIBRARY_NAME=nvidia \
__VK_LAYER_NV_optimus=NVIDIA_only \
exec "$@"