I’ve seen this happen to me, and a combination of plugging in a dummy HDMI dongle into the GPU, toggling on/off nvidia nvfbenc (hope I’m spelling this right) fixed it for me.
I got as far as "It looks like it is using software encoding and it should be doing nvenc or something, but it Works Fine for everyone else so nobody ever writes about how any of this shit actually fucking works so troubleshooting it is more effort than it is worth."
And it isn't clear to me -- at all -- how plugging in a dongle that acts like a monitor is going to help anything at all.
I already have three fucking monitors plugged in. All different shapes. A fourth one cannot help.
All that said, I don't think steam link does this automatically, but there are probably tools that have been made for switching your main display to the dongle and setting the correct resolution for it when you initiate a streaming connection. At least I know there are helper ones for Sunshine so I am assuming it is true for Steam Link as well.
Not that this would fix an encoding issue if that is your problem.