> Only downside is I couldn't make it work with my SSH agent, but that might be a problem with Paramiko and not Pyinfra.

Interested to hear more about this, might be something that can be added to paramiko (or on top of).