I guess, sure, companies are technically allowed to request you install malware on your computer. It's probably a CFAA violation, but nobody gives a rat's ass about the CFAA unless you're hacking the FBI, so who cares.
Also, make no mistake: a piece of software that is designed to record your screen, keypresses, etc is malware. It is spyware. You have absolutely zero guarantee it isn't install hooks into your system that will exist long after the application is gone.
Just do the interview in a browser sandbox like a normal person, and share your screen. Boom, problem solved, we figured this out 10 years ago. Really that paranoid about cheating? Then pay up and do an in-person interview, paper code test and all. I've had it done to me multiple times.