"Guacamole has much better performance ... custom, better-performing protocol" -> What aspect of performance? Latency? Bandwidth? Server CPU and memory usage? Scaling? Do you have some actual performance benchmarks comparing the two? And does guacamole perform better when connecting to the same VNC server or is that comparing VNC server with an RDP server. RDP itself is a newer, more complex and more efficient protocol than VNC but VNC is supported pretty much everywhere. RDP could pretty easily be implemented in noVNC by somebody with the time (in fact it's been on the aspirational feature list for years).

With noVNC the protocol decode/encode is done in the browser (modern browsers are plenty fast enough to do this easily), whereas with Guacamole the burden of decode/encode for every client happens on the server where the proxy/client part is running. Even if you need to run websockify to proxy/bridge noVNC, the only thing it is doing is shuttling network traffic and the python implementation of websockify can easily handle lots of simultaneous clients without breaking a sweat.

noVNC was designed with Infrastructure as a Server (IaaS) providers in mind so minimizing server CPU, memory and bandwidth was a goal in the design.

Also, guacamole does use websockets when it is available.