Right now, you're correct - the easiest way for another server to speak the Connect protocol is to use connect-go. We're planning to support more backend languages soon: Node.js is up next, with the JVM maybe afterwards. We'd also love to contribute (time, code, review, whatever) to any other backend implementations - the protocol is not that complex. File an issue on one of our projects or ping us in our Slack, it'll probably be an easy sell.
Envoy support is complicated. The honest truth is that we haven't looked into it all that much because it feels rude - Envoy's architecture requires first-party plugins to live in the main repo, and it feels impolite to attempt a C++ code dump for our new project. Maybe once Connect takes the world by storm :) Even if we did do that, a Connect-to-gRPC translation layer would only work for binary payloads - converting JSON to binary protobuf requires the schemas, and redeploying proxies every time the schema changes is an operational nightmare.
* Worth noting that gRPC-Web doesn't really have a specification. There's a narrative description of how gRPC-Web differs from standard gRPC, but they specifically mention that the protocol is defined by the reference implementation in Envoy.