even just e2e for 1:1 meetings would be great

How come this is hard to do with 100 participants? Not a crypto or video encoding expert, but I'd have thought each participant would have a decryption key for their stream, and this key they would transfer to others by encrypting it separately with each other participant's public key and then broadcast that bundle to everyone. Thus everyone else can decrypt only their entry in that bundle using their private key, and gain access to the shared decryption key for that individual stream. When a new person joins, they add their public key to the central list of public keys, and request that everyone update their bundle to include an entry encrypted with their public key. So basically, from the perspective of the new joiner, each stream would be encrypted until that streamer had updated their bundle to allow the newcomer access to their decryption key. But each streamer would still only need to encrypt their stream once, and then share access to the decryption key only to participants. Scaling wise, sending only incurs the cost of encrypting once - though as a receiver, you'd need to decrypt N streams using N different keys - but is that much harder to do than decoding N video streams in the first place?

Google Duo just announced E2EE for 8 participants.

Currently it encrypts the stream using TLS, just swap out that encryption key with one only known to the participants. Generate and share a symmetric key for the call (under the hood, of course; users should never have to deal with this unless they actively seek out the key verification feature) and things would work exactly the same from there, the server just needs to forward traffic.

It might currently not work in a browser because native TLS is way faster than doing the crypto in JavaScript, but in principle it should be equally heavy whether you make it end-to-end-encryption or encrypt-to-server.