We built our video calling app, Daily.co, on top of WebRTC and so we could relatively easily include end-to-end encryption from the day we launched.
A few things make e2e harder as you scale up. One is that metadata gets more and more important, for a lot of reasons, as you add features and try to grow a business. As has been pointed out here, Skype was end-to-end encrypted in the pre-Microsoft days.
Another challenge is that it's a lot easier to build e2e encryption for mesh network topologies than if you're routing things through central servers. For video calling, mesh networking breaks down as you add more users to a call. For us, that means we're e2e encrypted up to 4 people in a call. After that, we're forwarding video streams through our servers in the cloud, and we don't do e2e encryption anymore. (Not that we wouldn't like to, it's just a much heavier development lift.)