Hacker News Reader

Comment by CrankyBear | Hacker News Reader

I want to like Jitsi, but I can't. I've been testing it out on a variety of servers, bare-metal and cloud with Debian and CentOS. Regardless of platform, it doesn't scale, it eats memory like peanuts, and can saturate even 10GbE network connections. The service, as opposed to the server, clearly works well. But, the service doesn't have anything like the load that Zoom, Teams, Hangouts, etc. must deal with.

View on HN

I've been helping quite a few people set up Jitsi lately. The software works reasonably well, although documentation is lacking and a lot of configuration options are named terribly, but one thing I've noticed is people's expectations about bandwidth usage are way lower than actuality, especially outbound from the server. But a bit of napkin math suggests that Jitsi isn't doing anything fundamentally inefficient here; one high-res stream plus N low-res streams transmitted to N participants is just a lot of bandwidth.

Please tell us what the config options should be.

Config options for what?

lol. Seriously, for what?

This is so very true. We have been monitoring the bandwidth, CPU and mempools of our small Jitsi Meet instance living on an ESXi VM on a Mac Mini. We notice that 3-4 participants typically consume 5-7mbps downstream to the server and about 5mbps upstream from the server while CPU usage stay in the 15-20% range. The bandwidth scales fairly quickly with the number of participants, especially on phone type devices.

There are a few of things to be aware of when deploying your own instance in jitsi-meet/config.js:

Firefox simulcast is still experimental (edit: and disabled by default), so Firefox is only sending HD to the video-bridge and no LD stream. The HD is then relayed to everyone even as a thumbnail.

If you don't need to see everyone's video all the time, set channelLastN: 5 or a similar number, and only the last N speakers video will be broadcast

If you don't need 720p (1280x720), change the constraints: video section to be something like 360p (640x360)

Enable layer suspension so that HD is not sent to the server when not needed: enableLayerSuspension: true

I haven't tried this at scale, but was pleased with 6 participants on my $200 r710 hardware with 10megabits upload connection service. Memory usage is minimal, couple GB tops. Not running high-res, but was good enough. I mean they develop this and make it freely available for self-host, I'm pleased.

It could be worth reporting some feedback to them directly, if you haven't already?

They're quite responsive on GitHub ( https://github.com/jitsi/jitsi-meet ) and I'm sure additional details regarding any bottlenecks would be appreciated.

Identifying even modest improvements now could result in large memory and bandwidth resource savings over the next few weeks and months.

Done. I'm working with them to hammer things out.

Do you have any Firefox users? Not having simulcast basically makes it impossible to scale up to large meetings.

Good to know.

I've been running multiple deployments of it on Azure, on VMs with as low as 1 core/2 GB. I literally set it up every few days from scratch using https://github.com/rcarmo/azure-ubuntu-jitsi

For my use case (around 10-15 friends, for informal meet-ups) it has worked spectacularly well.

I hesitated to go through the hassle of setting this up because I was worried such a low-end VM wouldn’t be able to handle ~10 users.

Sounds like any of the $5/month VMs from the likes of DO, Linode, etc should handle this. Luckily there is a small hoster with a data center near me with similar specs/price. Their free offering on AWS works fine but it would be nice to get lower latency and have my own domain.

Thanks for sharing!

Why is that? Is each of the participants audio/video going over the server so that you need p^2 bandwidth? Are there cool P2P tricks they are/could be using?

I've been wondering about videoconferencing scale, since Zoom seems to have handled a huge explosion in usage very well. Are they just very good at autoscaling AWS instances, or do they use cool tricks to reduce bandwidth?

The server is receiving 1 stream per participant, and forwarding that stream to all other participants. So for say 10 participants, that's 10 inbound and 90 outbound.

If you switched to p2p instead, then each participant would need to send 10 streams instead of 1, and most people's upload bandwidth is much lower than their download (at least in the US).

It's possible for the server to make decisions on which video streams to forward (e.g. last x talkers) to reduce the number of outbound streams, but switching streams takes a bit of time (you'll need to get a new iframe from the just-switched-on participant) which affects the interactivity of the session.

Modern video codecs also allow for layering of into progressive levels of enhancement, so lower detail versions of the stream can be forwarded to participants will lower bandwidth without the server needing to transcode anything. (Not sure if Jitsi has implemented anything like that.)

Couldn't the server compose the streams itself? Perhaps as simple as tiling them all together and the client can manage that stream, or request single high quality streams if needed.

There'd be a CPU/bandwidth tradeoff being made here, I wonder how the prices on standard cloud providers would compare.

Sure, but that adds a lot of CPU load server side (and cloud CPU is still fairly expensive), adds latency for the extra buffering, decoding, and re-encoding, and limits the display options available client-side.

This also limits how the UI is presented. With 1 stream the client can not choose how the UI is presented — there's a single video stream.

Tradeoffs.

WebRTC should be capable of multicast, so nobody should be having to send X-1 streams for X users. I say should because it's incredibly hard to find out which parts the the WebRTC spec browsers actually implement. I just found an article from Mozilla from 2018 that just blithely mentions in passing some nebulous future date "when WebRTC is (hopefully) enhanced to support multi-user conversations." WTF have I been doing with more than 2 people in a WebRTC session for the last 5 years? What the hell has been going on in WebRTC for the last 9!?

IP multicast is unusable on the public internet. You can use it on small, controlled networks, but that's it.

Are we even at the point where routers reliably support multicast?

You can multicast using WebRTC with this program https://github.com/pion/ion But, it still needs work. FWIW, I think it's promising.

If you have money, scaling resources like bandwith is as simple as 'paying bigger AWS bills'.

What is you setup look like ? If you have largely 1:1 meetings you can enable p2p mode only , video will directly be shared between users. I have had no problems delivering 100-150 concurrent sessions at ~1 Gbps per bridge, beyond that we usually cluster the bridge ( Octo is out of the box solution jitsi uses for this)

Video stream, especially if it a group setting is always expensive. I can’t understand why people have such high expectations for video conferencing. This is the reason why Zoom and Microsoft Teams is killing it: video conferencing is inherently difficult and not a trivial task.

Reply on news.ycombinator.com