Sean-Der

I chose the name because of the particle.

I sent a letter to the Soviet Union asking if they were ok with me using the name, never heard back!

I am waiting for my trillion dollar offer to buy https://pion.ly !

Then I am all done fixing WebRTC bugs for life :')

DTLS 1.3 has quite a few cool features/upgrades. Wrote some down so you can see the cool stuff coming your way. Also a few notes on stuff that was surprising or frustrating while building it.

If you get a chance to try it out would love to hear your thoughts! Should be available soon in pion/dtls@v4 :)

We would love your help on Pion WebRTC!

What motivates you, certain topics or building things in a certain area? what are you most interested in?

Here is another thing I find frustrating, but I don't know the right way to express it. A lot of this AI tooling/coding seems self-indulgent? 'I built a better software factory!'. The tools are very powerful, but I don't see the results yet. The frustration outweighs results.

I am underwater dealing with it. I am getting outlandish PRs against Pion. So I spend my time triaging and feel burned out. The 'AI future' I was expecting would be people swooping in and clearing up the backlog in high quality ways.

Instead I get people trying to add new APIs when it can already be done with existing APIs and then arguing about it with me.

I personally used to enjoy the challenge of it. I enjoyed the craftmanship. I am so glad I got to enjoy the challenge of implementing IETF specs and the debugging interop etc... it's so sad to me now that people just bruteforce all this with agents.

I wonder how carpenters felt when powertools started taking off?

This is fantastic! I have wanted something like this for years.

When people are learning WebRTC they are really confused by the Offer/Answer. You have this big blob of text and no idea what it actually does. You look up values one-by-one and forget as you go. its annoying

Excited that the next generation learning WebRTC (for VoiceAI) has better tools.

Fantastic work on this. This idea has so much promise, I hope this is the project that makes it take off.

Self-hosting is pretty these days with docker. Exposing it to the internet is the annoying part. I do Jellyfin via netbird, but that means I don't really share it with my friends. I hope something like this getting baked into self-hosting flows is killer.

My other pet peeve is that people use 'Cloud Services' for File transfer. Sometimes when they are in the same LAN!

Nice work again and I hope it catches on/people understand how great this really is :)

Write up about the architecture is here https://openai.com/index/delivering-low-latency-voice-ai-at-...

What's not easy about it? Would love to hear about the gaps in software/education that could make it easier to use.

Thanks Woodrow :)

Accepting 'Big Changes' from people is VERY frustrating. These thoughts run through my head.

* Idea is usually good! Even if I don't understand it could help lots of others users.

* The contributor is very focused on just getting their feature in. The impact on the larger project isn't as much a concern.

* New contributors often don't have the grit to see it out. They will disappear before things are done. So I am left picking up the pieces (which is harder then doing it all myself)

----

What I try and remember is that their happiness/experience matters more then any code. I try to help the contributor learn/grow as much as possible and even see some career benefits out of it. Pion will cease to matter eventually, so I hope to help as many programmers with it as possible.

NLNet is a wonderful organization. They have supported two Pion projects!

I am grateful the code got written, but even better people got careers out of it/learned new stuff. If you are on the fence about taking on a project I encourage you to do it!

Does WebRTC not work inside/outside of the browser anywhere?

This should be fixed!

I added this in Pion here[0] and I remember testing against Chrome + FireFox and it seemed to work great!

[0] https://github.com/pion/webrtc/commit/e4ff415b2bff31382bdb80...

Amazing debugging, I loved reading that. HN doesn't get enough good posts like this anymore :)

If https://github.com/pion/sctp/issues/12 had happened (not just in Pion but across all implementations) this could have been fixed years ago. The hardcoding we all settle for is tragic.

I work on WHIP in OBS and an open source project Broadcast Box[0]. I made this site/video to talk about the reasons why https://webrtcforthestreamer.com

I hope this site can convince more people to check this stuff out. If you are curious or have feedback I would love to hear.

[0] https://github.com/glimesh/broadcast-box

I work on WHIP in OBS and an open source project Broadcast Box. I see a future where streaming is better.

I hope this site can convince more people to check this stuff out. If you are curious or have feedback I would love to hear

I have been working on making streaming cheaper/more private/lower latency (via WebRTC) in OBS. Been working on this site to get people excited https://webrtcforthestreamer.com/ after it is done want to record a YouTube video for it.

After that I want to spend the weekend just closing out Pion bugs/relaxing :)

I can add it! What format would like to see?

The spirit is alive with https://github.com/m1k1o/neko

The 'co-watching/co-streaming' is the best. I love watching a crappy horror movie with friends and bantering.

You run one encoder for all the viewers. CPU usage won't scale up from 1 -> 15 viewers.

I could get it lower by encoding once and then syncing to keyframes. It would make the code more complicated though. If someone asks for it/gets excited would love to do it though :)

The 'Seek' is done server side. So if you go to `n` seconds it is done server side and done for everyone.

I should reword the README though. If someone is savy enough they could totally grab the video. For most users it is like a Google Meet though. If you click `Show Controls` you can pause the video that is it.

With things like Insertable Streams[0] you can totally grab the video.

[0] https://developer.mozilla.org/en-US/docs/Web/API/Insertable_...

That is 100% controllable! By setting Playout Delay Header[0] you can pick between 'drop everything to stay live' or buffering up to ~40 seconds!

In this project I don't set anything though.

[0] https://webrtc.googlesource.com/src/+/refs/heads/main/docs/n...

1.) Latency vs quality doesn't come up enough to make people want to A/B test it unfortunately. At work I would say ~5 people care about WebRTC vs QUIC vs X. All effort is around the models (how can I provide tools to be support those doing that work)

2.) The model isn't processing just text anymore. Also taking into account breathing/emotion etc... not just spitting out big responses anymore. As it generates them it is taking into account the users response.

3.) It works with the LB setup today. Clients are sending ICE traffic, if it roams we lookup the ufrag and route appropriately.

4.) With DTLS 1.3 it is 1 RTT with SNAP[0] for WebRTC session. SCTP info goes in Offer/Answer, DTLS is packed into ICE. You are totally right about signaling though! [1] was my answer for doing WebRTC without signaling, couldn't get anyone to care though.

5.) I don't have anything that I need to tune. If I want to increase (or decrease) latency [3] is something I put into Transceiver. Otherwise I can't think of any 'change this WebRTC behavior' that has been asked by users/developers.

[0] https://datatracker.ietf.org/doc/draft-hancke-tsvwg-snap/

[1] https://github.com/pion/offline-browser-communication

[3] https://webrtc.googlesource.com/src/+/refs/heads/main/docs/n...

That is not the case. See get-realtime-translate[0 that's doing it as a trickle instead (not turn based).

[0] https://developers.openai.com/api/docs/models/gpt-realtime-t...

Responding to some technical points first, but then after that I do see a future that isn't WebRTC. I don't think it matches where WebTransport+WebCodecs etc is going though.

> …but as a user, I would much rather wait an extra 200ms for my slow/expensive prompt to be accurate

This is the opposite of the feedback I get. Users want instant responses. If you have delay in generating responses/interruptions it kills the magic. You also don't want to send faster than real-time. If the user interrupts the model you just wasted a bunch of bandwidth sending 3 minutes of audio (but only played 10 seconds)

> TTS is faster than real-time

https://research.nvidia.com/labs/adlr/personaplex/ Voice AI for the latest/aspirational is moving away from what the author describes. It is trickled in/out at 20ms

> We really hope the user’s source IP/port never changes, because we broke that functionality.

That is supported. When new IP for ufrag comes in its supported

> It takes a minimum of 8* round trips (RTT)

That's wrong. https://datatracker.ietf.org/doc/draft-hancke-webrtc-sped/

> I’d just stream audio over WebSockets

You lose stuff like AEC. You also push complexity on clients. The simplicity of WebRTC (createOffer -> setRemoteDescription) is what lets people onboard easily. Lots of developers struggled with Realtime API + web sockets (lots of code and having to do stuff by hand)

----

I think if I had my choice I would pick Offer/Answer model and then doing QUIC instead of DTLS+SCTP. Maybe do RTP over QUIC? I personally don't feel strongly about the protocol itself. I don't know how to ship code to multiple clients (and customers clients) with a much large code footprint.

I believe Gemini is Websockets? I have the same experience with heavy/custom applications that try to roll their own media stuff.

You run into issues around AudioContext and resumption etc... it's a PITA to have to handle all those corner cases :(

What platforms were you targeting that you found it painful! Sorry it was frustrating.

I hope it’s getting better with education/more libraries. It’s also amazing how easy Codex etc… can burn through it now

I am excited for VAD to go away. PersonaPlex totally seems like the future.

However things like 'Call center helpline' turn based actually seems better! You don't want to be interrupted when giving information back and forth (I think?)

Thanks for reading it!

You can't beat Websockets :) Especially since you have so much tooling/existing stuff that works with HTTP.

I have been trying to get a website off the ground that does Datachannels + SQlite in the browser and then users sync between each other. I have gotten distracted so many times though.