Show HN: Pion SCTP with RACK is 70% faster with 30% less latency | Hacker News Reader

Show HN: Pion SCTP with RACK is 70% faster with 30% less latency

pion.ly

62 pointspch075 comments

SCTP is a low level protocol focused on reliable packet transmission. Unlike hopelessly flinging packets from one device to another, it makes sure that the packets are correct using CRC, removes duplicate packets, and allows for packets to be sent in any order.

Going into an established library, I thought that everything was already implemented and that there wasn't anything to do until I went through the existing issues and organized all the tasks and decided on an order. Sean DuBois (https://github.com/Sean-Der), one of the co-creators and current maintainers of Pion, an open-source pure Go implementation of WebRTC (which uses SCTP), introduced me to a dissertation that was written about improving SCTP from 2021 (https://duepublico2.uni-due.de/servlets/MCRFileNodeServlet/d...). To my surprise, the features in it weren't actually implemented yet, and generally went unused even though it depicted pretty big improvements. This came as a bit of a shock to me considering the countless companies and services that actively use Pion with millions of users on a daily basis.

This led to two things: 1) implement the feature (done by me) and 2) measure the performance (done by Jo Turk https://github.com/joturk). If you're interested in reading more, please check out the blog post where we go over what SCTP is used for, how I improved it, and the effort that went into making such a large improvement possible.

This also marks a huge milestone for other companies and services that use SCTP as they can refer to the implementation in Pion for their own SCTP libraries including any real-time streaming platforms such as Microsoft Teams, Discord screen share, Twitch guest star, and many more!

For my personal background, please take a look at a comment below about what it was like for me to get started with open-source and my career related journeys. Thanks for reading!

Open articleView on HN

You have done such an amazing job on this, it's been such a joy watching you improve SCTP. Better Bandwidth Estimation has been the big list of things everyone has been wishing could be better for 5+ years.

I can't wait to see where your going/what you end up building :) I absolutely didn't have the skills you had at this point in my career.

-----

* What is the best way to get people interested/involved in Open Source/contributing? I am always looking to attract more people. What is Pion doing right and what could it be doing better?

* Doing this project what was the most surprising technical thing you learned? I haven't gone deep on Bandwidth Estimation, but it's super cool to me that loss + latency are really just heuristics. I get why it isn't possible, but it's a shame middle boxes can't just tell software 'You can send this much' :)

Thank you!! It wouldn't have been possible without your enthusiasm and encouragement!

I think a lot of people are interested in making cool things but bridging the gap between looking at the project from the outside and actually doing something can be scary, especially for people who are contributing to open-source projects for the first time. For people who are extremely new, it helps a lot to either have direct guidance from someone with experience ("synchronous" due to timely feedback) and/or reading/video educational material ("async" due to something that can be created once and used multiple times after its creation). With regards to Pion, I was lucky to have both kinds: constant feedback from Jo, plenty of material to read via https://webrtcforthecurious.com/ , RFCs, the countless issues in SCTP; it felt a little bit like I stumbled into a lab with all the tools at my disposal and all I had to do was put the pieces together, come up with a plan, and go for it.

Also the https://github.com/pion/webrtc/wiki/Projects-Worth-Doing wiki page was very helpful but this could also be something that we can improve on: having a better understanding of what we want to get done and when we want them to get done (roughly speaking) is good. Updating it can be a bit of a chore though so I wish there could be a more efficient way to keep track of things while getting things done. I didn't struggle with it as much myself but that was mostly because Jo and I were working together nonstop for the last few months, but I can see how it could potentially be confusing for newer folks. It would also be good to mark which issues/features are blocked by others, which is something that I added in SCTP's readme, which has been nice. One thing I wonder about is whether the older parts of the codebase are daunting, but our efforts to put together resources regarding what RFCs are relevant has been great. I wonder if the video streaming world could have some sort of public tracker for different tools/technologies for people to learn about. I feel like that might be the biggest bottleneck for us, especially compared to other specialized software fields.

As for technical things, there are definitely a couple: learning that Windows' networking doesn't play as nicely as Linux's has been a bit of a nightmare, I'm surprised it even works at all haha. Another is how little attention RACK has received; the dissertation (https://duepublico2.uni-due.de/servlets/MCRFileNodeServlet/d...) doesn't seem to have garnered much attention despite having a really big impact. I also think that it's really surprising to be able to sit down and deal with packets and remember that everything is just a bunch of packets and the way we keep track of things we've sent or received is entirely governed by whatever rules we come up with. It feels oddly primitive but very valuable to be able to learn about why and how we want to track whatever heuristics we come up with!

If you want to read about my personal background and how I ended up working on Pion: https://gist.github.com/philipch07/1ec05c103c7b85f2d43399a70...

Upvoted on Hackernews, Starred on Github, Read your github about page, Starred that and joined the discord server.

As another youngster (17 years to be exact) who was (Might I still add, "is") worried about both the college aspects of things and the job aspect.

Your post gave me assurance that there is a way to make it, that there is hope in its own way.

Really influential, Mad respects goes to you and I wish you all the best moving forward in life in all of your journeys whether coding or not and wishing you a although early, happy new year man!

I have a few questions but you mentioned that you met JoeTurki, Where do you exactly meet such people, On hackernews (I am presuming you have created a new account?)

Because one of the ways that I find software, perhaps even software I can contribute once I get good enough and learn the fundamentals (honestly I am a gopher head too :) ) but some projects are written in C so perhaps I will learn that just to contribute to some/read them to maybe even rewrite them in golang perhaps but those are other ideas.

So the approach I make is probably hackernews -> good project -> gives me ideas/github projects -> I search for ideas when any problem comes in my head -> I then find github projects which are interesting -> I join their discord server -> I chat with their creator.

I think this has only happened in 1 project to be fair (librisc-V), I really love the elegance of librisc-V and I appreciate the creator of that project a lot too

So I am interested in how you meet people, because although one can get in discussions with hackernews with some pretty good people, I sometimes think that the discussion is rather limited in context so there's that and thus I am interested with this.

You mentioned "I decided I’d try for a different strategy: I got involved in an open source project related to video streaming", so I am fairly interested in how you decided to "jump in"/ try since I feel like this advice might help me

Also once again, The project is really good and I hope more open source projects can perhaps be built on top of it, and I genuinely hope you a nice day man!

I'm happy to hear that you were encouraged by my story :) It can definitely be scary to begin with contributing to open source projects but I would recommend trying to find a place to start that 1) has an active community with people you can actively talk to and 2) find a project that you find interesting enough to care about.

An active community is really important: just a year ago Pion had very little activity until Jo picked it up as a maintainer and was an incredible person to talk to, including countless questions about almost everything. I'm really not exaggerating, I was able to do this with almost 0 prior knowledge on Go and video streaming. Part of working with open-source is luck based though; not all projects make it, and people are working on things in their free time. I would try to see how active people are and also how nice they are, because if the maintainers are toxic then realistically speaking, you probably wont be working on the project for a long time even if you found it interesting.

Honestly my strategy sounds really simple, it's basically saying "just do x", which is half of it. My mentality going into it was to try something. If I didn't like it, I would just drop it and move on, but if I did like it (for example, finding a project interesting enough to help out with, or a fun group of people to work with, or just a place to help with small things outside of work) then I would tag along for the ride. Another way to look at it is that by contributing, even just a little bit, you don't have to be responsible for an entire project. Being able to make tiny changes can still be meaningful because it forces you to have to understand the other parts of the codebase, as well as why the changes are necessary in the first place. And if you're ever unsure about either of those, that's where the community part comes in: asking questions is key!

It's important to remember that most open-source projects are just things that people work on in their free time. While it is fun, sometimes people get busy, which is why it's nice to have multiple projects to contribute to. It also gives you a chance to meet new people, that can be from joining whatever chat they're using (like discord), making posts like these to show off things that you've made, requesting reviews from maintainers, etc.

Luckily for me, Pion ended up being a place where I found something to do and was given the time and tools to make something cool!

Reply on news.ycombinator.com