RTP: One protocol to rule them all? | Hacker News Reader

RTP: One protocol to rule them all?

paper.wf

100 pointstodsacerdoti63 comments

It's a great idea, but could they please change the name?

There is already an RTP protocol [1] for UDP media streaming, and that is already frequently confused with RDP [2], for GUI remoting.

There is also an rtp:// URI scheme in use in various media applications [3, 4] to identity streaming endpoints.

So I'd kindly request not to increase the confusion even more.

[1] https://en.wikipedia.org/wiki/Real-time_Transport_Protocol

[2] https://en.wikipedia.org/wiki/Remote_Desktop_Protocol

[3] https://ffmpeg.org/ffmpeg-protocols.html#rtp

[4] https://wiki.2n.com/faqaudio/en/rtp-destinations-how-to-test...

Agree, calling it rtp is likely to cause confusion. Probably best to choose a four letter acronym. Assume all the good three letter ones have already been used.

Well, the official list of URI schemes is here: https://www.iana.org/assignments/uri-schemes/uri-schemes.xht... (and another "unofficial" one here: https://en.wikipedia.org/wiki/List_of_URI_schemes#Unofficial... )

I think if you don't insist on them making sense as an acronym, there are still some interesting 3-letter combinations free (if you avoid the obvious minefields, that is).

E.g., I've never heard of qqq:// . There is a QQQ Trust apparently, but no network protocol AFAIK.

Fully agree.

It's even stranger (to me) that they picked an existing protocol's name, given that they never even define "RTP" in their paper.

If it's just arbitrary letters, might as well avoid the collision. If it isn't arbitrary letters, please, if you're describing the protocol, you should start by describing the damn acronym.

Interesting! It's worth noting though that HTTP actually works very well for reliably downloading large immutable files.

And since this proposed protocol operates over TCP, there's relatively little that can be done to achieve the performance goals vs what you can already do with HTTP.

And because "everything" already speaks HTTP, you can get pretty close to max performance just via client side intelligence talking to existing backend infrastructure, so there's no need to try to get people to adopt a new protocol. Modern CDNs have gobs of endpoints worldwide.

A relatively simple client can do enough range requests in parallel to saturate typical last-mile pipes, and more intelligent clients can do fancy things to get max performance.

For example, some clients will do range requests against all IPs returned from DNS resolution to detect which servers are "closer" or less busy, and for really large downloads, they'll repeat this throughout the download to constantly meander towards the fastest sources. Another variation (which might be less common these days), is if the initial response is a redirect, it may imply redirects are being used as a load distribution mechanism, so again clients can ask again throughout the download to see if a different set of servers gets offered up as potentially faster sources. Again, all of this works today with plain old HTTP.

Last year I set up some QEMU VMs to test some things. I struggled mightily to get the FreeBSD one up and running. QEMU flags are not the easiest – lots of knobs to turn and levels to pull, but after quite a lot of time trying to get it to work, it turned out that the installer ISO was just damaged. Do'h. It's impossible to say why/how this happened, but probably during download(?)

Since then I've started to check the sums after downloading, just to be sure.

I wish every binary format would include a hash of the content.

Also this is something that can be in HTTP – it's kind of silly I need to manually download a separate sum file and run a command to check it. Servers can send a header, and user agents can verify the hash. I don't know why this isn't part of HTTP already, because it seems pretty useful to me.

TCP has built in checksums that prevent most data corruption. I believe this is why it’s not part of HTTP, because TCP should already be doing this for you.

I’m guessing that for your very large file download you had an unusually high number of corrupted TCP packets and some of those were extra unlucky and still had valid checksums.

Or something else went wrong, so the TCP packets are correct for what some backend told it to have, just wasn't what should have been served for 1-2 packets or whatever.

TCP's is quite simple, but I would think TLS's checksum would be more infallible.

It seems most likely that the corruption happened in RAM or the local storage device, after the TLS integrity check had already happened.

Scary to consider. And if that's so, it can also happen after your integrity check...

The most likely thing by far is that the download failed part way through, but the error was never reported, or the reported error was never checked.

Also, it's quite possible that the HTTP client didn't even know that the download failed: a common pattern is for the server to send a Content-Length of 0, and simply close the connection when it's done sending all of the traffic (i.e. set the TCP FIN flag on the last data packet). If the server decides to abandon the connection early for any reason, then it will... close the connection - which the client will just interpret as the end of the body, and have no idea that the file failed to download fully.

> I don't know why this isn't part of HTTP already

It could probably be improved, but HTTP does support this already:

https://datatracker.ietf.org/doc/html/rfc2616#section-14.15

A nice; I didn't know about that. Do Firefox and Chrome actually check it though?

I set out to create a "simpler HTTP" once. Ended up concluding that by the time you've added the basic features, you need most of what a minimal HTTP request gives you. There might be some marginal gains to be made, but it's simply not worth it. HTTP is too good.

Commenting on this proposal directly, I don't see how a stateful protocol could ever be simpler than a subset of HTTP/1.1 with range requests.

This, HTTP/1.1 already is a simple protocol with the “things you will forget to add in the first version” added.

This is not trying to replace HTTP, it's an attempt at an alternative to "HTTP for big blob downloads". Likewise for the other protocols mentions.

Does it add anything that HTTP can't do with range requests?

I remember hearing that range requests are clunky to implement for HTTP (reverse) proxies, but otherwise they seem to do their job just fine?

I'm guessing range requests are not problematic for proxies, just for caches (which are usually proxies also).

A pure proxy (reverse or not) should have no problem with a range request.

Ah yes, that's what I meant – problematic for caching proxies. Regular ones can of course just pass through the request.

> all integers are little-endian

They really should be big-endian, because that’s network byte order.

IMHO it makes sense to use decimal-encoded ASCII digits instead and keep the protocol readable. Nothing like ‘telnet host.example 80’ followed by ‘GET / HTTP/1.0.’

> (1 bit) request_type: integer representing type of request

With two types already defined. No room for future extensions, then. Is the idea to just use another protocol altogether if change is necessary?

I love how the term "endianness" was picked straight from Gulliver's Travels. It's a very good fit - old wise computer wizards must've known how it's gonna be.

I'm not sure. I was confused for years thinking that "big-endian" meant "the big number is at the end". It was only after reading Gulliver's Travels that it clicked.

I'm all for silly names, but I think this one went a little too much into obscure references and metaphors.

The RFC makes the reference explicit. https://www.rfc-editor.org/ien/ien137.txt

I know that's the standard but for all private projects I always write little endian so I can sleep soundly at night knowing I've saved precious picoseconds not flipping to cpu order.

Most CPUs suitable to implement such a protocol can do endian swapping for free e.g. a load+bswap instruction, a microarchitecture that fuses load+bswap. Don't worry about byte swapping fixed sized int across the network. Just keep in mind that for some applications it maters e.g. most bignum operations favour little endian, but databases profit from big endian (memcmp sorts in numerical order).

Huh, good to know. I guess the only other downside is actually making the compiler do the bswap depending on the language. Eg C has some weird compiler specific builtin keywords vs zig has a nice packed struct that does it for you.

> because that's network byte order.

That's not a good reason. It's the byte order used for some network protocols, but definitely not all. And given that protocols aren't interchangeable there's no advantage from having all protocols use the same endianness.

Little endian makes way more sense today because all modern computers are little endian. The network itself doesn't care what you use. It's just a stream of bytes to the network.

> Nothing like ‘telnet host.example 80’ followed by ‘GET / HTTP/1.0.’

The 80 in port 80 is not ASCII encoded on the wire. That's an UI feature of telnet and/or your OS. (The 1.0 in HTTP is, though.)

I think they were more references the human readability of HTTP/1.0 in general instead of binary encoded data.

> I think they were more references the human readability of HTTP/1.0 in general instead of binary encoded data.

Yup! You’re exactly right. I honestly thought that was obvious in context.

There’s something just right about being able to manually connect to a web server and run queries against it, with very little in the way of tooling to do so. Technically, of course, both telnet(1) and nc(1) are tools, and even one’s TCP/IP stack and OS are, too, which is why I write ‘very little.’ It’s a heck of a lot fewer tools, and more general tools, than JSON-over-HTTP or RTP prober.

And — please — do not refer to me as ‘they.’ I am not a collective, but rather a human being. I find being referred to as ‘they’ to be profoundly offensive.

Prior to recent years, the singular they was more often used as the generic pronoun. I'm old. No offense was intended.

https://en.wikipedia.org/wiki/Singular_they

> And — please — do not refer to me as ‘they.’ I am not a collective, but rather a human being. I find being referred to as ‘they’ to be profoundly offensive.

I can't seriously believe you're not trolling with this. The use of singular "they" to refer to an individual of unknown gender has been a thing for literally centuries.

Some people, like me, grew up on the Internet assuming every person they interacted with was male by default, because it was a statically sound assumption. We would then use the right pronoun if corrected (unless we thought you might be a G.I.R.L.) Some people, like the person you're ostensibly trolling, do not make this assumption, and just use the singular, gender-neutral "they", which is arguably the nicer, more humane approach anyway.

Being offended by such an approach just smacks of trying to "own the libs", or just otherwise bring contrarian for the sake of it. Neither involves being genuinely offended by anything, and is instead a transparent piece of performative posturing. Please don't bring that here.

Network byte order isn't a good reason for big endian.

The most compelling reason for big endian is that it's easier to read in hex dumps.

Years ago, there used to be a chasm between the Bell-heads who ran the telecom network and the Net-heads that ran ARPAnet. They would scoff at one another good naturedly and ignore what each other was up to.

So it's with some amusement that I see this new protocol is named RTP, which clashes perfectly with the RTP that represents how the Net heads and Bell people finally came to terms with each other. What next, SIP for Super Internet Protocol?

But there's already something called RTP -- protocol for transferring video and audio...

From the errata in the article:

> RTP is commonly used as an abbreviation for the Real-time Transport Protocol, and is not descriptive enough. This is a protocol for reliably downloading large files. it is not designed to be a drop in replacement for http or BitTorrent.

> I am currently drafting a successor proposal that addresses these issues.

There is also already a Reliable Transport Protocol used in Cisco's EIGRP.

https://www.cisco.com/c/en/us/support/docs/ip/enhanced-inter...

Having worked with RTP, I was honestly very confused why anyone would pick this as the one protocol to rule them all. (Especially as most applications need RTCP too, so at best it would be two protocols to rule them all…)

It’s odd that someone would propose a new protocol but not check the IANA port number registry for whether the name is already in use. Surely you’ll eventually want a port for your new protocol when it’s massively successful.

Open and read should be coalesced. When would you ever open and not read?

And, why not call it RDLF (reliably download large files protocol) instead of RTP? And what does RTP stand for in this case anyway?

Even if the three letter acronym space is a bit thin, perhaps best to avoid overloading the name of any one of the ten most used protocols out there?

Neat! A couple of suggestions:

* For the variable length data (uri in open request, error payload, etc) you'll want to specify a length in the message. It allows for more efficient parsing (knowing a buffer size up front) and prevents some security(ish) issues - e.g. i could just send an open followed by uri that follows the scheme but is essentially gibberish until the connection is closed by the server (or the server chokes on it somehow).

You may also want to consider a lenth field for READ responses - in the specified range where the server has all of the resource this is redundant, but if the server doesn't have all the resource, it allows the client to request from another at an adjusted offset even while still receiving the values from the first READ.

* Tokens: If i'm understanding the draft (incl erata) correctly you're using tokens as handles for message pairs, additionally one of the tokens is used to associate the open request and read request, and they are all chosen by the client.

Like this:

    Client                                        Server
    
    Open   ---------new_token(val1)-->   
         <---------reuest_token(val1)--           OpenResponse
    
    Read --new_token(val2), open_token(val1) -->
         < --- request_token(val2) ---            ReadResponse

I'd suggest making the payload of the OPEN request be a server chosen open_token. Having the client manage all of the token values forces the server to track tokens and sender IP or other, similar unique identifying information. It also opens the door for various token collision and/or exhaustion attacks.

* Specify various edge cases well (client closes connection early, connection breaks, etc), because they will have interop consequences, and affect server design (e.g. how tokens are handled in the program) too.

> One protocol to rule them all?

I'd expect the "one" protocol to be able to sync files, especially if they are "large", as advertised. In other words, instead of a case "transfer" implement a universal "sync" to "rule them all".

It's almost as if it makes sense to have more than one protocol, given that there is more than one possible task one might perform on data/objects in computer networks.

Tiny suggestion, possibly without merit (no comments or email in the article):

Use ULEB64 encoding instead of RAW unsigned 64-bit fields for STRING lengths.

ULEB64 (https://en.wikipedia.org/wiki/LEB128) is a simple encoding where the 7th bit is used to show if there are more bytes following. So, lengths less than 128 can be encoded in one byte and so forth.

I doubt the protocol will routinely send lengths that are more than, say, four gigabytes. The longest ULEB64 number is eleven bytes, as far as I recall.

Other than that, I know nothing about the ancestors of the proposed protocol and thus cannot comment.

> good single-source performance (unlike BitTorrent)

Is that true? I've never had use for it, but I've heard of people copying files from one machine to another with it and they seemed to think it worked well.

Single source performance is good if the source uploads with the desired download speed for a single leech. This is not always true.

Well... Sure? Unless I've missed a trick, I don't think it's physically possible to make a protocol that's not bottlenecked on the cumulative upload speed, which in the single-uploader case is the upload speed of that uploader. I would argue that the question is purely a matter of if bittorrent can or cannot saturate a given uplink if there is a single uploader, and I thought the answer is either yes or that it can get really close.

It's also a little extra chatty if the seeder turns on super-seeding mode. (refuse to send more pieces to a peer until that peer sends it to another) Bittorrent can be almost (hashing) as fast as http if you add a web seed though, something many people don't know about.

Kudos for creating this. However, as others have said, the HTTP/1.1 protocol is most of what is needed.

I do think there is room for improvement though. Not in the conceptual or logical HTTP/1.1 protocol, but in the physical over-the-wire implementation. I'd like to see a version of HTTP/1.1 designed to work with CBORS as the main over-the-wire format, possibly including support for CBORS over COAP.

French Fries - one dish to rule them all?

Refining the ideas behind pasta, noodles, lasagna to create a simple dish for reliably ending hunger across the world. I came up with the unique name "french fries" all by myself!!1

(this document is a first draft, and is not intended to be implemented in its current form. But I am posting my trivial 5 minutes write-up with a click-bait title to Hackernews, anyway.)

But it’s TCP that’s holding us back. You can get around it up with multiple range requests, but even then TCP is a bit chatty to be as efficient as theoretically possible.

What we need is an open-source Aspera. I gave it a shot a long time ago, but hammering the bugs out of something like that is really hard.

I think you could save one round-trip if you included the resource size in OPEN response. It could be optional if you want to support streaming responses.

Real Tidy? Reliable Transfer?

https://xkcd.com/927/

but in all seriousness this looks pretty interesting, definitely gonna check in again later :3

xkcd 927 anyone?

No offense against the author but I would be careful using a network protocol designed by someone who doesn’t know RTP and didn’t bother to check if the name of their new protocol was already in use.

Grep of etc/services would tell you to pick something else.

Reply on news.ycombinator.com