Something I absolutely love about text based protocols such as HTTP/1 is how easy you can implement it in any virtually programming language. Sure, the implementation is not top-of-the-notch, but it just damned works, it is portable, it is understandable by humans. That's something what's got lost with HTTP/2 and HTTP/3, respectively.
Text protocols have difficult problems like escaping or detecting the end of particular field that are frequent source of mistakes.
The issue is that many (especially scripting) languages treat binary data as second class.
The only real issue is that inspecting the binary message visually is little bit more difficult. You can usually easily tell if your data structure is broken if it is in text form. What I do is I usually have some simple converter that converts the binary message to text form and this helps me inspect the message easily.
I will agree that exhaustively defining text protocols is extremely hard, starting from character set / encoding and getting worse from there.
Typically you create message description which is then compiled to code that can serialize/deserialize messages in BER-TLV or PER-TLV.
I know because I wrote a complete parser/serializer for BER-TLV. It is simple protocol and any security issue is in the parser/serializer and not the protocol itself. That for simple reason that the protocol is nothing more than a format to serialize/deserialize the data.
BER-TLV is really nice protocol. I have worked with it for couple of years when I worked on an EMV application. It uses BER-TLV to communicate with the credit card but it is also very convenient format for all sorts of other uses and I would use it wherever I could. Think of it as Json but in binary form. It is not complicated and I would not even bother parsing the messages -- I could interpret hex dumps of them by sight very easily.
I don't disagree in principle but I've come across a handful of very poorly documentated binary protocols in my years and that is an extremely painful thing to deal with compared to text-based protocols.
GET /asdfasdfasdfdsaf
X-Asdf-Asdf: 83e7234
HTTP/1.0 202
X-83e7233: 1
X-83f730b: 4
Overall, I think it's kind of a wash. It's basically equally easy to take a documented text or binary protocol and write a working parser. Neither format solves any intrinsic problems -- malicious input has to be accounted for, you have to write fuzz tests, you have to deal with broken producers. It's a wash.
People like text because they can type it into telnet and see something work, which is kind of cool, but probably not a productive use of time. I can type HTTP messages, but use curl anyway. (SMTP was always my favorite. "HELO there / MAIL FROM foo / RCPT TO bar / email body / ." Felt like a real conversation going on. Still not sure how to send an email that consists of a single dot on its own line though.)
A line starting with a period is escaped by adding an extra period; the receiving side removes the first character of a line if it is a period.
It seems you prefix any line of data starting with a period with an additional period, to therefore distinguish it from the end of mail period.
>>* a legal notice, here is a blessing:
>>*
>>* May you do good and not evil.
>>* May you find forgiveness for yourself and forgive others.
>>* May you share freely, never taking more than you give.
Above all, this note at the beginning of the source code impressed me.
Nobody does that anymore and debugging is easily solved by converting your binary protocol to a textual form.
* https://en.wikipedia.org/wiki/History_of_email
Perusing document-based information was also done via clients, first with Gopher and then with the WWW: either GUIs like Mosaic, or on the CLI via (e.g.) Lynx.
well, other side of HTTP was telnet in the terminal - this is why it is ASCII to begin with
https://en.wikipedia.org/wiki/Line_Mode_Browser
But that didn't just spit out the HTTP response - it parsed and rendered the HTML.
However, HTTP/2 and HTTP/3 certainly are, though the reasons why they are complicated have nothing to do with choosing to use a binary based format. (They are complicated for good reason, though, and I hope that browsers and servers can continue to support HTTP/1 as the baseline till the sun burns out, just to make life easier.)
Sure enough, the function StrAppend potentially overflows a size_t size (without checking), and then writes into memory could be past the end of the allocated buffer. Given 5 minutes, I didn't look thoroughly if this is actually exploitable, but it's definitely a red-flag for the code. Be careful out there! Hopefully I am missing something, or this is just a simple oversight, but I would carefully audit this code before using it.
Submitted a ticket through the Althttpd website.
static char StrAppend(char zPrior, const char zSep, const char zSrc){ char zDest; size_t size; size_t n0, n1, n2;
if( zSrc==0 ) return 0;
if( zPrior==0 ) return StrDup(zSrc);
n0 = strlen(zPrior);
n1 = strlen(zSep);
n2 = strlen(zSrc);
size = n0+n1+n2+1;
zDest = (char*)SafeMalloc( size );
memcpy(zDest, zPrior, n0);
free(zPrior);
memcpy(&zDest[n0],zSep,n1);
memcpy(&zDest[n0+n1],zSrc,n2+1);
return zDest;
}How should this happen in practice? The three strings would have to be larger than the available address space...
However, the presence of one piece of code that is not integer-overflow safe definitely makes me nervous. This is just the one I found in 5 minutes, what else is in there?
At core, it's just request-reply + key-value metadata. Whenever it's text, or binary, it does not matter much. But writing HTTP/2 frame types in letters would not make them any easier to understand.
Now, HTTP/2 isn't even conceptually simple, I agree about that... it seems ugly.
well because the hard part is the client-side. caching is a client-side only thing with http, keep-alive is a thing that a server pushes to a client, the same with chunked transfer, which is not as easy to implement for a client like it was with content length.
basically a server does only need to implement certain headers, but a client needs to know all. also most clients even accept bad servers, like content for head requests, etc.. most stateless protocols put a lot of burden into clients.
h2 on the other hand is stateful and keeps the same hard semantics onto the client side and also makes servers more complex, because it's a state machine.
While '408 Request Timeout' is somewhat dubious fig leaf over TCP RST
200 FINE 200 KTHX 404 MISS 403 NO 500 FUCK
Another similar server in one file is busybox httpd command if you are interested https://git.busybox.net/busybox/tree/networking/httpd.c
flex -8iCrfa <<eof
int fileno (FILE *);
xa "\15"|"\12"
xb "\15\12"
%option noyywrap nounput noinput
%%
^[A-Fa-f0-9]+{xa}
{xa}+[A-Fa-f0-9]+{xa}
{xb}[A-Fa-f0-9]+{xb}
%%
int main(){ yylex();exit(0);}
eof
cc -std=c89 -Wall -pipe lex.yy.c -static -o yy045
Yahoo! serves chunked pages
printf 'GET / HTTP/1.1\r\nHost: us.yahoo.com\r\nConnection: close\r\n\r\n'|openssl s_client -connect us.yahoo.com:443 -ign_eof|./yy045This is what it produces for me when I run `lexit.sh us.yahoo.com` - https://stuff-storage.sfo3.digitaloceanspaces.com/ee.txt
The compiled program is only useful for filtering chunked transfer encoding on stdin. Most "HTTP clients" like wget or curl already take care of processing chunked transfer encoding. It is when working with something like netcat that chunked tranfser encoding becomes "DIY". This is a simple program that attempts to solve that problem. It could be written by hand without using flex.
The "gibberish" is GZIP compressed data. "yy054" is a simple filter I wrote to extract a GZIP file from stdin, i.e., discard leading and trailing garabage. As far as I can tell, the compressed file "ee.txt" is not chunked transfer encoded. If it was chunked we would first extract the GZIP, then decompress and finally process the chunks (e.g., filter out the chunk sizes with the filter submitted in the OP).
In this case all we need to do is extract the GZIP file "ee.txt" from stdin, then decompress it:
printf "GET /ee.txt\r\nHost: stuff-storage.sfo3.digitaloceanspaces.com\r\nConnection: close\r\n\r\n"|openssl s_client -connect 138.68.34.161:443 -quiet|yy054|gzip -dc > 1.htm
firefox ./1.htm
Similarly, if you write a server that only speaks HTTP/1.1 (or HTTP/1.0 + Host header), you can put it behind a reverse proxy or load balancer that handles higher versions, does connection management, and terminates TLS. It will work perfectly fine, only without some of the latest performance optimizations that you might or might not even need.
However even if it didn't, js based client-side apps are probably still attackable in the right set of circumstances.
This is even standard practice in many production deployments. Typically you want the proxy or load balancer anyway and there's often little benefit (if any) to using HTTP/2 or HTTP/3 over a very low-latency, high reliability local network.
The Lagrange browser seems quite polished.
I think the solution is to start with a simple protocol and upgrade to more complex protocols after that. While technically you don't need to support HTTP/1 to support 2 and 3, the upgrade over TCP happens mostly in that way.
This is not actually true of HTTP/2 (the upgrade doesn’t use HTTP/1), and slightly misleading of HTTP/3 (there’s nothing to upgrade because it sits beside TCP HTTP; but advertising HTTP/3 support is currently done over TCP HTTP).
Now for the details:
HTTP/2 upgrade is done at the TLS level, via ALPN. Essentially the client TLS handshake says “hello there! BTW do you do HTTP/2?” and the server either responds “hi!” and starts talking HTTP/1, or “hi! Let’s talk h2!” and starts talking HTTP/2.
So it’s perfectly possible (though not a good idea) to have a fully-functioning HTTP/2 server that doesn’t speak a lick of HTTP/1.
(HTTP/2 over cleartext, h2c, does use the old HTTP/1 Upgrade header mechanism, but h2c is more or less just not used by anyone.)
HTTP/3 upgrade, well, “upgrade” is the wrong word. It’s operating over UDP rather than TCP, so you’re not upgrading an existing HTTP thing to HTTP/3, you’re starting a new connection because you’ve learned that the server supports HTTP/3. This bootstraping is currently done by advertising h3 support via the Alt-Svc header (HTTP/1+) or by an ALTSVC frame (HTTP/2+), which the client can then remember so it uses the best protocol next time.
For best results, they’re working on allowing you to advertise HTTP/3 support on DNS, so that after a while it should be genuinely possible (though a very bad idea) to have a fully-functioning HTTP/3 server that doesn’t speak any TCP at all, yet works with sufficiently recent browsers in network environments that don’t break HTTP/3. https://blog.cloudflare.com/speeding-up-https-and-http-3-neg... is good info on this part.
No, it is (some) ASCII plus "opaque octets". ("A recipient SHOULD treat other octets in field content (obs-text) as opaque data.") If you want to say that historically it was such, that's not correct either; it was that plus "supporting other charsets only through use of [RFC2047] encoding" which is a nightmare of an encoding scheme.
> HTTP itself is mostly very simple, text "name: value" pairs separated by newlines
Except when it isn't: headers can be folded across multiple lines. (Which is also obsolete, and discouraged.)
Add to that 1xx responses, transfer encodings, chunks, chunk extensions, trailers. HTTP is far from "simple"…
I really hate when people do that. Static should only ever be const, init once, or something intrinsically singleton. There are very few exceptions.
But if this only ever wants to be an app binary, I guess it's sort of okay.
This program isn't meant to be multithreaded, it's meant to be multi-/process/. The difference is that each instance of the application has its own independent address space. So that's one address space per instance. And because of that there are no threading issues. Also as a result any communication between different processes has to be explicitly defined.
Making this program multi-threaded would be a big mistake because then you couldn't use any of the HTTP proxies out there to monitor for connections and hand them off to this program.
This is arguably a much better architecture from a safety/security standpoint then trying to spin multiple threads each sharing an address space. It forces you to use OS-provided mechanisms for shared state rather than simply manipulating the memory to share information.
From a practical perspective, with a binary protocol, it can be difficult to use across different languages or add support for a new language. If you use the simplest possible encoding, you’d send raw struct data. But this doesn’t always work across different OS/arch/versions/etc. if the server is in C, but the client is in Python, reading the binary protocol would require a far more complicated parser.
Obviously a more formal encoding (protobuf, etc) would be preferred, but if you already need to use an encoding mechanism, why not wrap it in a text format? It’s easier to write clients that can read/write text protocols in any language. The reason why text protocols are so popular aren’t because they are necessarily “better” but easier to adopt. This is why the most popular protocols are text based.
> with a binary protocol, it can be difficult to use across different languages
This is also true of text protocols that aren't well-designed. I don't think it's necessarily the case that binary protocols are more difficult to deal with. You just have a different set of concerns to address.
> If you use the simplest possible encoding, you’d send raw struct data.
This is the "simplest" in the sense that it's definitely easy to just copy this data on the wire, but I think this is a straw man. I don't think it's really any more difficult to write a simple protocol that uses binary data compared to text.
I don’t really think so either… I mean, I’ve done both and it’s really not terrible to use binary. I think text is marginally easier to parse, but once to have the routines to read the right endian-ness, the advantage is minor. As you said, the biggest concern (as always) should be the design. A good design can be implemented easily with either mode.
However, it is significantly easier to debug a text protocol. Attaching a monitor or capturing packets is easier with text as the parsers are much easier and more generic.