Hacker News Reader

> (I think it makes more sense for TLS vs non-TLS to be distinguished by the first byte that the client sends to the server (and the client distinguishes them by the URI scheme); using separate port numbers is not required.)

The beginning of a TLS session begins with a "client hello" message[1] which is pretty easy to collect, parse (even without OpenSSL). A few pieces of software I've run into do this check before deciding which protocol to use and I find it nice to have a single port open I can speak TLS and non-TLS to.

[1]: https://tls12.xargs.org/#client-hello/annotated

Yes, although it might be unnecessary to do all of that.

I would think that calling recv with MSG_PEEK could be used to read the first byte; if it is 0x16 then it can continue with TLS (using OpenSSL or other libraries to wrap the connection), and if it is anything else then it will continue without TLS. (This should work with any protocol that expects the client to send data first before the server does, and that 0x16 is not a valid first byte of the request.)

However, I have not actually tried this (since I have not set up any servers with TLS yet).

Spartan feels like a good starting point for a new Gopher-style protocol but appropriately :-) rough and unpolished. I agree about headers indicating size. Without TLS, you want another way to know whether the stream ended or was interrupted early. I see it as the biggest problem with Spartan. As for distinguishing between GET and POST requests, `content-length` in the request line effectively does, doesn't it? The only use a separate indicator would enable is zero-byte POST.

What I like about Spartan is that it requires no URL parsing on the server, has a small but sufficient set of status codes, and adds `=:` to gemtext. Making requests ASCII seems like a good idea, too. It means old computers can support the full protocol without implementing Unicode.

Maybe headers are what this kind of protocol needs. One problem with HTTP/1.1 headers is that they are deceptively simple. Processing them correctly requires handling letter case, leading and trailing whitespace, duplicate field names, and fields that span multiple lines. (A line the starts with a space or tab continues the previous.) A new protocol could, for example, limit the field names to a small single-case set of characters, like [a-z0-9-] or [A-Z0-9_]. It could also build on an existing unambiguous way to encode headers/arrays like SCGI (https://python.ca/scgi/protocol.txt) or RESP (the Redis protocol, https://web.archive.org/web/20211127173832/https://redis.io/...).

> scorpion://zzo38computer.org/smallweb.txt

This is an interesting document! I haven't seen a large comparison like this, and some of the protocols are new to me. It should really be available over other protocols. I couldn't find it on your Gopher site and downloaded it the way https://news.ycombinator.com/item?id=41159495 suggested. Since you support it anyway, I have mirrored the file at https://dbohdan.com/mirror/scorpion/zzo38computer.org/smallw....

> As for distinguishing between GET and POST requests, `content-length` in the request line effectively does, doesn't it? The only use a separate indicator would enable is zero-byte POST.

Perhaps I was unclear, because it is not quite what I meant. What I meant is GET requests with query strings vs POST requests. That is what Spartan does not. distinguish. It does distinguish POST requests from requests with no data, but it is not what I meant.

> Making requests ASCII seems like a good idea, too. It means old computers can support the full protocol without implementing Unicode.

Yes, I also think that it is a good idea. (However, implementing Unicode is not really necessary for a client to send requests anyways (which is the case even if the request is UTF-8, although specifying that requests are ASCII seems cleaner and better to me). The more important part is that the request does not use Unicode domain names and sends them in ASCII format instead, and that the server accepts this and does not expect Unicode requests.)

> One problem with HTTP/1.1 headers is that they are deceptively simple. Processing them correctly requires handling letter case, leading and trailing whitespace, duplicate field names, and fields that span multiple lines. (A line the starts with a space or tab continues the previous.)

I agree with this, they are the problems with the HTTP header format, and it is what I had thought too. However, there is also the consideration of, should you really need so many headers anyways?

> Since you support it anyway, I have mirrored the file at https://dbohdan.com/mirror/scorpion/zzo38computer.org/smallw...

I have added it to the list of mirrors at the bottom of the file.

> What I meant is GET requests with query strings vs POST requests.

Oh, I see. Yes, I would say that, conceptually, Spartan doesn't have GET requests with a query; it only has GET without a query and POST. `=:` is like HTML `<form action="post">`. If you wanted to add queries to Spartan, you would probably do it without breaking compatibility by parsing the query out of `path-absolute`. For example:

  example.com /weather?city=lisbon&time=tomorrow 0

The downside is that it would drag in part of URL parsing on the server, which the protocol currently avoids.

> However, there is also the consideration of, should you really need so many headers anyways?

I see it not as a matter of number but change over time. Headers leave room for protocol evolution and extension. You could also have evolution with protocol versions. A caveat is that if you ever go from sequential versions like 1.2 to versions that enumerate features like 1.2+foo+bar, you end up in a similar place to headers. Compared to headers, you get more flexibility in protocol parsing and a more ad hoc design. Most of your features probably don't need the flexibility, so you just pay the cost of an ad hoc design. (For example, `+foo` and `+bar` may each add a field to the request line. You'd have to make sure they didn't conflict.)

> I have added it to the list of mirrors at the bottom of the file.

Yay.

> I wrote a document about "small web" protocols: scorpion://zzo38computer.org/smallweb.txt

Is this available on other protocol? (Ideally HTTP, Gopher or Gemini would be good too). It seems you already have gopher site but I was not able to locate it there.

It looks like the scorpion protocol is described at https://github.com/zzo38/scorpion/. To retrieve the file mentioned above (without TLS), use:

    nc zzo38computer.org 1517 <<<'R scorpion://zzo38computer.org/smallweb.txt'

Oh wow, it even has a small subset of telnet. Impressive.