The Spartan Protocol Homepage | Hacker News Reader

The Spartan Protocol Homepage

portal.mozz.us

50 pointsnetworked44 comments

I can see a future where I mainly use "hobbyist" protocols and have largely abandoned the Web. Or at least the 99% of it that requires JavaScript... although I don't know of any functioning non-Chromium/non-Firefox browsers that will plainly render HTML5/CSS and are missing the 30 million lines of Google/Mozilla code that I don't want on my computer anymore. (Ladybird, someday, maybe?)

gemini:// and spartan:// resources are part of the Web. (So is Gopherspace, for that matter.) They're not mutually exclusive; the Web is not HTML over HTTP. To focus on one example: Gopher space is a subset of the Web.*

I encourage everybody to read TBL's book Weaving The Web and his other expository writing about the birth of the Web, the thinking that went into URIs/UDIs, and the absence of a universal information space—the hole that existed before he invented the Web to fill it.

* whether mainstream browsers by the big tech companies support or not is a separate matter

It depends what you mean by "The Web"; the words might have more than one possible meaning. Usually you would say HTML and HTTP(S) is "The Web", although you could also consider a wider definition, that since gemini:// and spartan:// resources have URLs, they can also be considered as a part of "the Web", too.

> you could also consider a wider definition

That's an understatement, which is sort of the whole point of what I wrote:

> I encourage everybody to read TBL's book Weaving The Web and his other expository writing

People who think they know what the Web is but thinks it doesn't include Gopherspace are simply wrong; they are ignorant—in the literal, denotative sense of the word.

When I say "the Web", what I mean is the Web.

> Usually you would say HTML and HTTP(S) is "The Web"

No. The Web is the Web, for all the reasons I already gave. "The Web doesn't mean "browser technologies".

When communicating, it's often useful to use words in their widely understood and shared meaning.

The Web is widely understood to refer to interconnected resources* with links between them. Aside from that, you're on pretty shaky ground here. The person on the street has no idea what HTML or HTTP(S) are. It's not unlikely that they have never even heard of them. Which means you've committed yourself to defending the semi-informed technologist who thinks they know what they're talking about except that they don't. That's a lot narrower of a demographic than your appeal to popularity here lets on (or would even help with—assuming that we don't throw it out on the grounds that it's just an appeal to popularity).

* usually pages; sometimes apps; and probably you could get people to pretty quickly acknowledge stuff like images, etc. if you brought it up ("oh, yeah; those too").

> whether mainstream browsers by the big tech companies support or not

If they did, I would be concerned Google was going to EEE (Embrace, Extend, Extinguish) them. So I hope they don't.

I've lightly struggled w/ this but I think the Web is HTTP/HTML/JS [0]. Like, the Web isn't FTP or SSH, or NTP or NNTP. You can maybe make a case for Gemini/Spartan/Gopher because of linking, but they don't really interact, so on some level they compete.

[0]: https://en.wikipedia.org/wiki/World_Wide_Web

According to that Wiki link, HTTP is "The Web" by definition. And I don't think many people would agree with the statement "Gopher is part of the Web".

Oh I agree, just trying to leave room for conversation :)

I wasn't disagreeing, just continuing the conversation! I notice that for some reason it's hard to "yes, and" on HN. I need to look into ways of conveying a more positive tone without overshooting and sounding insincere, or too cute.

Oh me too for sure. I generally end up littering my emails (for example) with exclamation points just to avoid sounding like "stern male software engineer" but like, is that unprofessional? Why is life so hard haha.

A lot of my thinking has been around "descope the web". Like, ignore the shortcomings of the other protocols (i.e. assume we can improve them like we improved HTTP) and move functionality to them whenever it makes sense. Downloads not related to a HTML file work over FTP or BitTorrent. Interactive apps (e.g. Spotify) can run over SSH or (gasp) run locally. Forums like HN or Reddit move to NNTP. When I peel all that back, I find that the essence of "the Web" is basically wikis.

But then I think that wikis are really a super weird case. They're like some kind of incepted Web where the difference is, idk curation? Organization? Maybe we were getting close to "wiki quality" with the semantic Web and microformats, but then SPAs took over.

> A lot of my thinking has been around "descope the web".

I thought so too.

> Downloads not related to a HTML file work over FTP or BitTorrent.

It is not related to whether or not the file is HTML; protocols such as HTTP and Gemini can still be used to download any kinds of files (and I think the range requests is a good feature of HTTP for this purpose; my own Scorpion protocol also has range requests, but unlike HTTP it does not allow multiple ranges per request and does not have range units other than bytes).

For some kinds of large files, BitTorrent would help. However, I think there are many problems with FTP, so I would not want to use that.

> Interactive apps (e.g. Spotify) can run over SSH or (gasp) run locally.

Yes. Using interactive protocols such as SSH, Telnet, etc, are good ideas, and local programs are also good ideas.

(A local program (and even a HTML-based web app) are not always usable on all computers and there may be security issues, but there are ways around these things; simple and limited VM codes can easily ignore them. Having them for multiple systems available, having FOSS (so that you can compile it yourself, view the code if necessary, and modify it if necessary (whether it is needed to work on your system or because what you want the program to do is different from what the existing implementation does)), operating system design, etc, can also help.)

> Forums like HN or Reddit move to NNTP.

Yes. I think it is a good idea.

> I find that the essence of "the Web" is basically wikis.

It is something that HTTP could work well with, if used in a better way (rather than adding the webapps and that stuff on top of HTTP that is used by Wikipedia and other wikis like that). However, to work better for a wiki, an additional command should be added into HTTP, which would be a HISTORY command; that is the only function that seems to be missing when using HTTP directly. (HTTP already has such things as: PUT, If-Match, etc.)

However, even then, it could be designed better; there are still problems with it. (Furthermore, protocols other than HTTP may also potentially be suitable, maybe.)

I think it's unfortunate to not have headers as part of the client request and/or server response. I think headers can add a lot of extensibility and future proofing for use cases that can't be anticipated now. Especially when the culture and/or inspiration of the protocol is oriented to hobbyists.

e.g. Using spartan protocol as inspiration [1], a client upload request could potentially be:

  example.com /upload/africa.mp3 3145728 1 <CRLF>
  Content-Type: audio/mp3 <CRLF>
  <binary stream>

Where the "1" in the request line stands for one header to follow. Or maybe it's just not needed at all, assuming you like HTTP's line break between headers and message body.

This borrows from HTTP a bit, but other approaches could be viable of course. But this would give the protocol much more extensibility and allow more freedom for inspired implementations.

[1] http://portal.mozz.us/gemini/spartan.mozz.us/examples.gmi

I don't know how I feel about other headers, but I don't like using a Content-Type header since it can disagree with the binary stream, and this has historically been a place to hide bugs.

A binary stream can be multiple formats at once. An out-of-band content-type is the only workable solution; auto-detection of file formats (a.k.a. ”sniffing”) has has a terrible security record.

> A binary stream can be multiple formats at once

So what? If Spartan doesn't do anything with them, it can save them to the disk: It doesn't need to know the difference between a zipfile and an xlsx sheet in order to do so.

> An out-of-band content-type is the only workable solution

Solution to what? I don't think "more metadata" solves anything, let alone security problems.

An "out-of-band content type" declaration is just that: an unsubstantiated claim as to a valid interpretation of the bytes-in-question.

Using a buggy png parser just because some other layer said it was "image/png" is where the security problems are.

> auto-detection of file formats (a.k.a. ”sniffing”) has has a terrible security record.

The single biggest source of security problems is code you don't need: Fix whatever program you've got that can't tolerate random bytes from the Internet, or don't give those programs random bytes you get from the Internet.

> Using a buggy png parser just because some other layer said it was "image/png" is where the security problems are.

If your PNG parser is buggy, you have bigger problems.

> Fix whatever program you've got that can't tolerate random bytes from the Internet

Like your PNG parser?

The underlying problem you are describing is the spirit of the web browser. It has been designed to be extremely resilient to sloppy / wrong coding. This is the reason JavaScript has weird features like Automatic Semicolon Insertion and why every browser tolerates sloppy HTML markup (even though they don’t clean it up in the same ways).

Do you never want to see a file in the browser?

Molerat protocol allows headers too; it is intended to be another "small web" protocol, but with headers and with a much more complicated file format. However, I think it is badly designed in many ways.

I know of Spartan protocol. Note that it can also be accessed by Gemini: gemini://spartan.mozz.us/

I think there are some problems with it.

Why does it allow a username/password in the URL even though there is no way for the protocol to use it (and it cannot send this to the server)?

It also says "The fragment component is allowed but has no special meaning"; however, the fragment component is meaningful but that depends on the file format and not on the protocol, and it is meaningful to the client and not to the server.

It also does not distinguish GET requests by PUT (or POST) requests.

There is no range requests and no response header indicating the file size (Gemini also lacks this, but it does use TLS close-notify to tell that it is finished).

I dislike that Gemini requires TLS, but Spartan does not allow TLS at all. (I think it makes more sense for TLS vs non-TLS to be distinguished by the first byte that the client sends to the server (and the client distinguishes them by the URI scheme); using separate port numbers is not required.)

Spartan also has some good ideas, such as the =: link type.

I wrote a document about "small web" protocols: scorpion://zzo38computer.org/smallweb.txt

> (I think it makes more sense for TLS vs non-TLS to be distinguished by the first byte that the client sends to the server (and the client distinguishes them by the URI scheme); using separate port numbers is not required.)

The beginning of a TLS session begins with a "client hello" message[1] which is pretty easy to collect, parse (even without OpenSSL). A few pieces of software I've run into do this check before deciding which protocol to use and I find it nice to have a single port open I can speak TLS and non-TLS to.

[1]: https://tls12.xargs.org/#client-hello/annotated

Yes, although it might be unnecessary to do all of that.

I would think that calling recv with MSG_PEEK could be used to read the first byte; if it is 0x16 then it can continue with TLS (using OpenSSL or other libraries to wrap the connection), and if it is anything else then it will continue without TLS. (This should work with any protocol that expects the client to send data first before the server does, and that 0x16 is not a valid first byte of the request.)

However, I have not actually tried this (since I have not set up any servers with TLS yet).

Spartan feels like a good starting point for a new Gopher-style protocol but appropriately :-) rough and unpolished. I agree about headers indicating size. Without TLS, you want another way to know whether the stream ended or was interrupted early. I see it as the biggest problem with Spartan. As for distinguishing between GET and POST requests, `content-length` in the request line effectively does, doesn't it? The only use a separate indicator would enable is zero-byte POST.

What I like about Spartan is that it requires no URL parsing on the server, has a small but sufficient set of status codes, and adds `=:` to gemtext. Making requests ASCII seems like a good idea, too. It means old computers can support the full protocol without implementing Unicode.

Maybe headers are what this kind of protocol needs. One problem with HTTP/1.1 headers is that they are deceptively simple. Processing them correctly requires handling letter case, leading and trailing whitespace, duplicate field names, and fields that span multiple lines. (A line the starts with a space or tab continues the previous.) A new protocol could, for example, limit the field names to a small single-case set of characters, like [a-z0-9-] or [A-Z0-9_]. It could also build on an existing unambiguous way to encode headers/arrays like SCGI (https://python.ca/scgi/protocol.txt) or RESP (the Redis protocol, https://web.archive.org/web/20211127173832/https://redis.io/...).

> scorpion://zzo38computer.org/smallweb.txt

This is an interesting document! I haven't seen a large comparison like this, and some of the protocols are new to me. It should really be available over other protocols. I couldn't find it on your Gopher site and downloaded it the way https://news.ycombinator.com/item?id=41159495 suggested. Since you support it anyway, I have mirrored the file at https://dbohdan.com/mirror/scorpion/zzo38computer.org/smallw....

> As for distinguishing between GET and POST requests, `content-length` in the request line effectively does, doesn't it? The only use a separate indicator would enable is zero-byte POST.

Perhaps I was unclear, because it is not quite what I meant. What I meant is GET requests with query strings vs POST requests. That is what Spartan does not. distinguish. It does distinguish POST requests from requests with no data, but it is not what I meant.

> Making requests ASCII seems like a good idea, too. It means old computers can support the full protocol without implementing Unicode.

Yes, I also think that it is a good idea. (However, implementing Unicode is not really necessary for a client to send requests anyways (which is the case even if the request is UTF-8, although specifying that requests are ASCII seems cleaner and better to me). The more important part is that the request does not use Unicode domain names and sends them in ASCII format instead, and that the server accepts this and does not expect Unicode requests.)

> One problem with HTTP/1.1 headers is that they are deceptively simple. Processing them correctly requires handling letter case, leading and trailing whitespace, duplicate field names, and fields that span multiple lines. (A line the starts with a space or tab continues the previous.)

I agree with this, they are the problems with the HTTP header format, and it is what I had thought too. However, there is also the consideration of, should you really need so many headers anyways?

> Since you support it anyway, I have mirrored the file at https://dbohdan.com/mirror/scorpion/zzo38computer.org/smallw...

I have added it to the list of mirrors at the bottom of the file.

> What I meant is GET requests with query strings vs POST requests.

Oh, I see. Yes, I would say that, conceptually, Spartan doesn't have GET requests with a query; it only has GET without a query and POST. `=:` is like HTML `<form action="post">`. If you wanted to add queries to Spartan, you would probably do it without breaking compatibility by parsing the query out of `path-absolute`. For example:

  example.com /weather?city=lisbon&time=tomorrow 0

The downside is that it would drag in part of URL parsing on the server, which the protocol currently avoids.

> However, there is also the consideration of, should you really need so many headers anyways?

I see it not as a matter of number but change over time. Headers leave room for protocol evolution and extension. You could also have evolution with protocol versions. A caveat is that if you ever go from sequential versions like 1.2 to versions that enumerate features like 1.2+foo+bar, you end up in a similar place to headers. Compared to headers, you get more flexibility in protocol parsing and a more ad hoc design. Most of your features probably don't need the flexibility, so you just pay the cost of an ad hoc design. (For example, `+foo` and `+bar` may each add a field to the request line. You'd have to make sure they didn't conflict.)

> I have added it to the list of mirrors at the bottom of the file.

Yay.

> I wrote a document about "small web" protocols: scorpion://zzo38computer.org/smallweb.txt

Is this available on other protocol? (Ideally HTTP, Gopher or Gemini would be good too). It seems you already have gopher site but I was not able to locate it there.

It looks like the scorpion protocol is described at https://github.com/zzo38/scorpion/. To retrieve the file mentioned above (without TLS), use:

    nc zzo38computer.org 1517 <<<'R scorpion://zzo38computer.org/smallweb.txt'

Oh wow, it even has a small subset of telnet. Impressive.

All of these limited versions of HTTP might be fun to write a client and server for, but for actual utility, even small web utility, you do want a defined subset of HTTP/(X)HTML. Maybe to make a small web HTTP, servers can reject clients that e.g. send user agents, and clients can refuse to render pages that contain a script tag.

After writing a second system, it seems that developers want to write a third system that eschews the complexity of the second system, only to learn why it was there. Just look at Wayland for an example.

Spartan and Gemini are so similar they shouldn't need to be different protocols. This route leads to 10000 protocol variations and your URL will need to indicate that the page is on a type 4854 server. (4=QUIC, 8=ASCII headers separated by LF with user agent forbidden, etc)

> even small web utility, you do want a defined subset of HTTP/(X)HTML. Maybe to make a small web HTTP, servers can reject clients that e.g. send user agents, and clients can refuse to render pages that contain a script tag.

For a "small web HTTP", see also: https://news.ycombinator.com/item?id=40980870

I think it is not necessary for servers to reject clients that send a User-Agent header.

I also think that HTTP and (X)HTML is not good for everything. Although they are excessively complicated, and a defined subset would help, still there are some complexities and other problems, I think.

> Spartan and Gemini are so similar they shouldn't need to be different protocols.

I also agree, although each has different problems (some are shared with both). (I think it is helpful to allow both TLS and non-TLS, although Gemini is only TLS and Spartan is only non-TLS.)

> you do want a defined subset of HTTP/(X)HTML. Maybe to make a small web HTTP, servers can reject clients that e.g. send user agents, and clients can refuse to render pages that contain a script tag.

That’s a great idea.

My idea was to go other way a bit re javascript. That is, allowed it to run but it’s under the control of the user, not the website. A bit like installing a native app. For anything functional, why not have companies just send JSON or have an API and the user can display as they see fit. sending sqlite files around would be another option.

I can’t help but thing that in the future people will laugh at how ridiculous we are with stuff like the l fact that the service provider can make me chase my tail around intentionally obfuscated UI in order to cancel.

> [A small web HTTP, with a defined subset of HTTP/(X)HTML is] a great idea.

It is not the only way, although it is one possibility.

> My idea was to go other way a bit re javascript. That is, allowed it to run but it’s under the control of the user, not the website.

Simpler designs can be made than JavaScript with HTML DOM. But, when it can be allowed, yes it should be under the control of the user and not the website.

> For anything functional, why not have companies just send JSON or have an API and the user can display as they see fit. sending sqlite files around would be another option.

I had a similar idea (although I think there are some problems with some file formats such as JSON, too).

However, it does help to be able to specify which files can be used to display unknown files. My Scorpion protocol/file-format specification includes such a feature called a "conversion file"; both server-side and client-side conversion are possible. It is optional to implement, but if it is implemented, the specification requires that the user is allowed to override it with their own specifications if desired.

> Simpler designs can be made than JavaScript with HTML DOM.

I'd agree here for sure. I suppose I was thinking that if JSON (or any other basic format) was specified in and out from companies, etc; a client-side browser replacement could be about anything. I think the fact that current browsers are 100s of millions of lines of code (or whatever) get in the way of the fact that, at its core, a browser doesn't have to insanely complex.

You need a way to put info on a screen and a way to talk over the network. That's pretty much the core. Not hard to add some flavor of storage and/or scripting client-side.

I've only toyed with a bit, but I think something like the Lua Love2D tool pretty much gets you there. Or any number of approaches.

> although I think there are some problems with some file formats such as JSON, too

Curious what you think those problems are. I'm not defending it particularly and have heard a few valid critiques tossed around online but am always open for hearing how folks think about it. Personally, I'm largely surprised that anything has managed to coalesce into a general format for data.

> My Scorpion protocol/file-format specification includes such a feature called a "conversion file"; both server-side and client-side conversion are possible.

Cool, I'll go take a look for it.

Feel free to ping me at the email in my profile if you're up for kicking any ideas around.

The topic has been sitting on my mind for a long time and I'm glad to get behind anyone's even half plausible technological approach. I think our 60 years of Moore's Law makes any one of a number of approaches possible. But prob not just me plugging away here solo in my living room.

> Feel free to ping me at the email in my profile if you're up for kicking any ideas around.

I did not find a email address in your profile.

However, I have a NNTP server and can use that to set up a public discussion forum for such things. Some of the appropriate newsgroups in Usenet might also be used (e.g. comp.infosystems and/or comp.protocols.misc) if you like to use that.

If you have a GitHub account, then the GitHub issue tracker can also be used for discussion of specific issues with the specific project (the Scorpion protocol/file-format specification, and the programs related to it (e.g. client and server software)).

> I suppose I was thinking that if JSON (or any other basic format) was specified in and out from companies, etc; a client-side browser replacement could be about anything.

Yes, and I think that such a thing (whether or not it uses JSON) is helpful. However, there would then also be something to specify display of such files (which the user can easily override with their own).

> You need a way to put info on a screen and a way to talk over the network. That's pretty much the core. Not hard to add some flavor of storage and/or scripting client-side.

Yes, and there are ways to do it. I think that uxn/varvara is a simple way to do it (especially if you only need one network connection at a time, since it can be connected to the standard I/O; you can disable the file device for security if desired), and is what the Scorpion conversion file specifies (although the conversion file is flexible and can be used to specify other things too, such as server-side conversion).

It doesn't need to be mandatory for clients to implement all (or any) of the features. But, if some features are implemented and the user is not required to use those provided by the server, and can also provide their own (the specification for the conversion file Scorpion protocol, requires this), then it can be less of a problem. (The use (and presence) of the conversion file itself is deliberately made optional, though.)

> a browser doesn't have to insanely complex.

Certainly it shouldn't have to be insanely complex, if it is better designed.

> Curious what you think those problems [about JSON] are.

Criticisms of the data model:

- Not a proper integer type. Although you can write 64-bit (and longer) integers, the corresponding type JavaScript is 64-bit floating point, and long integers are unlikely to work even in implementations that do not use JavaScript. (JavaScript does have a big integer type but these are not used in JSON.)

- Unicode string types. The use of Unicode string types in programming languages and general-purpose file formats isn't very good. It can be inefficient and can make assumptions that are not always valid such as that all data is text and that all text is Unicode and that all file names are Unicode, and then there is the handling of surrogates, etc. Byte strings are better; you can still store Unicode text with byte strings, but also any other data. (Using hex or base64 to the data model just adds additional steps; although such features could be appropriate for the text-based syntax of the file format.)

- Keys. They are influenced by the way keys are handled in JavaScript, which sometimes has some special cases that are not as clean as they could be. (In JavaScript you can use Object.create(null) to avoid the automatic handling of the keys that are inherited from Object.prototype, but this does not apply to the literal object syntax.) (I think keys should be integers and byte strings; JSON only allows Unicode strings, and JavaScript does not distinguish between numbers and strings as keys.)

Criticisms of the syntax:

- No optional trailing commas (even though JavaScript allows it).

- No comments (even though JavaScript allows it).

The things mentioned in the syntax are not applicable for binary formats (although different considerations may be relevant), although can be relevant if you want a text format as well.

(One possible alternative would be a subset of PostScript, rather than a subset of JavaScript. PostScript already has a binary format as well as a text format (although the binary format cannot use numbers longer than 32-bit integers and 32-bit floating point; and limits strings to 64K; these limits might be too small for some applications), has byte strings, has a hex syntax for byte strings, and does not use commas between items.)

> Cool, I'll go take a look for it.

There is a GitHub repository of Scorpion protocol/file-format specification (and some programs) at: https://github.com/zzo38/scorpion

The specification file can also be accessed by Scorpion protocol itself, at: scorpion://zzo38computer.org/specification.txt

I'm not interested; Gemini and Gopher are more than enough. If any, Titan it's the best companion to Gemini, for uploads. Inb4 'ftp it's easier...', you have never seen even a simple FTP client. It has workarounds for tons of OSes such as VMS.

https://communitywiki.org/wiki/Titan

Altough for a global file upload/downloading a la FTP, NFS, and so on replacing them all, 9p it's the easiest choice. Encryption? Use an encryption layer below. Wireguard, Tor... any proto you like. Permissions? Your OS should handle them fine.

Oh, I personally have seen an FTP protocol, and it's not simple at all. The actual data transfer is set up in a completely bonkers way, be it in active or passive mode. Even if you really, really need two separate connections for commands and for data (spoilers: you actually don't), why not have it done like this:

    * client connects to ftp-server:22
    > STOR ~/newfile.txt <CR><LF>
    < 227 server-generated-magical-cookie-nonce <CR><LF>
    * client again connects to ftp-server:22
    > DATA server-generated-magical-cookie-nonce <CR><LF>
    < 225 <CR><LF>
    > ...binary data follows

And never mind all the cruft needed to support OSes that pre-date UNIX or, in fact, TCP's wide adoption. Seriously, 99% of the time all the upload functionality you want is equivalent to

    <newfile.txt ssh ftp-server 'cat >newfile.txt'

The 1 remaining percent is upload resumption.

> It has workarounds for tons of OSes such as VMS.

Given enough time, every protocol will have workarounds for $SOMETHING_WRITTEN_IN_LAST_DECADE (or before).

This is the bane of backwards compatibility. There'll be always one off patches for something. Always.

Gopher does not have virtual hosting. (Gopher with TLS does have virtual hosting because TLS has virtual hosting, although the menu format of Gopher makes it difficult to know if TLS should be used for specific servers; however, storing the encryption parameters in DNS records might allow this to work.)

Titan is a suitable companion to Gemini for uploads (although I dislike the way that the file size parameters are added to the URL, and a few other details), but that is if you want to use Gemini, including the required TLS.

I think FTP is no good.

I also think that different protocol can have different advantages/disadvantages and can be good for different purposes.

Greeted with a sword picture (ascii), glorifying Sparta (what is this "Established 650 BC"?), not utf-8 but only "us-ascii". This doesn't smell good vibes to me.

Sorry but pass.

> not utf-8 but only "us-ascii"

This applies to the protocol, not to the file being transfered by the protocol. The file format is UTF-8, but the protocol is ASCII.

Reply on news.ycombinator.com