Small Internet protocol roundup

Home.

This document is adoped from smallweb.txt by zzo38. The document has been edited by D. Bohdan to add gemtext formatting: headings, paragraphs, lists, and links. Any other changes to the original are noted in the changelog. Like the original, this document is placed in the public domain.

The original text file on scorpion://
My web mirror of the original text file
This version on Gemini
This version converted to HTML
The gemtext source on the web

Changelog

[1]

Introduction

Here is a list of some other protocols and file formats. It is like the "SMALL INTERNET PROTOCOL ROUNDUP" document posted to the mailing list for Gemini previously, but with many differences.

This document is public domain. You are free to copy this document and to make changes, e.g. to add things which are missing, change things that are incorrect, make your own comments about any parts of it, etc. If you have your own mirrors (possibly with your own changes) then you can tell me in order that it may be added at the bottom of this document.

Simple explanations have been provided for some protocols. This is not the full specification, but it should be good enough to access a few files, such as the specification file (if it also uses that protocol).

Updated at: 2024-09-14

Gopher

Specification: RFC 1436
gopher://gopher.floodgap.com/0/gopher/tech/rfc1436.txt

The old one. It uses a menu format, which lacks any support for relative links; only absolute links are possible. It also does not use MIME types; you have to use a fixed set of item types, some of which are uncommon, but some new ones have been added since the original specification (such as "i" for text lines that do not link to anything).

It also does not use URLs; to link to something that is not one of the Gopher item types, usually a HTML link (type "h") is used with the target selector string that starts with "URL:" and then the client should recognize that as a URL link (if the client does not recognize it, the server should serve a HTML page that links and redirects to the specified URL, but some servers don't do that).

The Gopher protocol lacks any kind of response headers, so you cannot specify a "file not found" error or anything else like that. The file type must be known by the client ahead of time, usually by the item type specified in the menu.)

Gopher has the "+" item type for an alternate service link (e.g. mirrors and load balancing), although I read somewhere that it does not seem to be used by any server nor by any client, as far as I know.

Gopher and Gopher+ do not support virtual hosting. (Other protocols that do not support virtual hosting will also be mentioned in those sections.)

Simple explanation

Send the selector string (the path part of the URL, excluding the first character which is the item type) followed by CRLF. The response is the data of the file; there is no header.

Gopher+

Specification:
gopher://gopher.floodgap.com/0/gopher/tech/gopherplus.txt

Adds several features to Gopher, such as:

Gopher with TLS

Specification: (?)

Use of Gopher protocol with TLS. I think that it isn't very good, because Gopher does not have relative links, and the menu format of Gopher does not allow to specify the use of TLS or not, making it difficult to know if you should use TLS with a specific server.

(Presumably, virtual hosting is possible by the use of SNI.)

Gemini

Specification:
gemini://geminiprotocol.net/docs/specification.gmi

A protocol and file format which has been worked on for a while, and some minor changes to specifications have been made. (There have also been many suggestions, most of which have been rejected. Scorpion (see below) does some of the things that were suggested, but deliberately not all of them; most of them have been rejected for Scorpion as well. If you merely want non-TLS, you might also consider Spartan.)

The file format is limited: only normal paragraphs, block quotes, list items (nesting is not possible), preformatted blocks, and links. The MIME type specifies the character encoding (it recommends use of UTF-8, but allows any character encoding, as long as it is consistent within one file) and the language that it is written in (although mixed languages in one file is not implemented).

Some Gemini files have SGR codes, which are not a part of the specification. Some clients implement SGR codes, but some don't.

There is a response status line, which specifies a temporary or permanent error and the specific error, or can be an input request (meaning that the client must repeat the request with a user-specified query string), or a redirect, or specifying requiring a client certificate.

The response header can specify the MIME type of the file, but the file size is not specified.

It recommends use of TOFU (rather than certificate authorities) to verify server certificates.

Simple explanation

You must make a connection using TLS. Send the full URL followed by CRLF. The first line of the response is the header, and then the data follows.

Mercury

Specification:
gemini://gemini.circumlunar.space/users/solderpunk/gemlog/the-mercury-protocol.gmi

The specification is incomplete, but it is intended to be like Gemini but simpler. The only character encoding is UTF-8, most status codes are removed, and the language code used in Gemini is also removed.

Spartan

Specification:
gemini://spartan.mozz.us/specification.gmi

A protocol similar than Gemini, but without TLS, and has some other differences as well. The number of status codes is much less.

The file format is the same as Gemini but with an additional link type added for a link requesting input (similar than Gopher item type "7", but different in some ways). (However, the same MIME type is used.)

The request is the host name and the file path, and the upload size (zero if no upload). There is no way to distinguish the query string from upload data, and no way to distinguish a GET request from an empty upload. Although the user name and password are allowed in the URL, there is no way to use the user name and password (why?).

Simple explanation

Send the host name followed by a space and then the path (including the leading slash, but excluding any query string) with percent-encoding, and then another space and then the upload size (normally 0) and then CRLF. The first line of the response is the header, and then the data follows.

Titan

Specification:
gemini://transjovian.org/titan/page/The%20Titan%20Specification

An upload protocol for use with Gemini. The security token, file size, and MIME type are added on the end of the URL.

Scorpion

Specification:
scorpion://zzo38computer.org/specification.txt

Another protocol and file format. The protocol is similar than Gemini in some ways, but there is a request method, although not request headers.

Scorpion has most status codes of Gemini, but omits the 11 status code, and changes some of them (e.g. 20 now specifies the file size (which is optional; you can use ? if it is unknown) as well as the MIME type, and can optionally use ULFI instead of MIME). There are also some additional features such as range requests, uploads, etc. The username and password in the URL are allowed, unlike Gemini (client certificates are also possible, if you are using TLS).

Scorpion has its own file format, which is a binary format, although some of its features are vaguely like Gemini (others are different; e.g. mixed character encodings are possible within one file, unlike Gemini). Like Spartan and Gopher, there is a link type for requesting input from the user, although the 10 status code from Gemini is also available.

The specification document also defines some X.509 extensions, which can be used with other protocols as well. The "hashed:" scheme is also defined in this document, which is also independent of the protocol.

Simple explanation

Send R and then a space and then the full URL followed by CRLF. The first line of the response is the header, and then the data follows.

Text Protocol

Specification:
https://textprotocol.org/

Seems to use a similar file format than Mercury. There is only three status codes (OK, redirect, and error) and there is no input requests. It seems to use DNS Service Discovery though, and it supports Noise as well as TLS, although all of the encryption types seem to use the same scheme ("text"), so you cannot distinguish them by the scheme. The specification also says that C escape codes can be used.

Note that the port number with TLS is the same as the port number of Gemini. This is mostly not a problem; the only problem that I can see with this is that the error code 40 means a temporary error in Gemini but means any error in Text Protocol, which can potentially cause a problem when it is treated as a Gemini-based proxy server.

Simple explanation

Send the full URL followed by CRLF. The first line of the response is the header, and then the data follows.

SuperTXT

Specification:
https://supertxt.net/00-intro.html

Another protocol and file format, which uses SSH for connection and for authentication (and apparently a SSH key is required). There are no URLs; you have to send the command including the file name, to the SSH server. Instead of a response header, it uses exit codes and stderr to indicate error conditions.

The file format supports tables, using a format similar than GNU long options, e.g. --lat=82 --lon=28 "Buried Treasure"

The file format has links on lines by themself (like Gemini), but a link to another part of the same document can be included inside of the text in a paragraph.

There is also a comment mechanism for users to provide feedback comments to any file, by the use of the "ccmnt" command.

It is possible to run WebAssembly programs on SuperTXT, by the use of a system interface called "WA-Nine", which is like a combination of WASI with Plan9, and with a GUI layer called "Shiny Wire GUI".

Molerat

Specification:
https://molerat.trinket.icu/

Another proposed design, more complicated than Gemini and others; it has request and response headers like HTTP, and request methods like HTTP, and the file format seems complicated to parse.

I think it is badly designed in many ways, as well as being complicated. The documentation also has many mistakes in it. (I would recommend avoiding Molerat.)

It has forms, which seems to be the only possible upload format, and each field is required to be non-empty and cannot contain the sequence <tab><crlf> which is used to delimit form fields, so files cannot be uploaded that might contain such sequence.

It specifies the file size in the response headers, and apparently it is mandatory to specify the file size, which can be a problem. And, apparently the fragment part of the URL is supposed to be sent in the request, even though that is for the client to use and not for the server.

Like Gemini, the language can be specified in the MIME type.

TerseNet

Specification:
https://github.com/runvnc/tersenet/blob/master/README.md

This is not actually implemented yet, and is not designed much yet either, so it cannot actually be used at this time.

It defines a maximum file size of 5K for text, and other media and applications are not loaded automatically, but must be explicitly selected by the user from the Media tab or Application tab, respectively.

There is three kind of browsers, those only for viewing text, those for media applications, and those for extended media applications. Applications can use WebAssembly, which can communicate with the TerseNet P2P backbone and is able to use some cryptocurrencies (Bitcoin and Etherium).

TerseNet also uses P2P, and automatically adds everything to a full-text search system, which is powered by the peers in the network.

Nightfall Express

Specification:

echo nex/info/specification.txt | nc nightfall.city 1900 | less

It is a simple protocol to implement. However, there is no virtual hosting, and no response headers (so, like Gopher, you cannot check for errors, and there is no redirects).

The specification says that the file format is according to the file name extension, and that if it is empty or ends with a slash then it is a directory listing, which can have links with => like Gemini has.

Simple explanation

Send the path (excluding the initial slash) followed by a line feed. The response is the contents of the file; there is no headers.

Guppy

Specification:
https://raw.githubusercontent.com/dimkr/guppy-protocol/main/guppy-spec.gmi

This protocol uses UDP; it seems to be an experiment of making something like Gopher or Spartan but with UDP, and that might be usable with a microcontroller (e.g. Raspberry Pi Pico W, ESP32, ESP8266).

It uses multiple packets to transfer a file, so the client will acknowledge each packet so that the server can send the next one.

File Service Protocol

Specification:
https://sourceforge.net/p/fsp/code/ci/master/tree/doc/PROTOCOL

It is unclear whether or not this belongs here, but here it is anyways; someone on the Gemini newsgroup suggested it.

This is another protocol using UDP. It is intended to be like FTP, so it does not use a specific file format like the others mentioned above.

However, it won't work with multiple clients that have the same IP address (unless you have some way of sharing keys).

Misfin

Specification:
gemini://misfin.org/specification.gmi

A protocol intended for sending mail, to be used with Gemini. In my opinion, there are several problems with it.

At the time of this writing, the specification linked above is for Misfin(B); apparently there is also Misfin(A) and Misfin(C).

The entire request (which includes the message) is limited to 2048 bytes, and the only supported character encoding is UTF-8. Also, there is not the way to specify the language like is possible with Gemini.

The file format it uses has additional line types for the sender, recipients, and timestamp.

It uses client certificates to identify the sender. However, there does not seems to be anything inside of the certificate itself to specify that it is a Misfin certificate. (There would be ways to fix this, such as to use a "misfin:" URI in the Subject Alternate Name (instead of using the Distinguished Name and DNS name), or by using an extension.) (Some other protocols also say they use Misfin certificates, and that is then even more problem since the certificate is used with something else and that does not identify it as a Misfin certificate.)

Scroll

Specification:
gemini://scrollprotocol.us.to/spec.scroll

This protocol is designed to allow the user to specify what language of documents they want, and was also made due to some of the criticisms of Gemini. Others including myself have different criticisms and different ideas of how to improve it, and have made up different protocols and file formats, some of which are mentioned above. However, not all of the same criticisms are addressed by Scroll than other people do.

The "core values" and "goals" are similar (although not exactly the same) as those that most people who do "small web" protocols tend to agree with, including myself. However, I think there are problems with the design of Scroll protocol/file-format, and I do not agree with all of them, although there are a few good ideas. (The root of the domain also has (and links to) some commentary, which might be worth reading to know what criticisms it is trying to deal with.)

Like Gopher+, you can request an abstract of a document. Scroll protocol also uses Universal Decimal Classification.

The file format of Scroll is a bit more complicated than Gemini, and the inline formatting means that escaping will be required. However, it does have the benefit that the fragment part of URLs is meaningful.

It also says that client certificates can be used, and that client certificates with the fields for Misfin can be used for Misfin. To that, I have the same criticism as for Misfin itself.

Simple explanation

Send the full URL, followed by a space and then the intended languages (in BCP47 format) separated by commas. The first four lines of the response are metadata.