Ask HN: Anyone have any cool Open Source Projects and looking for contributors?

> There is no avoiding Unicode since that is what the world has settled upon. Anything else would limit international support.

Actually, it is not true. I believe that avoiding Unicode and treating strings as sequences of bytes (without requiring any specific character encoding) will have better international support, and support for other programs. Some character encodings cannot be effectively converted to/from Unicode or will have wrong character properties (or other problems such as Han unification, etc). Some programs might otherwise mistakenly use the Unicode string type for things that shouldn't be, and I see this problem in too many modern programs than I count (assuming that environment variables, file names, command-line arguments, all locales, etc are UTF-8; that is VERY WRONG and you should fix your program if you do that). I do not wish to compound that problem, so all of my own designs are not using Unicode, as much as I can avoid it. This often means inventing my own file formats (meaning many existing programs cannot deal with them), or using existing file formats in an unintended way (which can also make existing programs confused when trying to deal with them). Even one internal thing in Concise Encoding assumes Unicode text where it shouldn't, which is keys. This isn't very good if you want to store non-Unicode keys.

I know, because I have dealt with character encoding and have the conclusion that non-Unicode is better for international text. Also problems with assumptions that something is text even though it isn't text in any character encoding but is merely a null-terminated sequence of bytes. In some of my own formats that need to use international text and need to know the character encoding within the file format (which is not always necessary, but sometimes it is), they are specified as a 23-bit code page number (an extension of IBM's 16-bit code page numbers), where zero means it is not interpreted as text in any encoding.

> As soon as comments are allowed in a text format, they become part of the structure, and need rules for where they can and cannot occur.

In text formats yes, but not in the general structure which is independent of the file format, I think.

> Astronomical year numbering is essential in internal runtime representations, but terrible for anything human facing (which is why standard library date APIs present them in era form). I've observed time and time again people making off-by-one mistakes trying to input BC dates in ISO 8601 (which uses astronomical year numbering); it's just not user-frinedly

I disagree that it isn't user-friendly; some people will expect them to be astronomical year numbering. One alternatively might be that the text format allows year numbers to optionally specify "BC", "BCE", "AD", or "CE" (where "BC" is equivalent to "BCE" and "AD" is equivalent to "CE"); if none of these are specified then astronomical year numbering is used, and if converted to binary format then they are converted to astronomical year numbering format.

What I have done in some of my own designs is a UNIX timestamp (or, alternatively, use a TRON timestamp, which has a different epoch and may sometimes make variable-length encodings shorter), and the number of nanoseconds is allowed to exceed one billion in case of leap seconds.

> Media is useful for embedding objects that the receiving side will just know how to handle. Media types were added to email and HTTP and various other formats so that one could embed foreign objects into documents in a consistent, compatible way that the operating system could automatically display.

While it is useful for that purpose, it is not always sufficient. There are a few different ways of identifying file formats, including MIME and UTI, and my own "unordered-labels"-based format. MIME is the most common one (and is the one you used here), but cannot always identify files accurately, especally if they can be interpreted as more than one format, etc. (The use of + helps a little bit, but that is just a add-on to a badly designed format.) Using MIME does have its advantages that other programs can more likely be capable of it, though.

My own "unordered-labels"-based format has some advantages that I had considered. For example, you can have "text[1209]:plain:markdown+commonmark:document", "text[367]:html:document", "epub:zip:document", "image:jpeg", etc. You can also identify audio/video codecs, e.g. "ogg<vorbis,theora>:audio:video". A format such as PostScript can be text or binary, and can be treated as a program or as a document, so you can specify this. And then, there are also polyglot codes that can be interpreted as many kind of formats. (The plus sign is a abbreviation, so "markdown+commonmark" is the same as "markdown:markdown.commonmark". Labels can be in any order, so "image:jpeg" is equivalent to "jpeg:image".)

> if it were just a series of bytes, a Concise Encoding reader wouldn't know what to do with it

Yes, although a user could still display the hex dump, or copy out the bytes and display them using an external program, etc.

> Internally, all of the types follow 4-5 data layout strategies to keep the binary codec size small. The text format will likely see 2% of usage in the real world

It is a good idea, although all of the different requirements of different types and their relation can make it messy, and there is then decision, which ones to add or don't add? For example, someone wanted to add currency type, but I think that it would be excessive (and problematic in other ways, too). But you cannot be sure about how much usage the text format will be, compared with the binary format, I think.