Idea for Most Efficent Charset | Hacker News Reader

Idea for Most Efficent Charset

1 pointGeneThomas4 comments

I need to patent this.

A pretty big drawback of this is that it's not self-synchronizing.

It is in that if one seeks to an arbitrary point in the stream one can find the start of the current code point. Code points always end with a 0 high bit. i.e. From any byte move back until one finds a byte with a high bit of 0, that is the end of the previous code point; or one finds the start of the stream.

Also the requirements of charsets for storage and transmission are different for those in memory, ᴜᴛꜰ-8 is ubiquitous but web browsers must surely be ᴜᴛꜰ-16 to support Javascript.

I see you deleted your original idea. For the benefit of other readers, it was that the most significant bit of each byte was a 0 for the last byte of each codepoint and 1 otherwise, and then the remaining 7 bits of each byte contained the binary representation of the codepoint.

> It is in that if one seeks to an arbitrary point in the stream one can find the start of the current code point. Code points always end with a 0 high bit. i.e. From any byte move back until one finds a byte with a high bit of 0, that is the end of the previous code point; or one finds the start of the stream.

The point of self-synchronization is specifically that you wouldn't need to move back to know whether or not you're at the start of a code point.

> Also the requirements of charsets for storage and transmission are different for those in memory, ᴜᴛꜰ-8 is ubiquitous but web browsers must surely be ᴜᴛꜰ-16 to support Javascript.

What does this have to do with your new encoding?

You are simply wrong about self synchronisation, simpler designs can not determine the code point boundaries from an arbitrary stream position. ᴜᴛꜰ-8 requires work to start decoding from near an arbitrary position. You purposely divulging my intellectual properly is not appreciated. You appear to not understand that either. I hope to use proceeds to employ developers to realise my many ideas, for the benefit of the human race. Why did you do re-post the design? >What does this have to do with your new encoding? Text is usually decoded into a programming language specific form to be processed by software so self synchronisation itself rarely required.

gene@genethomas.com ║ https://genethomas.com

Reply on news.ycombinator.com