RFC4180 – Common Format and MIME Type for CSV Files (2005)

You're mixing things up. You're confusing "formal definition", which rfc4180 is, with the concept of an official standard, which no RFC is.

Yet, that doesn't stop the world from working based on RFCs such as this one.

> It's a description of one dialect of CSV

There you go.

> FWIW, i wrote a JSON tokenizer in 235 lines, and it really should be shorter:

I've browsed through your code and I have to say you did a good job. I see your coding style is very frugal with the linebreaks and in breaking up statements, and in a manner that doesn't affect readability that much. OTOH, your absence of comments should be taken care of.

However, I should point out a couple of design flaws in your lexer.

First, your lexer handles any invalid token by throwing exceptions. Not only is that bad form and a misuse of what exceptions actually are (a document with a parsing issue isn't an exceptional event), but any parser that uses your lexer is unable to catch any syntactical errors, provide informative error messages, and most importantly be able to recover from parsing issues.

One standard technique is to support frequent lexial/syntactical errors in the parser and use them to handle parser errors.

The other design flaw is that you failed to include length limits on your strings. That's a potential source of problems, including security issues.

Another issue is that your lexer fails to parse JSON numbers, which is the most complex part of the lexer. Your lexer simply dumps any character matched by isNumber to a string, and then try to convert it to a number by calling Java's Double.parseDouble(). As you may know, JSON's definition of a number doesn't match JAVA's specification. If your lexer supported JSON's number format then it would easily double the number of states it already has.

So, eventough your lexer is a couple of hunded loc shorted than my estimate, it is in fact missing basic and fundamental chunks required by a lexer for JSON. Once you add those in, you could recount your loc and see how it steers away from the 500 estimate I pointed out.