Show HN: USV = Unicode Separated Values | Hacker News Reader

Show HN: USV = Unicode Separated Values

github.com

32 pointsjph23 comments

Note, this (wrongly) uses Control Picture characters rather than the control characters themselves. There can be perfectly valid reasons to want to include the Control Picture characters in the data, and the inability to makes this unacceptable IMO.

(Just imagine using USV to export a set of blog articles, except you can’t because you talk about USV in the blog articles using the picture characters)

Yes you're right. For special character escaping like your blog post idea, have you seen Unicode Separated Values Extensions (USVX)? It's USV plus extras for backslash escape, whitespace trim, etc. These use a smarter parser.

The language being embedded should provide these features, so USV is kept simple.

If the data is HTML, then you can encode the special characters using ampersand notation. And that can be kept out of the USV spec entirely as an external convention. So that is to say, use the native escaping of whatever data format you are storing.

A similar principle is used in embedding JS or JSON in HTML. A sequence like "</script" could occur in a JS or JSON string literal. For that reason there is a way to escape the backslash \/. Or else the slash can be encoded numerically. This is all provided by the embedded language; the embedding language (HTML) knows nothing about it.

The user, or software, doing the embedding just has to know about this and put in the escapes if they occur in the to-be-embedded data.

USV is nice the way it is; don't make a complicated skullduggery out of it with escape mechanisms. The whole point of choosing those characters is to avoid it. If you're gonna have escapes, then might as well stick with commmas for field separation, newlines for records.

I hear you. The feedback from this post and other testers is approximately evenly split. Some people like the simplicity, some people want/need extras especially whitespace trim, backspace escape, and final newline.

What's your opinion of USV being simple (as you describe above) and USVX being a different standard that is USV + extras?

USV is not simple if USVX exists. There is a potential for confusion. E.g. someone carelessly documents that some software accepts USV, when they mean USVX, or vice versa. You will forever have to be double-checking. Imagine if we had JSON and JSONX to contend with.

White space trim means that if you don't expect trim, your significant white space is eaten, causing a bug.

Most of the time, USV will be machine-generated. If the machine generates a field with leading or trailing space, then that's what the data is.

> There can be perfectly valid reasons to want to include the Control Picture characters in the data.

What you want is null characters. Multiple null characters can be used for multiple separating roles/levels. Microsoft implemented this idea in the Registry; there are string tables where each string is null terminated, and then an extra null indicates the end of the table. e.g.

  foo<NUL>bar<NUL><NUL>

This is an empty table containing no strings:

  <NUL>

This is a table with one empty string:

  <NUL><NUL>

This looks like it should extend to more dimensions with more NULs: record, table, group, file, ...

The long-winded point here is that your fictitious blogger trying to write about the scheme in all likelihood will not have any luck embedding literal NULs into their article, so we are are safe.

Sometimes things are so obvious you think, “why didn’t we do this before?”. This makes a lot of sense I would definitely use it. Only barrier to adoption IMO is lack of implementation. A few libs for a few popular languages and the friction for using it is gone.

Great idea. I'll pledge $50 as a token of thanks and encouragement for any open source developer who can add USV to a popular language or tool such as R, Julia, OpenOffice, LibreOffice, TextQL, Q, JavaScript, Python, etc.

Julia's CSV package is pretty close to supporting it already. (it lets you pick the delimiter already.) It should be pretty easy to change the newline character.

Good news: the excellent VisiData (https://www.visidata.org/) supports USV.

Bad news: it looks like they got ␟ and ␞ mixed up, there are trailing ␞␟ characters after each record (should just be an ␞ between each record), and no characters at all after the last record:

    $ cat test.csv
    one,two,three,
    four,five,six
    seven,eight,nine
    $ vd test.csv -b -o test.usv
    opening test.csv as csv
    saving 1 sheets to test.usv as usv
    Think about what you're doing.
    test.usv save finished
    $ cat test.usv
    one␞two␞three␞␟four␞five␞six␞␟seven␞eight␞nine␞␟$             
    $

Would you be open to filing an issue with these details? https://github.com/saulpw/visidata/issues/new?assignees=&lab...

We do not have anyone who is deeply familiar with USV spec details on the team, and could use your help to understand what needs to be fixed.

As I'm no expert, perhaps you could contact jph as he's the author of USV?

Cool! Great way to not have to deal with all those csv escape rules :-)

I don't get the no-escape rule...it should be possible to escape

Why? Without escaping it's very simple to implement a parser or verifier for it. With escaping it has no real advantages over csv.

The characters are pretty much just for this, when would you ever need to put them in the data? If you're working on tooling for unicode itself? Just use... any other format. Literally any.

In what world would you ever use a tabular format that contains Unicode escapes as a natural part of the content?

No-escape makes serialization and deserialization simpler, faster, and easier to test.

For example, import/export is a breeze on the command line with typical Unix tools such as awk, sed, grep, mlr.

No-escape makes international standardization clearer for Backus–Naur form (BNF), the same way other standards use no-escape BNF.

For example, no-escape is the same choice as with the existing international standard for IANA tab separated values.

That's the problem of all CSV formats, wrong or custom implementations.

Someone will generate wrong usv e.g. escape or encapsulate values.

And if you get that wrong format no one cares if they gave you a wrong format. => Your parser grows and grows...

See, you can already see the problem with this:

1. Editors don’t return carriage on RS

2. There is no US or RS key on my keyboard

I am keeping my tsv. Tab is _never_ a data character. And if return carriage is a data character, you probably shouldn’t use a dsv file in the first place.

> Tab is _never_ a data character.

Until it is. Just like a semicolon for CSV.

> And if return carriage is a data character, you probably shouldn’t use a dsv file in the first place.

Agree.

EDIT:

> I am keeping my tsv

As a solely import-export data format (ie no manual editing) it has some virtue. But at this point you can just use XML, CLIXML or even SQLite *shurg_emoji*

This needs to be in base R, Julia, python, and all spreadsheet programs yesterday.

Group separator and file separator examples would be great.

Good feedback thank you. I'm adding these now for you.

USV with 2 units by 2 records by 2 groups by 2 files:

    a␟b␞c␟d␝e␟f␞g␟h␜i␟j␞k␟l␝m␟n␞o␟p

USV with typical shell commands and pretty output:

    $ echo "a␟b␞c␟d␝e␟f␞g␟h␜i␟j␞k␟l␝m␟n␞o␟p" | 
      sed 's/␟/,/g; s/␞/\n/g; s/␝/\n---\n/g; s/␜/\n===\n/g;'
    a,b
    c,d
    ---
    e,f
    g,h
    ===
    i,j
    k,l
    ---
    m,n
    o,p

Reply on news.ycombinator.com