Hacker News Reader

Comment by stcredzero | Hacker News Reader

The real problem is that Unix commands produce flat text output without any information about how to parse that text back into structured data. Any user who wants the structured version of the data has to parse it themselves, but these parsers are ad hoc and incomplete by their very nature.

There are a variety of serialization schemes that are quite easy to parse and would be suitable for the output of most Unix command-line tools. BEncode from bittorrent would do nicely, and it's quite easy to parse. JSON would do nicely as well.

Better yet, unify the shell with a virtual machine that is used to implement the OS, and have everything available as 1st class Objects.

View on HN

> JSON would do nicely as well.

Yep, some friends of mine did this with JSON, but didn't make the schema explicit like I mean to: https://github.com/benbernard/RecordStream

> Better yet, unify the shell with a virtual machine that is used to implement the OS, and have everything available as 1st class Objects.

Please no. This is the Microsoft PowerShell approach, where everything is a .NET object. Once you start dictating representations of objects, you are dictating far too much about the implementation of individual pipeline nodes.

Why does the protocol define the objects? Shouldn't the object metadata should have enough information to transform the output? E.g. object is a URL vs protocol is a URL

How can your tools all accept the same kind of structured data without dictating its representation? I don't get it.

Allow a shell variable to control output record and field separators. Default to space and newline if nothing is specified.

ORS=: OFS=, ls -l # rw-r--r--:1:uname:gname:1025:Jun 1:somefile.txt,...,...

Add a specifier for dates, usernames, groupnames, etc. DSF="%Y-%m-%d" ls -l

When I talk about a "representation," I mean an in-memory format. For example, the "representation" of an HTML tree is the DOM.

Yes, you have to agree on a serialization format (JSON, Protocol Buffers, etc), but that's not the same thing. From a serialization format you can represent the data however you see fit in your process. For example, a C++ user might represent a string as a std::string object whereas a Python user would represent it as a native Python string.

The VM-based approach (like PowerShell) defines an in-memory tree representation, namely .NET objects. This means that you can't really interoperate with this stack unless you use .NET too, since you don't have an easy way of converting .NET objects to your own objects.

Yes, but you could also just use IronPython and script in Python.

The more decisions you force on your users, the more reasons you give them to choose some other technology, and the less future-proof you are. For example, .NET is only really usable on Windows.

Just a nit: I think protocol buffers include representations, not just serialization formats. You need to have the schema of the proto to parse it correctly, know which fields are required, repeated, etc. Am I understanding you correctly?

Schema isn't representation the way haberman is using it. He means something like implementation, which protobuf has in many languages in various VMs and can be easily ported to more.

It's true that many Protocol Buffer libraries include representations, but these are for convenience; Protocol Buffers are defined in terms of their serialization format and schema.

I don't follow. Isn't schema the same as representation? It's the equivalent of the DTD for an XML document.

To be super concrete, I can't read a file containing protos without knowing their type, what fields they contain, etc. I can however read JSON just fine without knowing the precise schema being encoded.

> JSON would do nicely as well.

Until you need to incrementally process 2GB of data.

BSON in that case?

Reply on news.ycombinator.com