Hacker News Reader

Comment by rcfox | Hacker News Reader

The problem with piping a large XML file is that you pretty much have to load the entire thing into memory before you can do stuff with it. What if the field you've requested goes on for 5GB-worth of text? You need to load 5GB to get the value.

The usual solution to dealing with large files such as these is to use a memory-mapped file. I'm not sure if it would help with the issue I mentioned above though...

View on HN

You don't need to build the entire document tree in memory. You just have to use a sequential parser like SAX or StAX.

How would this work, exactly? Usually stream parsing requires handler functions, which would imply passing functions at the commandline. I can't think of how you would do this with pipes.

You could write a wrapper that outputs SAX events to predetermined files or named pipes, say, selectable as command line arguments; for instance, the startElement event could be outputted to startElement.txt. Then your script can monitor these files and do the processing from there.

I'm sure there are better ways to do it, though.

You're definitely right, that will work. I guess my point is: the OP seems to be wanting a dynamic way to handle XML at the commandline such that he can handle new types of events on the fly by piping together a parser with other Unix filters, etc. Doing things the SAX way implies creating hard and fast (and specific) handlers for each type of event, indicating that you're anticipating these events in advance.

I'm sure some handlers could be generalized a bit (for example, scanning for regular expressions). But I don't think that the SAX/commandline solution is much better than the SAX/program solution.

Am I missing something?

Reply on news.ycombinator.com