Show HN: Xq – command-line XML and HTML beautifier and content extractor | Hacker News Reader

Show HN: Xq – command-line XML and HTML beautifier and content extractor

github.com

104 pointssibprogrammer33 comments

Shoutout to my go-to: https://github.com/EricChiang/pup#readme (also golang) and my 2nd favorite https://xmlstar.sourceforge.net/

Definitely seconding pup, love it and it's easy to use with css selectors. Often use it to parse HTML tables of data on random websites into usable CSVs / etc.

xq looks interesting for pure XML, will add it to my notes.

Regarding xmlstarlet, I don't use it much but here's some stuff from my notes:

XML filter on xpath: xmlstarlet sel -t -c configuration/appender logback.xml; echo

XML editing: xmlstarlet ed -d configuration/root -d configuration/logger -s configuration -t elem -n woof -v 'Woof!' logback.xml

The last one deletes elements given by xpath "configuration/root" and "configuration/logger", then adds a new element with the contents <woof>Woof!</woof> under the xpath "configuration".

> Regarding xmlstarlet, I don't use it much but here's some stuff from my notes:

It is also handy doing at some PoC work, and then one can feed it "-C" to output the backing xslt for use elsewhere

    $ echo '<a><b>hello</b></a>' | xmlstarlet sel -t -m /a -c 'count(*)' --nl
    1
    $ echo '<a><b>hello</b></a>' | xmlstarlet sel -C -t -m /a -c 'count(*)' --nl
    <?xml version="1.0"?>
    <xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">
      <xsl:output omit-xml-declaration="yes" indent="no"/>
      <xsl:template match="/">
        <xsl:for-each select="/a">
          <xsl:copy-of select="count(*)"/>
          <xsl:value-of select="'&#10;'"/>
        </xsl:for-each>
      </xsl:template>
    </xsl:stylesheet>

using `xmlstarlet` to generate XPATHs to visualize document structure

https://github.com/TomConlin/xpath2dot

The pup readme looks pretty great, but a heads up that the project has had limited activity over the past few years. Several raised issues about problems compiling with latest Go releases.

Yeah, 234 forks is not a good look, but that said `brew install -s pup` with `go version go1.19.3 darwin/amd64` reports a-ok so I'd guess it just doesn't have a `go.mod` like normal go developers are used to, and that's why the brew formula moves it into the old style GOPATH setup: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/...

234 forks might be 234 contributors and PRs!

Neat! Like Jq but for XML and HTML.

Have you considered adding css sectors as an alternative to xpath? For many simple things a css selector is easer to write and more people already know them.

I believe it's possible to translate css selectors to xpath so it wouldn't need another selection engine.

Thank you for your support! I didn't consider CSS selectors as an alternative to Xpath, but really like the idea. Absolutely agree CSS selectors will be more convenient :)

The issue with CSS selectors is they can only express element selection. There's no concept of picking the text of the element or the value of an attribute. There's been proposals over the years to support pseudo elements for that, so you could make something up like ::attr(name) and ::text, but nothing in the standards.

htmlq does this by having various command line options like --attribute and --text.

https://github.com/mgdm/htmlq#examples

Quite a few scraping (such as Scrapy) or browser automated testing frameworks (like Playwright) have standardised on ::text. I would take a look around and if other tools are using the same none standard selectors for these features, it would be perfect for Xq. It's not like you are complying with standards for browser compatibility.

htmlq does that.

So there are now approximately 1 million command line tools with various overlapping feature sets for extracting data from XML, JSON, YAML, TOML, and CSV-like delimited data. Is anyone ambitious enough to have constructed a feature matrix for all of them? Is there even a complete list out there?

I wonder if this space of tools is a bit like static site generators: it's almost as easy to write your own as it is to learn somebody else's.

There is also yq [1], which attempts the same for yaml, toml and xml. (And confusingly also contains a binary named "xq" for the xml part - however, it uses jq for querying instead of xpath)

[1] https://github.com/kislyuk/yq

  curl https://news.ycombinator.com/ | xq

yields:

  <title>Hacker News</title>XML syntax error on line 4: element <link> closed by </head>
1

According to the README on Github main page for the project, you need to provide the -m flag when working with HTML.

— The tool parses XML by default, which is much stricter than HTML. Hence the error you are seeing (in HTML5, one can omit various close tags and the HTML is still 100% valid — that's what you are encountering here)

HTH

There's an ancient formatter I've been using for years (gasp, probably well over a decade) https://xml-coreutils.sourceforge.net/ ... xml-fmt (https://xml-coreutils.sourceforge.net/xml-fmt_man.html)

It'll be nice to try something new. swiss army style command line XML tools has been pretty neglected.

doesn't seem to work great with

    https://ap-playerservices.streamtheworld.com/api/livestream?station=NOVA_919&version=1.9

so xqilla and learning xlst et al still seems like my to-go for complex documents

Hadn’t heard of XQilla, but looks like it’s limited to XPath 2.

You should try an XPath 3 based implementation. XQuery 3.1 has some nice features not found in XPath 2 implementations e.g. native JSON support, the arrow operator (piping), group by clause in FWLORs. XSLT 3.0 has some nice new stuff too, but I use that less

I would switch but, finding a simple binary that does what I want (input -> complex query -> output) is so frustrating. Everything is a framework or library.

Thankfully XQilla has its own json support which was all I needed so far

Works fine for me, what doesn't work?

    $ curl -fsSL 'https://ap-playerservices.streamtheworld.com/api/livestream?station=NOVA_919&version=1.9' | xq -x //transport
    http
    hls
    hlsts
    http
    http
    hls
    hlsts
    http
    hls
    hlsts

it works if you need to not use the structured nature of the document

    for $mp in /ls:live_stream_config/ls:mountpoints/ls:mountpoint
        for $server in $mp/ls:servers/ls:server
            return { concat(
                "https://", $server/ls:ip, "/", $mp/ls:mount, ".",
                local:codec-to-ext($mp/ls:media-format/ls:audio/@codec)
            ) }

is the core of my xqilla work, its something I could replicate in jq, I don't think I could replicate it in xq however

There is also hred https://github.com/danburzo/hred

Which extracts XML and HTML as JSON.

Also cool: https://github.com/dbohdan/structured-text-tools/ "A list of command line tools for manipulating structured text data"

related discussion, orthogonally:

JC – JSONifies the output of many CLI tools

https://news.ycombinator.com/item?id=33448204 2022-11-03

What would be the differences to lxml/beautifulsoup/etc.?

Looks good! Does the xpath expression support XML namespaces?

No, and arguably worse: it completely ignores them

  $ echo '<a xmlns="http://www.w3.org/2005/Atom"><b xmlns="urn:beta">hello</b></a>' | xq --xpath '/a/b/text()'
  hello

Only partial support for functions, too, it would appear

  $ echo '<a xmlns="http://www.w3.org/2005/Atom"><b xmlns="urn:beta">hello</b></a>' | xq --xpath 'count(/a/*)'
  $

Some A++ error handling, too, as it just calls panic when angry

Ah ok; Namespace support before XPath 3.0 with Q{nsname}localname would make cli tool like this a bit too verbose. Of course you could store ns-prefix to ns mappings to conf file or environment variables...

I actually tried the lxml syntax "/{urn:beta}b" and that's how I learned about the panic-based error handling, but I didn't know about that Q syntax, so thank you for bringing it to my attention

    $ echo '<a xmlns="urn:alpha">hello</a>' | xq --xpath '/Q{urn:alpha}a/text()'
    panic: /Q{urn:alpha}a/text() has an invalid token.

    goroutine 18 [running]:
    github.com/antchfx/xmlquery.Find(...)
        github.com/antchfx/xmlquery@v1.3.8/query.go:76
    ...

It seems to be a problem with that backing xmlquery library, and I did a quick GitHub topics search and found https://github.com/zzossig/rabbit which claimed to be XPath 3, but then https://github.com/zzossig/rabbit#what-is-not-supported says "lol, namespaces, wat?" so :-(

There aren't that many libraries available with full XPath 3.x conformance. In order to give some constructive feedback, perhaps an optional .conf file with ns-prefix->ns-name mappings could be read and passed to the XPath engine as ns-context? Even w/o the XPath functions support and only XPath selector expression this would make the tool to cover most of the usecases eg. simply finding elements.

xmllint --format -

xsltproc

Yes, I've generally used curl | xmllint --html --xpath for this (usually piped to awk). Not always very pretty, but seems pretty available on MacOS / Linux.

This is awesome. Hugo has been missing a tidy feature for a long time. I hope they use this to implement it finally or that the author might even consider creating a PR.

Reply on news.ycombinator.com