Htmlq: like jq, but for html

Interestingly, the only other context in which I've come across Prolog is from friends who studied at Cambridge, here in the UK. For some reason, the CS 'tripos' (course) there is really heavily focussed on Prolog, and everyone I know from there ended up a huge fan of the language. I'm not sure why that's the case, though, given that almost all other universities seem to use more common languages (Java, C++, etc).

In a certain sense (for example, when measuring brevity), it is indeed easy to write this example in Python. However, the Python version also illustrates that many different language constructs are needed to express the intended functionality. In comparison to Prolog, Python is a quite complex language with many different language constructs, including loops, objects, methods, assignment, dictionaries etc. all of which are used in this example.

As I see it, a key attraction of Prolog is its simplicity: With a single language construct (Horn clauses), you are able to express all known computations, and the example queries I posted show that only a single language element, namely again Horn clauses to express a query, is needed to run the code. The Prolog query, and also every Prolog clause, is itself a Prolog term and can be inspected with built-in mechanisms.

As a consequence, an immediate benefit of using Prolog for such use cases is that you can easily reason about user-specified queries in your applications, and for example easily allow only a safe subset of code to be run by users, or execute a user-specified query with different execution strategies etc. In comparison, Python code is much harder to analyze and restrict to a particular subset due to the language's comparatively high syntactic complexity.

The libraries I mentioned can be loaded by invoking the use_module/1 predicate on the toplevel, here is the complete transcript that loads the SGML, HTTP and XPath libraries in Scryer Prolog:

    ?- use_module(library(sgml)).
       true.
    ?- use_module(library(http/http_open)).
       true.
    ?- use_module(library(xpath)).
       true.

The second query also uses portray_clause/1 from library(format), which you can load with:

    ?- use_module(library(format)).
       true.

After all these libraries are loaded, you can post the sample queries from above, and it should work.

There are also other ways to load these libraries: A very common way to load a library is to use the use_module/1 directive in Prolog source files. In that case, you would put for example the following 4 directives in a Prolog source file, say sample.pl:

    :- use_module(library(sgml)).
    :- use_module(library(http/http_open)).
    :- use_module(library(xpath)).
    :- use_module(library(format)).

And then run sample.pl with:

    $ scryer-prolog sample.pl

You can then again post the goals from above on the toplevel, and it will work too.

Another way is to put these directives in your ~/.scryerrc configuration file, which is automatically consulted when Scryer Prolog starts. I recommend to do this for libraries you frequently need. Common candidates for this are for example library(dcgs), library(lists) and library(reif).

Personally, I start Scryer Prolog from within Emacs, and I have set up Emacs so that I can consult a buffer with Prolog code, and also post queries and interact with the Prolog toplevel from within Emacs.

I like xmlstarlet too, if only because it's old enough that I can reliably get it in package repositories and the dependency footprint is tiny (less an issue now with this tool written in Rust, but previously I was comparing to NPM- and PyPI-based affairs).

lxml is one of the most pleasing to use Python libraries ever, managing to wrap a hot mess of XML APIs in a consistent and Pythonic fashion that you rarely need to escape. IIRC I used beautifulsoup to parse the HTML of a site, and then lxml and either find items and fields by CSS in IPython for quick and dirty data munging, or knock up an XSLT file to transform what I'd scraped into good data in an XML file :)

I'd like to add my support here too, but with a note.

When scraping and parsing (or writing integration test DSL), I always start out with CSS selectors. But always hit cases where they lack or require hoop-jumping and then fall back on Xpath. I then have a codebase with both CSS-Sel and Xpath, which is arguably worse then having only one method.

I suspect here, one uses this tool untill CSS selector limitations are getting in the way, after which one switches to another tool(chain)

Playwright ppl had to solve this for themselves, you can mix them as they are distinct, have few small custom modifications to help with selectors. Playwright compatible selectors would be nice.

Well, here’s your first lesson then: if you prepend (: to your comment it will become a valid XQuery document!

(: XQuery comments are marked by mirrored smilie faces, like this. :)

Everything that isn't a (: happy comments :) is a FLWOR:

  <users>
  {
    for $user in //users
    let $comments = //comment[@uid = $user/@id]
    where count($comments) > 0
    order by $user/lastName, $user/firstName
    return <user id="{ $user/@id }">
      <name>{ concat($user.firstName, " ", $user.lastName) }</name>
      <comments count="count($comments)">
      {
        for $c in $comments return <comment id="{ $c/@id }" />
      }
      </comments>
    </user>
  }
  </users>

It's the bastard child of SQL and XPath 2 lol.

http://www.stylusstudio.com/xquery-flwor.html

> Well, jq is grep as well as sed and awk, but yeah, htmlq seems to be just grep, for sake of comparison.

Exactly, and that is what I mean. If you want to compare, compare it with grep, not jq.

Someone else posted xidel[0] in this thread, which I've not used, but it seems to be the "jq but for html".

[0] https://github.com/benibela/xidel

"A good opportunity to introduce `gron` to those unfamiliar!"

Thank you - appreciated.

I haven't done much work with json but have had reasons recently to do so - and I immediately saw how difficult it was to pipeline to grep ...

But what I still don't understand is that some json outputs I see have multiple values with the exact same name (!) and that still seems "un-grep-able" to me ...

What am I missing ?

Yes, and awk and others. I meant something semantically closer to the need, with css selectors.

and I use it almost every day. It's great, thank you very much!

It's like a programmable loom, but for logical and mathematical operations.

It's a person who does mathematic calculations all day. For example, creating range tables for artillery, calculating averages or totals of a large range of values, or solving complex integrals or differential equations, and so on. They're commonly used in industry or government, especially in astronomy, aerospace and civil engineering for both simulation and analysis. Perhaps the most well-known computers were the Harvard Computers, which operated in the late 19th and early 20th centuries.

As a job, computers were largely automated out of existence by solid-state transistor based automated computers and integrated circuit transistor automated computers in the 60s, 70s and 80s, which replaced the enormously expensive and often largely experimental electro-mechanical automated computers while radically reducing cost and improving performance both by several orders of magnitude.

Here - this explains it really succinctly:

https://www.youtube.com/watch?v=lE1bS-Mn2Mk

Sort of related: [Expecting Short Inferential Distances](https://www.readthesequences.com/Expecting-Short-Inferential...)

I disagree. The 2nd sentence contains, "extract bits content." What is that?

If you're going to write a minimal introduction, at least make sure it's not confusing.

I get the feeling the author felt compelled to write an introduction and did so with as little effort as possible.

this isn't what a dogwhistle is. it's just explanation by analogy to a model presumed to be shared by the intended audience. a dogwhistle offers a surface meaning to the uninitiated that's anodyne but communicates a hidden, coded message to those who possess some undisclosed, shared knowledge with the author. this kind of analogy entirely lacks the surface meaning and the message shared via jargon also communicates something about how you might learn enough to understand the analogy.

I can't speak for people who don't know jq, but knowing jq, this is a great tagline: it gives me an immediate understanding of what it does, how I could expect to use it and what value and ease of use I can expect.

I'll be trying it out next time I'm on a PC.

"$links" works too because PWSH is not case sensitive. But I should have used $LINKS like you said for cleaner write-up.

JSON schemas are a thing that exists and can be useful, and `jq` probably covers the 80% of use cases for querying JSON.

XSLT can be an amazing tool when used properly and I've wondered about a JS equivalent over the years and started writing one on a couple of occasions. But JSON is just a data structure and not structured markup, and there's no sweet spot for a transformation tool like XSLT - you're more likely to be doing a "find items in JSON, filter() them, then map()/reduce() to output format" task that takes a minute or two in Node and then never gets used again, or doing a complete map from one domain to another where you'd need to do it in JS because of the complexity of processing and ability to handle errors, use third-party tools and even write tests.

An XQuery-esque language allowing selecting bits of JSON file(s) with filtering, grouping and ordering built-in, combined with a way of projecting results that's no worse than JS allows for i.e. not having to put quotes around everything and the like :)

Make sure you add :first-child as in my example, otherwise you'll get all the stories smooshed together.

jq didn't get away with it! Have you never tried searching for anything to do with it? How I wish it were called `jsonq`!

Hahah, I love how this is your second comment in 10 years on HN.

Just looked into this and I think it's worth mentioning that there are two different projects called `yq`. The first one that came up (written in go instead of python) is not the right one and doesn't have the `xq` tool.

xmlstarlet is really nothing like jq, as a language. But yes, I use it because it is the best commandline xml processor I'd found. That's the only similarity to jq.

Is this the yq? https://kislyuk.github.io/yq/ It does contain an 'xq', as a literal wrapper for jq, piping output into it after transcoding XML to JSON using xmltodict https://github.com/martinblech/xmltodict (which explodes xml into separate JSON data structures).

This is a bash one-liner! But TBF it really is a 'jq for xml'. I think it would be horrible for some things, but you could also do a lot of useful things painlessly.

Sure, that sounds nice, but having two simple tools each doing the job well in its own space is perfectly fine for me — do you imagine needing to combine Xpath and CSS queries in a single run?

What language implements regexps that actually correspond to regular languages though?

Kuchiki materialises what they call a “DOM-like tree”. I’d consider it a DOM tree, myself, despite the differences in precise API.

But it’s not using a full browser to back it, which I suspect is what’s really being asked.

And the obligitory caveat from the comments:

> While arbitrary HTML with only a regex is impossible, it's sometimes appropriate to use them for parsing a limited, known set of HTML.

"Oh Yes You Can Use Regexes to Parse HTML!"

https://stackoverflow.com/a/4234491

Haha, Indeed its a very good utility :D

Well once there's an HTML parser, then a pdf viewer, and then everything needed for PDFs (ie., programming, emailer, video support, etc.) we'll finally have that ideal operating system we've been waiting for.

JQ is not just a parser but a tool for doing operations, many of which are (or should be) generic across any tree-like data format. Reusing that part across different input formats makes a lot of sense.

sounds a lot more like blockchain.