The unexpected return of JavaScript for automation

Interestingly I learnt recently that macOS includes a ``jsc'' command, which provides a built-in JavaScript REPL and interpreter [0]. It is part of the JavaScriptCore framework. I can find it at:

  /System/Library/Frameworks/JavaScriptCore.framework/Versions/Current/Helpers/jsc

But a Stack Overflow comment says it moves around between releases, so you may want to find it first using:

  $ find /System/Library/Frameworks/JavaScriptCore.framework -iname jsc

It doesn't support web-specific APIs such as window. But it includes extras such as a print function and a readline function.

  $ cat greet.js
  print("What is your name?")
  const name = readline()
  print(`Hello, ${name}!`)
  $ jsc greet.js 
  What is your name?
  ns
  Hello, ns!
  $

Have fun :)

[0]: https://furbo.org/2021/08/25/jsc-my-new-best-friend/

macOS APIs have poor documentation overall, but JXA is on another level. It's not even like Carbon APIs which are still necessary to use today, but which documentation was removed from Apple's website. At least here, you can find an archive somewhere. No, JXA was simply never documented.

You basically have the AppleScript docs, which themselves are under-documented, and you have to guess what JXA APIs exist based off it. My recommendation is to use Safari and play with the debugger at runtime. That way you can test and see which APIs are available by trial and error. You can also hack around and display JS objects internal variables/prototypes to get an idea of what things may be available.

I've had to dig deep into this for my FOSS app recently, and I'll like to share these 2 gems of very-hard-to-find information:

- How AppleEvents (AppleScript or JXA triggered events) permission works (https://www.felix-schwarz.org/blog/2018/08/new-apple-event-a...)

- (Bitter) testimony from someone involved at JXA's inception (https://news.ycombinator.com/item?id=21803874)

Hopefully this message saves someone hours of research~

We have been deploying infrastructures composed of Hadoop and Kubernetes clusters in JavaScript for years with Nikita (https://nikita.js.org) and the developer experience is great. We released patches and features very fast, writing unit tests is easy and the all process is very flexible. Most of our customers impose the usage of Ansible to accomplish similar tasks, and we lose the simplicity, control and agility. To achieve expressiveness similar to YAML, we use CoffeeScript which looks and feel like YAML with the benefit of a programming language. No need to work around templating limitations, not speaking about configuration management (it is so complex to work around it that we have an article just on this topic in the pipeline).

One really cool little JS library I've been using for a bunch of desktop automation tasks lately is nut.js and the lower level libnut library it's implemented on top of:

https://github.com/nut-tree/nut.js

https://github.com/nut-tree/libnut

It provides a means to send user input (mouse movement/clicks and key presses) and read and react to changes in visual state (through screenshots), and works across Windows, Linux and MacOS. It automates at a much lower level of abstraction than the approaches mentioned in the article that script against programmatic APIs.

What I really like about this lower level approach is that you don't need anyone's permission to automate anything, since there's no programmatic API that the system owners has to provide for you and thus can limit or take away when it becomes inconvenient (or more likely they'd just never provide it in the first place).

Any task that can be accomplished though looking at stuff on the screen and clicking a mouse and pressing keys on a keyboard (i.e. what a real person would do to accomplish the same task) can be automated, and it's actually surprisingly easy and effective to do this with nut.js. What really helps is that OpenCV has become ridiculously good and ridiculously fast at matching/identifying objects from a screenshot, with latencies usually in the low double digit miliseconds, so latency-based flakiness isn't nearly as much of an issue as I remember it being in the old days. I've also played around with OCR with tesseract but haven't had as much success with it in terms of perf, and remember seeing latencies of several seconds for even recognizing a single word from a tiny pre-cropped screenshot containing only the word itself, so these days I end up just taking screenshots of text to identify with OpenCV instead.

The main tradeoff to this approach compared to automation through APIs is that because it works by simulating real user inputs, it's not very amenable to running in the background while a user is actively interacting with the same machine, so a separate machine or VM is often needed. That's an acceptable tradeoff for some use cases but complete deal breaker for others, so YMMV, but just wanted to bring this cool little tool to people's attention.

I don't know about PyObjC, but JXA is very awkward when coming from JS: it brings in conventions from Objective-C, and it's not explained anywhere what patterns I'm supposed to use. E.g. with collections, it seems that instead of just values, you get some very lively views that change while you're trying to use them and refuse to act the same on subsequent access—so the robust way to get values from a collection repeatedly is to obtain the collection each time anew (or convert it to a sane Array). You have to figure out by yourself how AppleScript is translated to JS—or, more realistically, by copying examples from the web. Either I didn't find the proper documentation, or it's pretty much nonexistent. As the article mentions, the Github repo is the primary knowledge compendium on how the hell one uses JXA.

It's pretty much impossible to casually write JXA scripts now and then, just knowing JS and consulting reference docs—instead, it seems that you'll have to learn arcane ObjC conventions from the 80s and keep translating those to JXA, without much help from Apple. Add to this that Swift/ObjC docs themselves seem terse, done with a hefty bit of ‘fuck off’ attitude, and are more of ‘here's what we have’ reference with too little of ‘here's how to do the thing you want’. IIRC Swift docs just drop pages on old versions of classes and functions, so when Apple keeps rearranging things again in new versions, and you inevitably have old examples from the web, you have to grope for how each change works, instead of just opening the deprecated page and reading there that this new thing is now used.

So you have to translate JS ← JXA ← AppleScript ← Swift/ObjC, with resistance on every step.

Oh, and the system settings? I haven't seen comprehensive docs on them either. Just apple.stackexchange and a bunch of howto sites. I guess Apple prefers to provide living for these sites instead of spending some of the billions on a technical writer.

I believe this is just the start. The more businesses move to SaaS models, automation takes a different shape as well, i.e automation scripts involving business objects within/outside the system. Most systems call them "custom scripts" - and the primary language of choice is Javascript. Why? Javascript engines have done a phenomenal job at sandboxing and extensions.

So that'd be another reason why JS will live long as a language of choice of automating cloud objects.

You do get a lot of unexpected returns in JS.

might as well throw jsdb into the mix here http://www.jsdb.org/

And Windows still comes with Jscript too.

Re: "Monterey has deprecated the pre-installed python on macOS"

Interestingly, there's a /usr/bin/python3 (3.8.9) on my new MBP which doesn't have the Apple deprecation message when starting into the REPL. Not sure if it was preinstalled or Xcode put it there though.

I've found JXA useful for simple web automation.

There is a site where I want to go to a few pages and extract some data. Their terms of service allow this as long as you do not access the site faster than a human browsing would do.

I used to do it with curl, with suitable delays between each page fetch.

Then they apparently got hit with some DOS attacks and started using Cloudflare protection and curl no longer worked.

I then switched to Selenium. That got stuck at hCaptcha, refusing to go on no matter how many times I correctly identified all the buses or trains or whatever.

Adding some experimental options in Selenium got around the hCaptcha loop:

  options.add_experimental_option("excludeSwitches", ["enable-automation"])
  options.add_experimental_option('useAutomationExtension', False)
  options.add_argument("--disable-blink-features=AutomationControlled")

A while later though the site started occasionally bringing up that Cloudflare "Checking your browser before accessing <site_name>" automatic check. That just sits there, occasionally reloading.

Same thing if I use undetected-chromedriver [1] which is supposed to be less likely to trigger anti-bot services.

Next up was using a bookmarklet. It was easy to make a bookmarklet that used an XMLHttpRequest to post the content of document.documentElement.innerHTML to a server of mine for processing. That bookmarklet could then advance to the next page of interest. It wouldn't be as nice as the prior solutions because I'd have to click the bookmarket for every page, but that wouldn't be too bad.

The bigger annoyance there is if my server isn't HTTPS the XMLHttpRequest is blocked because of mixed content. In Chrome you can tell it to ignore that on a per site basis, but it isn't persistent across browser launches. There are also issues if the server is on my LAN, which runs into the "not let JavaScript on a page from the internet access the LAN because it might be trying to hack your IoT" stuff.

I was just starting to look into browser extensions to see if that would be a viable approach. I don't know if (1) an extension can save to the local filesystem (which I'd prefer instead of having to do that XMLHttpRequest runaround), or (2) if an extension can carry out actions on multiple pages without user intervention (if I still would have to do the click per page I can just stick with the bookmarklet approach).

I was even contemplating trying to make a fork of Chromium or Firefox with built-in support for this stuff.

Then I remembered AppleScript, and that things that support AppleScript can also be controlled with JavaScript which is much much much less weird than AppleScript.

Here's some JXA code that saves current tab's source from Safari:

  var cur = Application.currentApplication()
  cur.includeStandardAdditions = true;
  var app = Application('Safari');
  app.includeStandardAdditions = true;

  function save_page(file_name)
  {
    let source = app.windows[0].currentTab.source()
    let path = Path(file_name)
    let f = cur.openForAccess(path, {writePermission: true})
    cur.setEof(f, {to: 0})
    cur.write(source, {to: f, startingAt: 0})
    cur.closeAccess(f)
  }

  save_page("/tmp/page.html")

Here are the changes to do that for Chrome (sort of...it doesn't quite get the full source):

  3c3
  < var app = Application('Safari');
  ---
  > var app = Application('Google Chrome');
  8c8,9
  <     let source = app.windows[0].currentTab.source()
  ---
  >     let ct = app.windows[0].activeTab
  >     let source = app.execute(ct, {javascript: "document.documentElement.innerHTML"})

Changing pages is also easy. For Safari,

  app.windows[0].currentTab.url = "..."

I don't remember what it is for Chrome, but it is similarly easy. And on both there is also the option of injecting JavaScript onto the page to do the navigation, if for some reason just setting the URL isn't good enough.

(No Firefox examples because Firefox has absolutely abysmal AppleScript support).

With JXA then I can write a handful of short command line utilities that can get the current URL, get the current page content, and navigate to a given URL and I've got everything I need.

(There might be a way to sort of make it work with Firefox. I believe (but am not certain) that with JXA I can tell the system send mouse-click events to anywhere I want in an application window. Firefox does provide enough AppleScript support to get the title of the current page. So maybe the bookmarklet approach, with JXA watching for the page changes and clicking the bookmarklet, would work).

[1] https://github.com/ultrafunkamsterdam/undetected-chromedrive...