> Maybe somebody should move this functionality out into a standalone library?
I've been thinking about this for a while. Below is an edited excerpt from a message that I wrote to a colleague a few months ago, about how I think I should go about such a project:
---
I've been thinking about the problem of multiple programming languages, and even multiple programming styles within the same language. Do we implement a C library that would make hard-core Unix and Linux folks happy? A library in a subset of C++ that some developers of games and graphics would be comfortable with? A more modern style of C++ that would make some other developers happy? And where would that leave folks working in Java, or C# like the Unity crowd? Of course, different platforms have varying levels of support for different languages. And there are more languages coming into popularity or on the horizon, like Swift and Rust.
So I think what we really want is a cross-platform message format or protocol, with multiple implementations: multiple providers on the application/toolkit side, and multiple client libraries on the platform side. The client libraries could be written in each platform's native language (e.g. C++ for Windows, Java for Android, Swift for Apple platforms, or JavaScript for the web), and separately, provider libraries could be implemented in multiple languages. All major programming languages can work with binary data buffers. And since none of us want to work directly with those raw bytes, we could use an existing standard like Google Protocol Buffers, which already has implementations in several languages. Initially, for desktop and mobile platforms as well as web applications, this protocol would just be used internally between components in the same application process. But I can dream about platforms themselves adopting the protocol someday. There would have to be glue layers between cross-platform providers and platform-specific clients, but if we design this right, the glue could be kept pretty thin, with most of the complexity being kept on one side or the other, so we don't have a multiplication of effort for n toolkits or applications on m platforms.
I think the protocol should be push-based, rather than pull-based like Windows UI Automation and some other accessibility APIs. That is, the application/toolkit would push full information about objects in the accessibility tree when it first creates that tree, and incremental updates when objects are created or destroyed, when properties change, when text content changes, etc. That's probably the only thing that's going to work for the web platform, and I think it's a model that other platforms would do well to adopt. (It's probably too late for Windows, but I dream of replacing the current accessibility model on desktop Linux someday, after my non-compete with Microsoft expires.) The challenge with a push model is making it efficient; we don't want to re-push the whole contents of a large text box when the user types a single character, and when we do need to push all contents, we need to do so efficiently. And sometimes we really do need to push a lot of information. For example, for a text box, we need the screen coordinates of every character, plus all of the boundaries between words.
Luckily, a push-based accessibility architecture has been done before, internally in the Chromium browser engine. As you probably know, Chromium has a multi-process model, where web pages are rendered in sandboxed processes, which communicate with a master browser process that interfaces with the OS. The browser process is not allowed to do blocking IPC requests into the renderer processes, so it can't implement synchronous, pull-based accessibility APIs like UIA in the obvious way. So the Chromium team implemented a protocol where the renderer processes push their accessibility trees, and incremental updates to those trees, over to the browser process, which can then store the trees in memory and then provide information to UIA or other APIs on request. The pushed trees are comprehensive, including the information I mentioned about text. Chromium does this using its own binary protocol called Mojo, which is kind of like Protocol Buffers but strongly tied to Chromium. So, while I'll take design inspiration from Chromium, I probably won't take the actual protocol or code.
I also want my protocol to scale down to embedded platforms. Accessibility on devices running embedded software (i.e. not Windows or another general-purpose OS) is basically an unsolved problem; as far as I know, device makers have to implement their own custom self-voicing interfaces, if they do anything about the problem at all. But imagine a standard where a user can pull out their smartphone, connect to the specialized device over Bluetooth or WiFi, and get an accessible interface to the device on their phone. Yeah, I'm swinging for the fence with this project. To pull this off, I think the protocol would need to be not only push-based but streaming, allowing providers to send out accessibility information without having to build up and maintain much extra state in memory. This would also help with the immediate-mode GUIs that are used in some games and game development tools. If we design the protocol right, these immediate-mode toolkits should be able to push out accessibility information and events at the same time that they're making OpenGL (or similar) calls to draw the current frame, again without having to hold much extra state in memory, which is something that these toolkits try to avoid.