Hacker News Reader

Comment by milansuk | Hacker News Reader

Do operation systems have APIs, so the program can send them what is on the screen? I mean just description like: There is the button with title "Push me" in the rectangle(50, 50, 100, 70), etc.

Or are there any "screen reader" libraries?

View on HN

Even systems like Qt, where things are drawn using non-native widgets support that (I was long proponent of wxWidgets, because I thought assistive tech is only support only if you use native widgets. The idea of API where you explain what you are showing, and how to interact with it never came to my mind, but with ARIA - like any browser that's the way to go).

Not only that, but I thhink assistive tech should allow you do to easier GUI testing too, but haven't digged into that idea yet (probably folks already are)

Yes, OSes have APIs. Another commenter talked about the Mac one, Linux has AT-SPI and Windows has MSAA (historic) and UIA. If I'm not mistaken, they generally are about representing the UI as a tree of items as this is what the accessibility hardware expects / works best with.

Yes. I don’t know about Windows but I know that for OSX you need to implement the NSAccessibility protocol in AppKit. This gives you a way to describe the hierarchy of the UI and to notify the system of important changes.

There's a10n standards defined by w3c, https://www.w3.org/TR/using-aria/.

Yes they do, at least Apple, Google and Microsoft ones.

On GNU/Linux, I think the only thing that exists in the Gtk+ features that were originally sponsored by Sun.

Aren't you then trying to bridge an immediate mode API (rendering controls) and a retained mode API (assistive tech).

Yes. This has influenced the design of my AccessKit [1] project. With AccessKit, the application (or GUI toolkit) pushes tree updates to the platform adapter, which maintains a complete tree that it uses to implement the platform-specific accessibility API. Each application-supplied tree update can be a full tree or just the nodes that have changed. So an immediate-mode GUI can push a full tree every frame. There just has to be some way of keeping node IDs stable across frames.

[1]: https://github.com/AccessKit/accesskit

Reply on news.ycombinator.com