Using Accessibility Labels? (Java / Qt Apps)

Questions & Suggestions
,
#1

Most applications contain accessibility metadata that is used by screen readers for visually impaired users. There are some automation tools for Windows and Linux that utilise this information so that you can avoid relying on "found image" type macros and instead do something more like "click button with a11y label = login".

Does Keyboard Maestro have a way to access this metadata? It would be extremely useful for automating non-native Mac applications, specifically IntelliJ in my case. But it could be applied to all sorts of other apps.

#2

That's a fascinating question. I always like using accessibility features of MacOS to solve problems. I have used accessibility in MacOS to solve problems with apps before. This idea of yours would open up new possibilities if it were possible. At the moment, my gut is saying no. But I don't give up easily. I'm hoping that someone with more AppleScript knowledge can answer this. But there may not be an API for it.

It's inevitable that many apps don't provide any accessibility metadata.

Another thing to keep in mind is that MacOS Monterrey has a SUPERB ability to perform OCR on any window, and I've used it to help solve problems with apps like the ones you are alluding to. I can't begin to tell you how superb the OCR abilities are.

It's inevitable that ALMOST ALL apps provide some form of text display which OCR can access. My sails are set in the direction of using OCR (and away from Find Image) to deal with difficult apps. But I'm intrigued by your idea.

A hybrid solution, which may not help you, is for OCR (or Find Image) to read the accessibility cues from the screen. This is the kind of hacking that I love to do.

#3

It looks like AppleScript has access to it, as was expected, but AppleScript can be baroque even for seemingly simple tasks. Check out the documentation for a few more details (see Clicking a Button) Mac Automation Scripting Guide: Automating the User Interface

In my experience, wiring together a KB macro takes 5x LESS time than trying to write an equivalent AppleScript action, so it would be great to work with similar functionality in KB.

#4

That's wonderful. But I'm going to let other wizards take it from here, as I'm not qualified to give advice on AppleScript.

#5

Personally, I much prefer to use JXA (JavaScript for Automation), than AppleScript. The documentation is sparse to non-existent, but at least JavaScript is a "real" programming language. If you need help, just holler.