OS X Accessibility Inspector (UIElementInspector) Tool for UI Scripting

Questions & Suggestions
, ,
1

OS X Accessibility Inspector (UIElementInspector) Tool for UI Scripting

UPDATED: 2019-03-07 21:28 GMT-6

Since I posted this topic nearly 3 years ago, I've learned a lot.
So, I no longer recommend this tool, as it does not work with all macOS's.
Instead, I suggest you try one of these tools:

  1. UIElementInspector provided as part of the free Xcode download (as @Tom suggests)
  2. Use the AppleScript tool by Chris @ccstone
    Front Window Analysis Tool for System Events
  3. For more complex UI scripting, you will benefit from UIBrowser, which works will all macOS including Mojave.

Sometimes an app, or even the OS X system itself, does NOT provide a direct scripting element that allows you to achieve your objective. You may then need to resort to UI scripting.

Here is a free tool, provided by Apple, that will help you identify the UI elements you need to script:

EXAMPLES OF ACCESSIBILITY INSPECTOR

For more details, see
A Strategy for UI Scripting in AppleScript using UIElementInspector

OTHER TOOLS

If you need to do a lot of UI scripting, or work with a very complicated app, then you may want to consider a professional tool, like UI Browser. At $55, it is not cheap, but could save you many hours of development time.

See UI Browser vs. Apple's UIElementInspector

1 Like
2

The Accessibility Inspector is also contained in Xcode, in case you have it already installed. Menu bar: Xcode > Open Developer Tool > Accessibility Inspector. (You can also launch it through Spotlight without Xcode, it’s in /Applications/Xcode.app/Contents/Applications/)

One problem with the Accessibility Inspector is that often it doesn’t provide you with the exact terms (UI element names) you need for a working UI script.

For example recently I had the problem to get the name of a selected reminder in the Reminders app. The correct “path” is this one:

text field 1 of UI element 1 of (rows whose value of attribute "AXSelected" is true) of table 1 of scroll area 1 of group 1 of splitter group 1 of window "Reminders"

But, as I first checked the reminder with Accessibility Inspector I couldn’t find all of the needed names of the elements. Some names correspond to the ones from accessibilityRoleDescription in Accessibility Inspector, for example table, scroll area, but others don’t (UI element 1, group, splitter group).

To get the correct element names for your script you have to query for UI elements, for example like this:

tell application "System Events"
	tell window "Reminders" of process "Reminders"
		UI elements
	end tell
end tell

This gives you {splitter group 1 of window "Reminders" of…etc. Then you have to repeat the same with tell splitter group 1 of window "Reminders" of process "Reminders" to get the next object, and so on. This way you get the correct (= usable) element names. It’s very well explained here.

Fortunately some friendly people have posted a script in the MacScripter forum that gives you the complete structure (with usable element names) of a given process in one go. (There are several versions of the script in the thread. I think the best one is the script in post #33.)

2 Likes
3

Tom, thanks for sharing your experience with getting the UI elements.

The script you referenced from MacScripter looks like another good tool to have in one's tool kit.

I haven't used either the Accessibility Inspector (AccInspt) or the above script, but just testing both against the Evernote app, here is what I found:

  1. The script (compiled/saved as an app) did NOT report anything for "menu" or "window" options.
  • It seems to work with the Finder, but not with Evernote
  1. The AccInspt seems to work well with Evernote.
  • I haven't tried to create an AppleScript using it, but the displays look encouraging, and also are dynamic:
    .

Hopefully I'll have some time in the next week or so to create a KM Macro that uses AppleScript UI scripting to provide a better interface for changing Evernote text format.

4

Run it from the Script Editor.

This is an output example of the script (only the very first lines):

I think you misunderstood my post. AccInsp indeed is working perfectly fine, no problem. The only “inconvenience” is the terminology from the AccInsp, which often not corresponds to what you actually will need in your script.

For example the AXSplitGroup which you see in your AccInsp screenshot has to be reworded as splitter group in your later AppleScript. AccInsp doesn’t tell you that. (Compare with line 2 in the screenshot of the script.)

Of course, for an experienced scripter this is not an issue since he knows all the right names off the top of his head :wink:

BTW, it seems that the “UIElementInspector” from the Apple download link you posted above is an old precursor of the Accessibility Inspector. I got version 1.4 from 2010 from that download. The current version of Accessibility Inspector is 4.1 (the one that comes with Xcode).

With the new version you can focus on an element by double-clicking in the hierarchy and it gives you a live view (overlay) of that element.

5

To throw in my fourteen cents, I have been using UI Browser enthusiastically and with wonderful results for many years. It has saved me hundreds of hours, taught me much about application UI structure, and made many things possible that I wouldn’t have figured out how to do using just UIElementInspector. Among other wonderful things it constructs pieces of AppleScript you can paste into your scripts for references, actions, and so on. $55 sounds pricey, but if you do a lot of UI Scripting — or if you would if it were easier — your sanity and abilities are worth much more than that. Thank you PFiddlesoft!

3 Likes
6

This sounds good. I will give it a try tomorrow.

7

I played around a bit with UI Browser. Here some (superficial) observations:

  • Basically it gives you the same information as the combination of Accessibility Inspector and the above mentioned AppleScript does give you. That is, you get both terminologies, the technical Accessibility element names and the names for System Events.

  • Sometimes it can go further down the hierarchy. For example, with the font menu in Evernote, the script drills down until pop up button "Font", whereas UI Browser goes two levels deeper (menu 1 > menu item [x]). It can do this because it runs in the background while you are opening the pop up menu.

  • It has some additional convenience features: for example, from the detected elements it can auto-compose the tell block for you. Very handy. Or it can execute UI actions, etc.

  • Sometimes it behaves sluggish. For example when targeting the Reminders application. But Reminders is known to be very slow/buggy when operated by an UI script, so UI Browser’s sluggishness may well serve as an indicator of what you have to expect from your future UI script :wink:

I agree with you: If you do a lot of UI scripting UI Browser is well worth the price. In my case —an occasional UI scripter— it’s a “very nice to have” tool and I’m not sure if I will buy it.

1 Like
8

I should add, with some embarrassment, that I am basing my opinions on my experience with the old UI Inspector and have not tried the new Accessibility Inspector. It may well be that Accessibility Inspector moves far enough in the direction of UI Browser’s capabilities to make the latter less of an advantage over the former.

9

I had the same thought, considering that UI Browser was born many years ago. But the uncomplicated access to System Event’s terminology is still an advantage. (Among others.)

10

All changes are basically superficial UI changes.

UI Browser is kept updated by its developer Bill Cheeseman who clearly has a greater interest in accessibility than Apple.

Apple regularly breaks accessibility in their own apps!

-Chris

1 Like