JXA GUI Scripter

Macro Library
,
1

One of my biggest quarms with GUI Scripting in Applescript was not really knowing what which GUI element did what… Fortunately when JXA came out I had a surge of inspiration to start back on an old project which relied heavily on GUI scripting. I ran into the same issue that I had before so I decided to solve it.

Element Scripter

A useful Mac utility which scans GUIs for scriptable elements and returns the scripting path of the element and descriptive information related to the element. This allows a developer to quickly find which UI Element they need to press/set the value of and gives them the scripting address for them to quickly do so.

Example output:

Address|Title|Name|Description|Help|Role|Enabled|Focused|Position|Size|Value
Application("System Events").applicationProcesses.byName("ScreenFlow").windows.byName("Untitled").buttons.at(0)|||close button||AXButton|true|false|7,26|14,16|
Application("System Events").applicationProcesses.byName("ScreenFlow").windows.byName("Untitled").buttons.at(1)|||full screen button|this button also has an action to zoom the window|AXButton|true|false|47,26|14,16|
Application("System Events").applicationProcesses.byName("ScreenFlow").windows.byName("Untitled").buttons.at(2)|||minimize button||AXButton|true|false|27,26|14,16|
Application("System Events").applicationProcesses.byName("ScreenFlow").windows.byName("Untitled").groups.at(0)|||group||AXGroup||false|0,45|1631,1|
Application("System Events").applicationProcesses.byName("ScreenFlow").windows.byName("Untitled").buttons.at(3)|||Fast Forward|fast forward|AXButton|true|false|744,811|42,38|
Application("System Events").applicationProcesses.byName("ScreenFlow").windows.byName("Untitled").buttons.at(4)|||Play|play|AXButton|true|false|704,811|41,38|
Application("System Events").applicationProcesses.byName("ScreenFlow").windows.byName("Untitled").buttons.at(5)|||Rewind|rewind|AXButton|true|false|663,811|42,38|
Application("System Events").applicationProcesses.byName("ScreenFlow").windows.byName("Untitled").buttons.at(6)|||Resize Canvas|resize canvas|AXButton|true|false|11,811|41,38|
Application("System Events").applicationProcesses.byName("ScreenFlow").windows.byName("Untitled").buttons.at(7)|||Upload Manager||AXButton|true|false|1603,814|37,31|

Using the application:

  1. Download Element Scripter.app.zip
  2. Go to “System Preferences > Security & Privacy > Accessibility” and allow “Element Scripter” to ‘control’ your computer.
  3. Run the software.
  4. Choose the software you want to analyse from the list.
  5. Choose the window you want to analyse from the list.

You will be notified when the script has finished and the contents will be inserted into a new textedit document.

From here you can do many things:

  1. Replace | with Tab characters in text edit and paste it into numbers
  2. Use MSExcel’s Text to column with a delimeter “|”
  3. Transfer the data to google sheets and use the split(a1,"|") formula

etc.

The source code can be found here.

It may be beneficial in some circumstances to use the single-app script.

Run via keyboard shortcut!

It may also be beneficial to setup a keyboard shortcut to run the application, either:

  1. With Keyboard Maestro
  2. System Preferences > Keyboard > Shorcuts > Services / App Shortcuts (which ever takes your fancy).

If you choose service, remember to setup the JXA as a Service with Automator!

Happy scripting!

5 Likes
2

Thanks for sharing. This looks very useful. :thumbsup:

3

Hey @sancarn,

Your single app script fails with TextEdit when run from the Script Editor on macOS 10.12.2.

Error -1700: Can't convert types.

-Chris

4

Hmmm... Try the latest version of the single-app script? I wouldn't know for sure without testing the application itself and finding where the bug is... But now at least I've added some error checking so if one object isn't retrievable it will use a "Can't get element" message... Hopefully this fixes the issue. Otherwise maybe it is an OS issue, though this would surprise me...

One thing I dispise about JXA - the error messages are awful!

5

The new version works – although it would seem there are a bunch of unknown objects.

Might I suggest too that the script activate the reporting app (TextEdit).

I was left hanging in Script Editor and didn't notice for a while that there was a result in TextEdit.

-Chris

6

That's a good idea, I thought I had added that feature already, but you're right, I too have spent many minutes waiting for a response which is already there... Will add it. Also should probably add these things to the application!

Although it would seem there are a bunch of unknown objects

Yeah I'm not totally sure how to get around that. I get the same thing with Safari. Will look into it tonight.