Front Window Analysis Tool Using AppleScript System Events (List UI Elements)

[ccstone]

You realize that there's a free version that is far more powerful than Apple's Script Editor?

[Nige_S]

@ccstone

You realize that there's a free version that is far more powerful than Apple's Script Editor?

Yep -- and I've got it installed on at least two machines here and another couple in the office. I've dabbled with SD since (at least!) version 4.

It's totally me -- I've just never been grabbed by it. Probably because the little AppleScripting I do is simple enough that Script Editor is more than adequate. And I'm equally sure that if I scripted more, or merely made the effort to use SD properly, I'd soon be a convert.

Fear of change + laziness == inertia

[CJK]

@ccstone

there's a glitch in the code that needs fixing first

There is...?

[noisneil]

@CJK, I had a play with your brilliant script and came up with something that might be worth developing.

It uses the relevant data in the output of your script to generate an AppleScript tell block.

Here's a quick demo with Logic Pro:

I'm certain there will be circumstances under which it will fail, but I think it might have some potential.

Create Tell Block for UI Interaction - "UI Browser Lite"

This is a work-in-progress attempt to replicate the functionality of UI Browser that I find most useful. I started work on this last night, and am quite excited about it. It builds on a clever script by @CJK, which returns information pertaining to the UI element under the mouse pointer, and uses this data to automatically create a functioning AppleScript tell block. Usage: Hover your mouse pointer over the element you'd like to interact with and trigger the macro, which comes with its trigge…

[olifrost]

I was wondering if the same script by @CJK could be used to change a macro's behaviour based on the element the mouse is hovering over

For example, in Logic I can search the results for the description "sendButton" or "audioPlugin" to see where I am in the mixer.

That worked, but it takes a few seconds to get the result, which is too long for that purpose. I'm not sure where to go from there.

[noisneil]

As much as I hate to resort to found images, Logic's lack of scriptability often forces my hand:

Mouse Over Plugin?.kmmacros (60 KB)

Macro screenshot

Incidentally, when I ran @CJK's script, I couldn't see any reference to sendButton or audioPlugin, just button.

Formatted Script Output

UI element:button 1 of group 2 of UI element 1 of UI element 1 of group 3 of list 1 of group 2 of window Untitled - Tracks of application process Logic Pro X
minimum value:missing value
orientation:missing value
position:37, 641
class:button
accessibility description:open
role description:button
focused:missing value
itle:missing value
size:32, 16
help:missing value
entire contents:
enabled:missing value
maximum value:missing value
role:AXButton
value:missing value
subrole:missing value
selected:missing value
name:missing value
description:open
AXWindow:window Untitled - Tracks of application process Logic Pro X
AXRole:AXButton
AXCustomActions:missing value
AXDescription:open
AXRoleDescription:button
AXFrame:37, 641, 69, 657
AXValue:missing value
AXParent:group 2 of UI element 1 of UI element 1 of group 3 of list 1 of group 2 of window Untitled - Tracks of application process Logic Pro X
AXTopLevelUIElement:missing value
AXSize:32, 16
AXPosition:37, 641
_AXActions:AXPress

P.S. If you want the output formatted to be more readable, as above:

Get Info for UI Element Under Mouse.kmmacros (48 KB)

Macro screenshot

[olifrost]

Thanks for the macros! The first one has another thing I've been looking for which is how to search the screen area around the mouse.

I believe "sendButton" or "audioPlugin" only show up on a blank slot, under "accessibility description:". BTT has conditions to scan the hovered element, but that's another app…

[noisneil]

I've covered plugins and sends here, in case that's of any help.

Also, PlugSearch now has lots of new handy features that are worth checking out. IMHO it's a must-have for Logic users.

[DaveRI]

Thanks very much... I just found this Macro and it was invaluable for finding a cryptic CheckBox.
and FYI it works just fine under Sequoia

[noisneil]

Is it just me or is anyone else having problems with this script since updating to Sequoia? It seems to just run and run without outputting anything.

Edit: actually I seem to be having a few AppleScript-related issues today. Must be the weather...

[Nr.5-need_input]

Hello Neil (@noisneil)

@noisneil

Is it just me or is anyone else having problems with this script since updating to Sequoia? It seems to just run and run without outputting anything.

Well, I don’t have Sequoia on any of my Setups - but this OS Version seems to be the worst in accessibility via GUI Scripting from what I’ve read and heard so far.

Can you imagine that you aren’t able to scan incoming Notifications from anywhere in the Notification Center ?! At least without some tools which have to rely on private API‘s ?!

Yes, you’re not at the wrong place… this has become an reality…

I don’t know where this is going but it seems to be that Apple is going to ditch GUI Scripting Support until there’s nothing left. Sorry for being so harsh … but this is a Bummer … believe me - I am not happy about this.

What’s automation without Scriptability ?!

Sorry for being rude and maybe a little bit off topic here - but this must have been said.

Greetings from Germany

Tobias

[Nige_S]

@noisneil

Is it just me or is anyone else having problems with this script since updating to Sequoia? It seems to just run and run without outputting anything.

Are you talking about the "Window Analysis Tool for System Events v1.50" macro? I've just downloaded it and it seems to be working (I assume -- it creates an UI element list anyway).

macOS 15.1.1; tested against Safari, KM Editor, TextEdit, Mail.

[noisneil]

Sorry, false alarm. Nothing to do with this tool; none of my AS stuff was working but a restart fixed it. Not sure what was going on there.