Front Window Analysis Tool Using AppleScript System Events (List UI Elements)

[hello]

Screenshots?

[ronald]

[Nige_S]

@ronald

Let's say that I am in Bear (could be any app), and I want to write an AppleScript to click on Note in the Menu, how do I go from the output of your macro to an AppleScript ?

At a complete guess: The simplest would be that each indent is a tell bock until the last, which is the action to take. So try this:

tell application "System Events"
    tell application process "Bear"
        tell window Bear
            tell splitter group 1
                tell scroll area 1
                    tell outline 1
                        tell row 1
                            tell UI element "Notes"
                                click UI element 3
                            end tell
                        end tell
                    end tell
                end tell
            end tell
        end tell
    end tell
end tell

But I'm no UI-scripting guru -- I'd take the above as the start point and try various tweaks until it works!

[ronald]

thanks very much for giving the issue some thought.

This is the error message in KM Engine log

2022-10-17 16:53:44 Action 11497178 failed: Execute an AppleScript failed with script error: text-script:112:117: script error: Expected end of line but found identifier. (-2741)
2022-10-17 16:53:44 Execute an AppleScript failed with script error: text-script:112:117: script error: Expected end of line but found identifier. (-2741). Macro “zz test” cancelled (while executing Execute AppleScript).

[ccstone]

@ronald

Let's say that I am in Bear (could be any app), and I want to write an AppleScript to click on Note in the Menu, how do I go from the output of your macro to an AppleScript ?

Hey @ronald,

The first thing to do is to examine the title of the macro.

It says: "Front Window Analysis Tool..." – it does not say menu analysis tool – or application analysis tool.

After doing that you might come to the (correct) conclusion that my macro will not give you the information needed to select the "Note" menu from the Bear menu bar.

Can it be done? Sure. Can my macro as written do it? Nyet.

You may want to try out @CJK's script which allows one to return a reference to the object under the mouse cursor to the clipboard. (Although wait a bit – there's a glitch in the code that needs fixing first.)

Front Window Analysis Tool Using AppleScript System Events (List UI Elements) - #40 by CJK

Your other alternative is to buy a copy of Script Debugger, which gives you tools to visualize the GUI hierarchy. These work particularly well with @CJK's script.

If I feel like it I might adapt @CJK's script to provide a hierarchical view like my macro does in addition to the objects it returns.

I'll have to think that over and decide it its worth my time.

-Chris

[ronald]

OK. Thank you. I have a better understanding now.

[Nige_S]

@ronald

OK. Thank you. I have a better understanding now.

So take it from basics and use AppleScript to find out what's available. Not having Bear, let's try in Script Editor itself and look for the "Find..." menu item.

We'll need to start with a menu bar -- how many of those are there?

tell application "System Events"
	tell application process "Script Editor"
		return every menu bar
	end tell
end tell
--> {menu bar 1 of application process "Script Editor" of application "System Events"}

Just the one, which is handy. We could look at everything in menu bar 1, but let's take a guess that the menu is named the same the screen displays it and try and get the contents:

tell application "System Events"
	tell application process "Script Editor"
		return every menu item of menu "Edit" of menu bar 1
	end tell
end tell
--> {menu item "Undo" of menu "Edit" of menu bar item "Edit" of menu bar 1 of application process "Script Editor"...

Looks promising. And if we search the results we find menu item "Find" of menu "Edit"..., which looks even more promising. So:

tell application "System Events"
	tell application process "Script Editor"
		return every menu item of menu item "Find" of menu "Edit" of menu bar 1
	end tell
end tell
--> {}

Whoops! Not so good. Ah, but "Find" has a sub-menu, so we have to get the menu before the menu's items:

tell application "System Events"
	tell application process "Script Editor"
		return every menu of menu item "Find" of menu "Edit" of menu bar 1
	end tell
end tell
--> {menu "Find" of menu item "Find" of menu "Edit"...

And we're back on track. And we know menus have menu items, so:

tell application "System Events"
	tell application process "Script Editor"
		return every menu item of menu "Find" of menu item "Find" of menu "Edit" of menu bar 1
	end tell
end tell
--> {menu item "Find…" of menu "Find"...

...and that first item looks to be the one we want. And remembering that "..." is an ellipsis, not three periods:

tell application "System Events"
	tell application process "Script Editor"
		click menu item "Find…" of menu "Find" of menu item "Find" of menu "Edit" of menu bar item "Edit" of menu bar 1
	end tell
end tell

And it works!

OK, so it's a long-winded way of doing things -- but it is a very cheap way And doing similar in Bear might get you started on finding what you want.

[ccstone]

Hey Guys,

When I'm manually exploring an app, and I don't jump right to an object using @CJK's script I use this method:

tell application "System Events"
   tell application process "Finder" -- the process you're exploring...
      
      set diagnosticsList to {¬
         "----- PROPERTIES -----", ¬
         properties, ¬
         "----- UI ELEMENTS -----", ¬
         UI elements, ¬
         "----- ATTRIBUTES -----", ¬
         attributes, ¬
         "----- ACTIONS -----", ¬
         actions, ¬
         "----- END -----"}
      
   end tell
end tell

This method gets painful in the Script Editor though – which is one reason I created the analysis tool in the first place.

Script Debugger makes this much easier, because it can pretty print lists and also present an object in an object-browser format.

List:

Object-Browser:

Using @CJK's script I also have a method of jumping straight to a specific object in Script Debugger. It's not quite as reliable as UI Browser, but it works 99.x % of the time.

-Chris

[ronald]

I will give it a try with Bear. thank you

[ronald]

Just curious: do you have script debugger ?

[Nige_S]

@ronald

Just curious: do you have script debugger ?

Not me. I do try it every now and then but can never get into it and go back to Script Editor -- though I think that would change if I made an effort and/or was more than an occasional AppleScripter.

Before digging deeper into Bear's UI with the above, check out @appleianer's post about UI Browser. And rejoice!

[ronald]

thanks very much

[ccstone]

You realize that there's a free version that is far more powerful than Apple's Script Editor?

[Nige_S]

@ccstone

You realize that there's a free version that is far more powerful than Apple's Script Editor?

Yep -- and I've got it installed on at least two machines here and another couple in the office. I've dabbled with SD since (at least!) version 4.

It's totally me -- I've just never been grabbed by it. Probably because the little AppleScripting I do is simple enough that Script Editor is more than adequate. And I'm equally sure that if I scripted more, or merely made the effort to use SD properly, I'd soon be a convert.

Fear of change + laziness == inertia

[CJK]

@ccstone

there's a glitch in the code that needs fixing first

There is...?

[noisneil]

@CJK, I had a play with your brilliant script and came up with something that might be worth developing.

It uses the relevant data in the output of your script to generate an AppleScript tell block.

Here's a quick demo with Logic Pro:

I'm certain there will be circumstances under which it will fail, but I think it might have some potential.

Create Tell Block for UI Interaction - "UI Browser Lite"

This is a work-in-progress attempt to replicate the functionality of UI Browser that I find most useful. I started work on this last night, and am quite excited about it. It builds on a clever script by @CJK, which returns information pertaining to the UI element under the mouse pointer, and uses this data to automatically create a functioning AppleScript tell block. Usage: Hover your mouse pointer over the element you'd like to interact with and trigger the macro, which comes with its trigge…

[olifrost]

I was wondering if the same script by @CJK could be used to change a macro's behaviour based on the element the mouse is hovering over

For example, in Logic I can search the results for the description "sendButton" or "audioPlugin" to see where I am in the mixer.

That worked, but it takes a few seconds to get the result, which is too long for that purpose. I'm not sure where to go from there.

[noisneil]

As much as I hate to resort to found images, Logic's lack of scriptability often forces my hand:

Mouse Over Plugin?.kmmacros (60 KB)

Macro screenshot

Incidentally, when I ran @CJK's script, I couldn't see any reference to sendButton or audioPlugin, just button.

Formatted Script Output

UI element:button 1 of group 2 of UI element 1 of UI element 1 of group 3 of list 1 of group 2 of window Untitled - Tracks of application process Logic Pro X
minimum value:missing value
orientation:missing value
position:37, 641
class:button
accessibility description:open
role description:button
focused:missing value
itle:missing value
size:32, 16
help:missing value
entire contents:
enabled:missing value
maximum value:missing value
role:AXButton
value:missing value
subrole:missing value
selected:missing value
name:missing value
description:open
AXWindow:window Untitled - Tracks of application process Logic Pro X
AXRole:AXButton
AXCustomActions:missing value
AXDescription:open
AXRoleDescription:button
AXFrame:37, 641, 69, 657
AXValue:missing value
AXParent:group 2 of UI element 1 of UI element 1 of group 3 of list 1 of group 2 of window Untitled - Tracks of application process Logic Pro X
AXTopLevelUIElement:missing value
AXSize:32, 16
AXPosition:37, 641
_AXActions:AXPress

P.S. If you want the output formatted to be more readable, as above:

Get Info for UI Element Under Mouse.kmmacros (48 KB)

Macro screenshot

[olifrost]

Thanks for the macros! The first one has another thing I've been looking for which is how to search the screen area around the mouse.

I believe "sendButton" or "audioPlugin" only show up on a blank slot, under "accessibility description:". BTT has conditions to scan the hovered element, but that's another app…

[noisneil]

I've covered plugins and sends here, in case that's of any help.

Also, PlugSearch now has lots of new handy features that are worth checking out. IMHO it's a must-have for Logic users.