Front Window Analysis Tool Using AppleScript System Events (List UI Elements)

I downloaded the app, and my analysis tool works fine with it.

UI Browser has some problems with it though and can crash when trying to get certain information.

-Chris

Hi ccstone, thank you for your response, how do I overcome the problem that I am facing?

Hey Ali,

How could I possibly know?

If I can't reproduce exactly what you're doing and test – then I can't help much.

You could join the newly formed UI Browser Users Group and provide more information about what you're doing.

Perhaps someone in the group has used Daylite.

-Chris

In fact using BBEdit as editor folding is just around the corner... the simple tuning is to change ".txt" file name extension in ".py" in this AppleScript handler:

Explanation:

The conventions to encode in a compact way the hierarchy of the GUI elements are identical to that of the Python code: it is therefore enough to take full advantage of this homology by transforming the temporary text file into a pseudo Python file, by changing the extension, so that BBEdit automatically activates the folding :wink:

Before:

After
For example, "scroll bar 1" elements are folded:

-Alain

3 Likes

@alain, that's very clever! :+1:

Indeed!

Somehow I always knew that Python must be good for something. :grin:

3 Likes

2 posts were split to a new topic: How Do I Access Specific Window Elements?

Hey @ccstone sorry for maybe asking super basic questions about this macro... but I am like the rookie of the rookies of the rookies when it comes to this subject.

So my question: this macro only shows a list of elements, right? That's the sole purpose or am I missing something else?

Would it be possible for the script to identify the cursor's position and based on that, either show the information for that element only (similar to UI Browser) or even if it shows the long list, maybe add some unique information next to that element that we could easily search for using the Search bar inside TextEdit?

Again, sorry for maybe some silly questions, but my knowledge when it comes to this is like -1000, but I would like to start using more AS when it comes to KM instead of using Found Image actions whenever possible.

Thanks! :slight_smile:

1 Like

I hope I'm not stepping on any toes. I remember posting my handler that does exactly this on this forum a couple of years ago, but to save you (and me) from finding it, here's its most recent incarnation:

use framework "AppKit"
------------------------------------------------------------------------
return the infoForUIElementUnderMouse()
------------------------------------------------------------------------
### HANDLERS:
on mouseCoordinates()
		tell the current application to tell mouseLocation()'s {x, y} ¬
				of its NSEvent & NSHeight(its NSScreen's mainScreen's ¬
				frame) to return the {(item 1), (item 3) - (item 2)}
end mouseCoordinates

on infoForUIElementUnderMouse()
		return infoForUIElement at the mouseCoordinates()
end infoForUIElementUnderMouse

on infoForUIElement at __Ref as {list, record, reference}
		local UIElement
		
		tell application id "com.apple.SystemEvents"
				if {__Ref}'s specifiers = {} then
						tell __Ref to if its class ≠ list ¬
								then set __Ref to its {x, y}
						set __Ref to click at __Ref
				end if
				
				set UIElement to __Ref
				if the UIElement = missing value then return {}
				
				script Object
						property parent : UIElement
						property AXAttributes : a reference to (the ¬
								attributes in me whose (name is not ¬
								"AXURL") and (name is not "AXPath"))
						property AXValues : value of AXAttributes
						property AXList : the name of AXAttributes
						property AXRecord : a reference to the the ¬
								{«class usrf»:my AXList}'s contents
				end script
				
				
				set my text item delimiters to linefeed & linefeed
				tell (a reference to the Object's AXList) ¬
						to set the contents to paragraphs ¬
						of (it as text) & ""
				
				tell the Object to repeat with i from 1 ¬
						to the length of its AXValues
						set its AXList's item (i * 2) to ¬
								item i of its AXValues
				end repeat
				
				return {UI element:the Object's contents} ¬
						& the Object's properties ¬
						& (the Object's AXRecord as any) ¬
						& the {_AXActions:the name of ¬
						every action in the Object}
		end tell
end infoForUIElement
-------------------------------------------------------------------❮END❯

The handler you being called for your purposes is infoForUIElementUnderMouse(). That handler first retrieves the mouse pointer's coordinates by calling the handler mouseCoordinates(). This is passed to the handler that does the real work, infoForUIElement, which enumerates and evaluates all properties, attributes, and actions that belong to the UI element identified as occupying the same coordinates directly under the mouse pointer.

A couple of points:

  1. Should you ever try to call infoForUIElement yourself, and wish to pass it a reference to a UI element, this must resolve to a single UI element class object belonging to System Events, and not a collection of objects.

  2. There are two points in the script where it can occasionally hang, neither of which are the fault of the script per se, but rather caused by System Events stalling. This most likely occurs when the element underneath the mouse belongs to an application that hasn't exposed its object hierarchy very well (or at all): therefore, it can happen with Electron-based applications, which will often stall at the start of infoForUIElement when trying to perform a click; and I've noticed it happens with some objects in Monterey's System Preferences app despite passing a perfectly good direct reference to infoForUIElement.

But, on the whole, it works very well for me in Monterey, and I use it regularly.

7 Likes

Note that @CJK's script fails in macOS 10.14.6 Mojave.

Although there is a version which does work on that OS, which I have and use now and then.

Any details on the nature of the failure ?

This line fails to compile:

set the UIElement is missing value then return {}

Presumably should be:

if the UIElement is missing value then return {}
2 Likes

Thanks.

Right you are. I've made the correction and a few other edits, mostly stylistic, with the exception of the class list for the at parameter of infoForUIElement where I've swapped out point for list.

3 Likes

Hello @ccstone I followed your advice. Is there any reference you could suggest to help me convert the output to an AppleScript

For example, simply to press the Notes Menu Item, how would i write the AppleScript based on the following

1- I imagine that you use a standard AppleScript wrapper

2- how do you add the info below into the wrapper ?

thanks very much

application Process "Bear"
	window Bear
		splitter group 1
			scroll area 1
				outline 1
					row 1
						UI element "Notes"
							static text "Notes"
							image 1
							UI element 3

What notes menu item? The one in the menu bar?

1 Like

I will make my question more generic.
You offered a solution for non expert users in order to avoid having to resort to UI browser.
With UI browser, the user records the screen → is given the AppleScript on a silver platter (the report to which one has to add the wrapper).
What I am asking is what the non expert should do with the output of your macro.
Let's say that I am in Bear (could be any app), and I want to write an AppleScript to click on Note in the Menu, how do I go from the output of your macro to an AppleScript ?
thank you Chris

Screenshots?

1 Like

1 Like

At a complete guess: The simplest would be that each indent is a tell bock until the last, which is the action to take. So try this:

tell application "System Events"
    tell application process "Bear"
        tell window Bear
            tell splitter group 1
                tell scroll area 1
                    tell outline 1
                        tell row 1
                            tell UI element "Notes"
                                click UI element 3
                            end tell
                        end tell
                    end tell
                end tell
            end tell
        end tell
    end tell
end tell

But I'm no UI-scripting guru -- I'd take the above as the start point and try various tweaks until it works!

1 Like