Front Window Analysis Tool Using AppleScript System Events (List UI Elements)

Well, yeah - Xcode is free, isn't it? :slight_smile:

I’ll keep thinking on this. Seems to me there’s got to be a way to create a good view, in HTML. I’m not an HTML wiz by any means, but if I come up with something I’ll let you know.

You just focus on giving us good results. We’ll focus on everything else. :smile:

Are you talking about normal code-folding?

Or is there something better?

-Chris

With Atom, you can collapse each node and level. I don’t think you can collapse all or expand all, but at least it helps hide the stuff you’re not interest in.

I pasted you output into it, and it works fine.

Atom > Edit > Folding > Fold All | Unfold All

⌥⇧⌘[
⌥⇧⌘]

1 Like

Chris, that is HUGE! :thumbsup:

The SD Results Explorer Window gives you a great drill-down of the UI for whatever app it is focused on.

1 Like

Yep.  :smile:

Script Debugger is an absolute game-changer for AppleScripters.

The forthcoming Script Debugger 6 will have significant upgrades that support AppleScriptObjC.

Hopefully Script Debugger 7 down the road will also support JXA.

-Chris

Don't leave home without it. It is inconceivable to do more than a little work in AppleScript and not use this. Don't the the price dissuade you: it will repay itself month after month, possibly week after week, or even day after day.

UI Browser too. It not only shows UI structure but writes pieces of AppleScript for you.

Really.

Unfortunately (so far), no JXA support. From the web site:

Script Debugger 5 does not support the new JavaScript for Automation feature introduced by Apple in Yosemite (Mac OS X 10.10).

Ironically, we once offered our own JavaScript implementation for Script Debugger versions 2 and 3 and discontinued it due to lack of interest. We are watching how much adoption Apple’s JavaScript for Automation receives. Offering debugging for Apple’s JavaScript implementation is non-trivial so this isn’t something we can easily add back into Script Debugger.

If sufficient interest in JavaScript for Automation develops we will seriously consider adding it to Script Debugger, or possible create a JavaScript specific version of the product.

Hi I can’t seem to get this to work for the application named ‘Daylite’

I get this error message as the result:

/var/folders/ct/lsmxdhzn2hv4p378kqlxxf8c0000gn/T/Keyboard-Maestro-Script-2FB29C27-5AC2-401E-9945-A6DC96D73819:1133:1148: execution error: System Events got an error: AppleEvent timed out. (-1712)

Is anybody else experiencing this bug too?

Hey Ali,

Are you talking about Daylite from Market Circle?

-Chris

I downloaded the app, and my analysis tool works fine with it.

UI Browser has some problems with it though and can crash when trying to get certain information.

-Chris

Hi ccstone, thank you for your response, how do I overcome the problem that I am facing?

Hey Ali,

How could I possibly know?

If I can't reproduce exactly what you're doing and test – then I can't help much.

You could join the newly formed UI Browser Users Group and provide more information about what you're doing.

Perhaps someone in the group has used Daylite.

-Chris

In fact using BBEdit as editor folding is just around the corner... the simple tuning is to change ".txt" file name extension in ".py" in this AppleScript handler:

Explanation:

The conventions to encode in a compact way the hierarchy of the GUI elements are identical to that of the Python code: it is therefore enough to take full advantage of this homology by transforming the temporary text file into a pseudo Python file, by changing the extension, so that BBEdit automatically activates the folding :wink:

Before:

After
For example, "scroll bar 1" elements are folded:

-Alain

3 Likes

@alain, that's very clever! :+1:

Indeed!

Somehow I always knew that Python must be good for something. :grin:

3 Likes

2 posts were split to a new topic: How Do I Access Specific Window Elements?

Hey @ccstone sorry for maybe asking super basic questions about this macro... but I am like the rookie of the rookies of the rookies when it comes to this subject.

So my question: this macro only shows a list of elements, right? That's the sole purpose or am I missing something else?

Would it be possible for the script to identify the cursor's position and based on that, either show the information for that element only (similar to UI Browser) or even if it shows the long list, maybe add some unique information next to that element that we could easily search for using the Search bar inside TextEdit?

Again, sorry for maybe some silly questions, but my knowledge when it comes to this is like -1000, but I would like to start using more AS when it comes to KM instead of using Found Image actions whenever possible.

Thanks! :slight_smile:

1 Like

I hope I'm not stepping on any toes. I remember posting my handler that does exactly this on this forum a couple of years ago, but to save you (and me) from finding it, here's its most recent incarnation:

use framework "AppKit"
------------------------------------------------------------------------
return the infoForUIElementUnderMouse()
------------------------------------------------------------------------
### HANDLERS:
on mouseCoordinates()
		tell the current application to tell mouseLocation()'s {x, y} ¬
				of its NSEvent & NSHeight(its NSScreen's mainScreen's ¬
				frame) to return the {(item 1), (item 3) - (item 2)}
end mouseCoordinates

on infoForUIElementUnderMouse()
		return infoForUIElement at the mouseCoordinates()
end infoForUIElementUnderMouse

on infoForUIElement at __Ref as {list, record, reference}
		local UIElement
		
		tell application id "com.apple.SystemEvents"
				if {__Ref}'s specifiers = {} then
						tell __Ref to if its class ≠ list ¬
								then set __Ref to its {x, y}
						set __Ref to click at __Ref
				end if
				
				set UIElement to __Ref
				if the UIElement = missing value then return {}
				
				script Object
						property parent : UIElement
						property AXAttributes : a reference to (the ¬
								attributes in me whose (name is not ¬
								"AXURL") and (name is not "AXPath"))
						property AXValues : value of AXAttributes
						property AXList : the name of AXAttributes
						property AXRecord : a reference to the the ¬
								{«class usrf»:my AXList}'s contents
				end script
				
				
				set my text item delimiters to linefeed & linefeed
				tell (a reference to the Object's AXList) ¬
						to set the contents to paragraphs ¬
						of (it as text) & ""
				
				tell the Object to repeat with i from 1 ¬
						to the length of its AXValues
						set its AXList's item (i * 2) to ¬
								item i of its AXValues
				end repeat
				
				return {UI element:the Object's contents} ¬
						& the Object's properties ¬
						& (the Object's AXRecord as any) ¬
						& the {_AXActions:the name of ¬
						every action in the Object}
		end tell
end infoForUIElement
-------------------------------------------------------------------❮END❯

The handler you being called for your purposes is infoForUIElementUnderMouse(). That handler first retrieves the mouse pointer's coordinates by calling the handler mouseCoordinates(). This is passed to the handler that does the real work, infoForUIElement, which enumerates and evaluates all properties, attributes, and actions that belong to the UI element identified as occupying the same coordinates directly under the mouse pointer.

A couple of points:

  1. Should you ever try to call infoForUIElement yourself, and wish to pass it a reference to a UI element, this must resolve to a single UI element class object belonging to System Events, and not a collection of objects.

  2. There are two points in the script where it can occasionally hang, neither of which are the fault of the script per se, but rather caused by System Events stalling. This most likely occurs when the element underneath the mouse belongs to an application that hasn't exposed its object hierarchy very well (or at all): therefore, it can happen with Electron-based applications, which will often stall at the start of infoForUIElement when trying to perform a click; and I've noticed it happens with some objects in Monterey's System Preferences app despite passing a perfectly good direct reference to infoForUIElement.

But, on the whole, it works very well for me in Monterey, and I use it regularly.

7 Likes