Front Window Analysis Tool Using AppleScript System Events (List UI Elements)

Macro Library
, , ,
40

I hope I'm not stepping on any toes. I remember posting my handler that does exactly this on this forum a couple of years ago, but to save you (and me) from finding it, here's its most recent incarnation:

use framework "AppKit"
------------------------------------------------------------------------
return the infoForUIElementUnderMouse()
------------------------------------------------------------------------
### HANDLERS:
on mouseCoordinates()
		tell the current application to tell mouseLocation()'s {x, y} ¬
				of its NSEvent & NSHeight(its NSScreen's mainScreen's ¬
				frame) to return the {(item 1), (item 3) - (item 2)}
end mouseCoordinates

on infoForUIElementUnderMouse()
		return infoForUIElement at the mouseCoordinates()
end infoForUIElementUnderMouse

on infoForUIElement at __Ref as {list, record, reference}
		local UIElement
		
		tell application id "com.apple.SystemEvents"
				if {__Ref}'s specifiers = {} then
						tell __Ref to if its class ≠ list ¬
								then set __Ref to its {x, y}
						set __Ref to click at __Ref
				end if
				
				set UIElement to __Ref
				if the UIElement = missing value then return {}
				
				script Object
						property parent : UIElement
						property AXAttributes : a reference to (the ¬
								attributes in me whose (name is not ¬
								"AXURL") and (name is not "AXPath"))
						property AXValues : value of AXAttributes
						property AXList : the name of AXAttributes
						property AXRecord : a reference to the the ¬
								{«class usrf»:my AXList}'s contents
				end script
				
				
				set my text item delimiters to linefeed & linefeed
				tell (a reference to the Object's AXList) ¬
						to set the contents to paragraphs ¬
						of (it as text) & ""
				
				tell the Object to repeat with i from 1 ¬
						to the length of its AXValues
						set its AXList's item (i * 2) to ¬
								item i of its AXValues
				end repeat
				
				return {UI element:the Object's contents} ¬
						& the Object's properties ¬
						& (the Object's AXRecord as any) ¬
						& the {_AXActions:the name of ¬
						every action in the Object}
		end tell
end infoForUIElement
-------------------------------------------------------------------❮END❯

The handler you being called for your purposes is infoForUIElementUnderMouse(). That handler first retrieves the mouse pointer's coordinates by calling the handler mouseCoordinates(). This is passed to the handler that does the real work, infoForUIElement, which enumerates and evaluates all properties, attributes, and actions that belong to the UI element identified as occupying the same coordinates directly under the mouse pointer.

A couple of points:

  1. Should you ever try to call infoForUIElement yourself, and wish to pass it a reference to a UI element, this must resolve to a single UI element class object belonging to System Events, and not a collection of objects.

  2. There are two points in the script where it can occasionally hang, neither of which are the fault of the script per se, but rather caused by System Events stalling. This most likely occurs when the element underneath the mouse belongs to an application that hasn't exposed its object hierarchy very well (or at all): therefore, it can happen with Electron-based applications, which will often stall at the start of infoForUIElement when trying to perform a click; and I've noticed it happens with some objects in Monterey's System Preferences app despite passing a perfectly good direct reference to infoForUIElement.

But, on the whole, it works very well for me in Monterey, and I use it regularly.

7 Likes