Hey @peternlewis,
Is it possible for you to provide a method of displaying exactly what buttons Keyboard Maestro can see?
A text-token would be sufficient (%list-available-buttons%), although some sort of name-picker UI would be nice.
-Chris
Ha ha ha, you're funny ;-). Surely @ccstone you know very well that the accessibility of controls is extremely variable from window to window and application to application.
Heck, I've seen windows with infinitely recursive accessibility controls!
So Radio Buttons are generally buttons, and its absolutely worth a try using the Press a Button action, but nothing is a guarantee.
The todo list includes an item related to selecting the button from a popup menu like the Select a Menu Item action.
2 Likes
That would save users a lot of confusion I do believe.
If it's not a priority then please consider the text-token method as an interim solution.
Using that and a Prompt with List action we could whip up a pretty effective macro in no time.
-Chris
I'm pretty sure you could whip up an AppleScript to query the accessibility of a window and list all the radio buttons faster than I could do it in Keyboard Maestro 
Could be.
But I don't know how deep Keyboard Maestro looks.
If you'll give me a clue to that I'll try fooling with it.
Also:
Keyboard Maestro will let you provide a path for menu item discovery.
Is there any way to give a Press Button action a full path to the button?
-Chris
1 Like
I created this macro a while ago to identify UI elements for the purposes of AppleScripting. I position the mouse over the element I'd like to query, hold down ⟨control⟩+⟨option⟩, then tap ⟨command⟩. An AppleScript record is copied to the clipboard containing some pertinent information on the UI element
AppleScript
use application "System Events"
use framework "Foundation"
use scripting additions
property NSEvent : a reference to current application's NSEvent
property NSScreen : a reference to current application's NSScreen
-- Screen dimensions
set display to NSDeviceSize ¬
of deviceDescription() ¬
of item 1 ¬
of NSScreen's screens() as record
-- Mouse position relative to bottom-left of screen
set mouseLocation to NSEvent's mouseLocation as any
-- Adjust mouse position coordinates to be relative to top-left of screen
set mouseLocation's y to (the display's height) - (mouseLocation's y)
set obj to click at the mouseLocation's {x, y}
set R to {UIObject:obj ¬
, UIProperties:obj's properties ¬
, UIChildren:obj's UI elements ¬
, UIAttributes:name of obj's attributes ¬
, UIActions:name of obj's actions}
set the clipboard to ¬
"tell application \"System Events\" to set R to ¬
" & coerceToString(R)
--------------------------------------------------------------------------
to coerceToString(object)
local object
try
set S to object as text
on error E --> "Can’t make %object% into type text."
set the text item delimiters to "Can’t make "
set S to rest of text items of E as text
set the text item delimiters to " into type text."
set S to text items 1 thru -2 of S as text
end try
end coerceToString
Identify UI Element.kmmacros (18.6 KB)
6 Likes
@CJK, this sounds very interesting, but I can't get it to work.
When I try it I get nothing on the clipboard.
When I run it from Script Debugger, I get this error. What am I doing wrong?
Can’t set clipboard to "tell application "System Events" to set R to >¬
{UIObject:«class tbar» 1 of window "Identify UI Element Under Mouse.scpt" of «class pcap» "Script Debugger" of application "System Events", UIProperties:{«class minW»:missing value, «class orie»:missing value, «class posn»:{2048, 44}, class:«class tbar», «class rold»:"toolbar", «class axds»:missing value, «class focu»:false, «class titl»:missing value, size:{1183, 40}, «class valL»:missing value, «class help»:missing value, «class enaB»:missing value, «class maxV»:missing value, «class role»:"AXToolbar", «class ects»:{}, «class sbrl»:missing value, «class selE»:missing value, name:missing value, «class desc»:"toolbar"}, UIChildren:{«class sgrp» 1 of «class tbar» 1 of window "Identify UI Element Under Mouse.scpt" of «class pcap» "Script Debugger" of application "System Events", «class sgrp» 2 of «class tbar» 1 of window "Identify UI Element Under Mouse.scpt" of «class pcap» "Script Debugger" of application "System Events", «class butT» 1 of «class tbar» 1 of window "Identify UI Element Under Mouse.scpt" of «class pcap» "Script Debugger" of application "System Events", «class butT» 2 of «class tbar» 1 of window "Identify UI Element Under Mouse.scpt" of «class pcap» "Script Debugger" of application "System Events", «class butT» 3 of «class tbar» 1 of window "Identify UI Element Under Mouse.scpt" of «class pcap» "Script Debugger" of application "System Events", «class butT» 4 of «class tbar» 1 of window "Identify UI Element Under Mouse.scpt" of «class pcap» "Script Debugger" of application "System Events", «class butT» 5 of «class tbar» 1 of window "Identify UI Element Under Mouse.scpt" of «class pcap» "Script Debugger" of application "System Events", «class sgrp» 3 of «class tbar» 1 of window "Identify UI Element Under Mouse.scpt" of «class pcap» "Script Debugger" of application "System Events", «class sgrp» 4 of «class tbar» 1 of window "Identify UI Element Under Mouse.scpt" of «class pcap» "Script Debugger" of application "System Events"}, UIAttributes:{"AXOverflowButton", "AXParent", "AXChildren", "AXFocused", "AXSize", "AXRole", "AXTopLevelUIElement", "AXSelectedChildren", "AXHelp", "AXChildrenInNavigationOrder", "AXPosition", "AXWindow", "AXRoleDescription", "AXFrame"}, UIActions:{"AXShowMenu"}}".
1 Like
All the way to the bottom (except it gives up for infinite loops).
No, there is no such concept (at least, not yet, I am pondering ideas for the future).
Hey Peter,
Cool.
@CJK's script has given me an idea that I think is going to work for virtually any UI-Element.
I've got to test some more to be reasonably sure though.
-Chris
1 Like
Sorry, my bad. I omitted the line use scripting additions from the start of the script. I've made the correction to the original reply post now, and you can download the macro here:
Identify UI Element.kmmacros (18.6 KB)
Apologies for the error.
Thanks, that fixed the problem.
May I suggest one small change to your macro so that the user will know something happened?
I was never intending on publishing the macro for public consumption, but came across this thread and posted it because I thought it might be useful and relevant to the discussion.
My feeling was that a macro of this nature is a bit niche, appealing really only to scripters rather than the KM community at large.
That said, I think I'm going to implement your suggestion for my personal use, as I usually glare at the menu bar's icon to wait until it stops animating. I think your way is a bit more sane.
Blah blah. Done. Macro links updated.
s/scriptures/scripters/ 
1 Like
@CJK, thanks again for this very useful macro/script.
It worked fine on Sierra, but the script seems to hang now that I've upgraded to Mojave.
Running Keyboard Maestro 8.2.4 on macOS 10.14.5.
I also tried running the script from FastScripts, and it hung there as well.
Running the script in SD7, it seems to hang at this statement:
set obj to click at the mouseLocation's {x, y}
Can you confirm this issue on Mojave, or do you have a fix?
Thanks.
I'm afraid I don't have Mojave (I'm bricking it about Catalina).
Below is the latest version of the macro that I use on High Sierra. I reviewed the code today and one thing I did notice is that the previous version didn't import the AppKit framework, which it now does. This might resolve the hang, and I would advise anyone who uses this macro to update it to use the corrected AppleScript.
Identify UI Element.kmmacros (19.6 KB)
AppleScript: Latest version on GitHub
System info: AppleScript version: 2.7 System version: 10.13.6
The handler that retrieves the mouse position is below. To do a quick operational check without having to download a full macro or visit GitHub for the script, then here it is:
on mouseLocation()
on mouseLocation()
script
use framework "Foundation"
use framework "AppKit"
property this : a reference to current application
property parent : this
property NSEvent : a reference to NSEvent of this
property NSScreen : a reference to NSScreen of this
on mouseLocation()
set [list, [number, height]] to ¬
NSScreen's ¬
mainScreen()'s ¬
frame()
tell NSEvent's mouseLocation() to ¬
{its x, height - (its y)}
end mouseLocation
end script
result's mouseLocation()
end mouseLocation
1 Like
I'm looking to create macros that click specific buttons on the UI without using "Click at Found Image", both for stability and share-ability.
Could this help me do that? I can see the script outputting a variable "Result", but I don't know how to make KM click at the found element.
Thanks in advance! This would be a big deal.
If you have a number of macros that you want to create using the app's UI, then AppleScript will provide you with the most capability. To ID the UI elements for AppleScript, the best tool is UI Browser.
If your needs are simpler, then KM does offer a few tools to work the the UI:
Representative Examples (there are more):
App UI Elements
- Insert_Text_by_Typing
- Manipulate_a_Window
- Move_a_Window
- Move_and_Resize_a_Window
- Move_or_Click_Mouse
- Press_a_Button
- Select_or_Show_a_Menu_Item
- Type_a_Keystroke
Web Browser UI Elements
(Offered for "Front Browser", "Safari", and "Chrome")
- Click_Front_Browser_Link
- Find_Image_on_Screen
- Focus_Front_Browser_Field
- Previous_Front_Browser_Tab
- Select_Front_Browser_Field
- Select_Front_Browser_Tab
- Set_Front_Browser_Checkbox
- Set_Front_Browser_Field_to_Text
- Set_Front_Browser_Radio_Button
System UI Elements
- Set_Keyboard_Layout
1 Like
Thanks @JMichaelTX.
I grabbed UI Browser and it's able to see the window position and dimensions of "floating windows", which is one of the problems I'm trying to solve!
I just need a way to tile all those floating windows, i.e. put them next to each other.
Here is a screenshot from UI Browser of a floating window in the app "Live".
In this case the floating window name is "Legend/7 Bass", but we don't know the names of the floating windows ahead of time, and the floating windows may share the same name, so we'd just need to tile all windows in "Live" with the type or subrole "floating window".
To be clear, the windows shouldn’t be resized… just placed next to each other with their current size.