Understanding AppleScript UI-Scripting to Click Menus

Questions & Suggestions
21

Oh! Then I take that back. It is a native application. It just seems to operate so differently to others that I assumed it had been written using non-Apple frameworks (which is what I meant by non-native), such as ElectronJS or ReactJS, or Java or something that was developing outside the confines of what is provided by the Apple ecosystem.

That error was simply caused by you cancelling the execution (or the script timing out).

I didn't anticipate you'd run the script initially from within KBM. But this was my mistake for assuming your approach would necessarily be the same as mine.

Whenever someone hands you some new code to try, you shouldn't simply plonk it straight into your KBM workflow and cross your fingers. The two main reasons for this are that, while KBM has the means to catch errors from an executing script, it doesn't have the means to introspect the error event and so cant tell you anything meaningful about what went wrong. It also can't return to you every data type that a script could return upon successfully completing execution, and that's pertinent in this case. The second reason is that if you're testing new code, or trying to debug your workflow, you need to do so in an environment where there are the fewest moving parts, i.e. things that are happening other than the execution of the code being tested, which needs to happen in isolation of everything else.

So right now, we have no information about why the script failed. In fact, we have no idea whether the script failed at all, or timed out by itself, or succeeded and returned a type of data that KBM cannot handle.

Therefore, you should launch Script Editor, creating a blank document, and running the script I gave you from there. The Script Editor window has a pane at the bottom that you can pop up, in which the resulting output from a script is displayed.

The obvious reasons for the script to choke is if it references UI elements that don't exist. I just used the same references that you used, which you got from UI browser, but that doesn't mean the references are infallible, and so you need to be able to see what's actually happening when you run the script. The other situation that comes to mind is the unlikely event that the first script doesn't return true, doesn't return false, but returns the entire contents of... that object reference. You said KBM didnt seem to be doing anything except spinning the icon...that's what it would look like if it reaches that point in the script and starts enumerating the descendants of the menu UI element.

1 Like
22

There is a bug with the click command. It’ll work immediately the first time round which is fine if you’re just toggling a button like low latency mode

Ie


Tell application “System Events” to tell process “Logic Pro”
       Set tracks_window to title of first window whose title contains “_Tracks”
       set front most to true
       Tell checkbox “Low Latency Monitoring Mode” of group 1 of window tracks_window
             Click
        End tell
End tell

But if you need to send more events after the click, I would download a command line program called cliclick (command line interface click)

Then, find the ui element, note that the ui element can change depending on the window layout in logic. so if you can find the accessibility description rather than entering button 2 you could say

Here’s an example using cliclick

`Tell application “System Events” to tell process “Logic Pro”
       Set tracks_window to title of first window whose title contains “_Tracks”
       set front most to true 
Set theImage to (The first button of UI element 1 of group 2 of list 1 of window tracks_window whose accessibility description = “xxx”
Set {x, y} to position of theImage
Do shell script “/user/local/Cellar/cliclick/5.x/bin/cliclick dc:” & x & “,” & y `

After that, in the same tell block, you can use keystrokes and key codes to navigate through a pop up menu. With accessibility descriptions you can do a lot with gui scripting in Logic Pro. Here’s some examples from apple scripts I’ve worked into better touch tool

23

Here’s an AppleScript that on run will give you a 5 second head start to position your mouse on an element in the gui then it will list all the available info of the ui element under the cursor.


delay 5
#!/usr/bin/osascript
--------------------------------------------------------------------------------
# pnam: GET MOUSE POSITION | IDENTIFY UI ELEMENT
# nmxt: .applescript
# pDSC: Identifies the UI element reference located under the mouse cursor

# plst: -/Users/dh/AppleScriptive/scripts/Get Mouse Position | Identify UI Element.applescript

# rslt: «text» : A script containing a reference to the UI element's properties
#       -      : Element not accessible
--------------------------------------------------------------------------------
# sown: CK
# ascd: 2018-09-02
# asmo: 2019-06-09
--------------------------------------------------------------------------------
use application "System Events"
use scripting additions
--------------------------------------------------------------------------------
# IMPLEMENTATION:
set s to __string__({UIElement:it ÂŹ
	, UIProperties:properties ÂŹ
	, UIAttributes:name of attributes ÂŹ
	, UIActions:name of actions} of ÂŹ
	(click at my mouseLocation()))

-- Text manipulation -- 
set [a, b, c] to [length of "{UIElement:", ÂŹ
	offset of "of application \"System Events\", " in s, ÂŹ
	length of "of application \"System Events\", "]
set UIElement to text (1 + a) thru (b - 2) of s
set UIRecord to ["{", text (b + c) thru -1 of s]

-- Composite string --
set UIString to the contents of [ÂŹ
	"use application \"System Events\"", ÂŹ
	linefeed, linefeed, UIElement, ÂŹ
	linefeed, linefeed, UIRecord] as text


set the clipboard to the UIString
return the UIString
--------------------------------------------------------------------------------
# HANDLERS:
# __string__()
#   Returns a string representation of an AppleScript object
to __string__(object)
	local object
	
	try
		set s to object as missing value
	on error E --> "Can’t make %object% into type missing value."
		set tid to the text item delimiters
		set the text item delimiters to "Can’t make "
		set s to text items 2 thru -1 of E as text
		
		set the text item delimiters to " into type missing value."
		set s to text items 1 thru -2 of s as text
		
		set the text item delimiters to tid
	end try
	
	return s
end __string__

# mouseLocation()
#   Gets the current cursor position relative to the top-left corner of the
#   screen
on mouseLocation()
	script
		use framework "Foundation"
		use framework "AppKit"
		
		property this : a reference to current application
		property parent : this
		
		property NSEvent : a reference to NSEvent of this
		property NSScreen : a reference to NSScreen of this
		
		on mouseLocation()
			set [list, [number, height]] to ÂŹ
				NSScreen's ÂŹ
				mainScreen()'s ÂŹ
				frame()
			
			tell NSEvent's mouseLocation() to ÂŹ
				{its x, height - (its y)}
		end mouseLocation
	end script
	
	result's mouseLocation()
end mouseLocation
24

In answer to your question the following script would select humanize


Tell application “System Events” to tell process “Logic Pro”
       Set tracks_window to title of first window whose title contains “_Tracks”
       set front most to true 
Set theImage to («class menB» "Functions" of «class sgrp» 1 of «class sgrp» 1 of «class sgrp» 3 of window tracks_winndow
Set {x, y} to position of theImage
Do shell script “/user/local/Cellar/cliclick/5.x/bin/cliclick dc:” & x & “,” & y
Keystroke “mi”
Key code 125
Key code 124
Keystroke “hu”
Key code 34
End tell

you might have to add some values to the x, y data as cliclick doesn’t always move the mouse to the expected position. Eg:


Do shell script “/usr/local/cellar/cliclick/5.x/bin/cliclick dc:” & x + 10 & “,” & y + 7
25

It's a timing issue rather than a bug. You need to wait for a popup menu to actually appear before you can act upon it. So if you want all the keystrokes in one script (i.e. without using native KM actions), you can do this (no cliclick required):

Script 
try
	with timeout of 0.3 seconds
		activate application "Logic Pro X"
		tell application "System Events"
			tell process "Logic Pro"
				
				set tracks_window to title of first window whose title contains "- Tracks"
				
				click menu button 2 of group 1 of group 1 of group 4 of window tracks_window
				
				delay 0.1

keystroke "MIDI"
key code 124
keystroke "Humanize"
key code 36

		end tell
		end tell
	end timeout
end try
26

I’m not sure what you mean by timing issue. But I’m pretty sure it’s a bug that’s been around for years. If you tell a pop up menu to click, it will click and reveal the pop up menu pretty much instantly but it won’t carry out any more events until about 5 seconds after click command. This is an issue with the click command. Look it up online. Most posts discussing the topic recommend using cliclick as a work around.

27

I understood this to mean that you were unable to interact beyond the first click. The script I posted above will fail if you don't include the 0.1s delay as it affords the popup time to appear. Once the delay is in there, subsequent interaction is successful.

Have I misunderstood?

28

I didn’t say you can’t interact with the interface after using a click command. I said any following events would be delayed by several seconds even if you use ignore application response

29

Sorry, Include delay after what part of what script? I run some pretty complex and long scripts interacting with pop up menus in logic successfully without any delay. But if your script needs it then fair enough. But it doesn’t change the fact there’s a bug with the delay command. It applies to click and AXPress. Take my word for it if you want to write UI scripting routines using AppleScript that require a click that’s more than toggling a button then use cliclick as a do shell script.

30

Not in this post, hence my misunderstanding. All good though.

CleanShot 2023-03-19 at 23.16.10@2x
CleanShot 2023-03-19 at 23.16.10@2x820×772 53.7 KB

(In this post)

I'm sure you're right. However I've personally never run into any such problem with Logic, so I'm interested to know the circumstances under which that bug would present itself.

Could you give a real-world example of something that will fail without cliclick?

31

If click command worked correctly I doubt I’d need the delay before keystrokes. In any event it’s neither here nor there because the click command in any instance will leave a several second delay before it will process any future actions. So in response to your question in what instance would the issue occur the answer is every instance click is used. So unless you’re just toggling a button if you don’t want to wait 5 seconds before your next action, then use something like cliclick. Gui automation is meant to spread processes up not slow them down. By the time you’ve waited for the script to start responding again you may as well have just performed the task manually. Theres also an error in your script to select the correct sub menu you would need to add key code 125 after keystroke midi so it selects midi transform rather than midi region parameters. (Also the less text you type in keystrokes the better) in this instance keystroke “M” followed by key code 125, 124 keystroke “hu” key code 34. Would suffice “m” rather than midi and “hu” instead of humanise As if the script failed after the first character of your keystroke text the ‘I” would open the track info d would open event list and I again would open or close the track ui element. You can test what the least amount of characters needed in keystrokes is by clicking on the menu and typing the subfolder name letter by letter until it selects the correct subfolder.

32

If you're into UI scripting, don't forget that UIElementInspector still works with the current OS. Not as user friendly as the sadly now defunct UI Browser, but easier for most than manually digging through menu items.

33

Sorry, I'm not sure I understand. There's little or no delay between simulated clicks in this, for example:

tell application "Safari"
	activate
end tell

tell application "System Events"
	tell process "Safari"
		click menu "File" of menu bar 1
		click menu "Edit" of menu bar 1
	end tell
end tell

Or are you talking specifically about Logic? In which case the bug isn't in the click command but in the way Logic handles it.

34

That must be quite frustrating but it certainly isn't an issue for everyone, and I suspect, as @Nige_S points out, that it isn't, in essence, an AS 'bug', as such.

Actually it's not an error; it's using the Functions menu in the Piano Roll as an example to demonstrate the use of a short delay. For the other menu it would be:

Script 
try
	with timeout of 0.3 seconds
		activate application "Logic Pro X"
		tell application "System Events"
			tell process "Logic Pro"
				
				set tracks_window to title of first window whose title contains "- Tracks"
				
				click menu button 2 of group 1 of group 1 of group 3 of window tracks_window
				delay 0.1
				
				keystroke "MIDI Transform"
				key code 124
				keystroke "Humanize"
				key code 36
				
			end tell
		end tell
	end timeout
end try

If you run that on your setup, do you still get a 5s delay?

Agreed, but in my experience, the difference when dealing with short strings is negligible, and my script was an example, so I wanted it to be plainly readable. It runs extremely fast, but sure, you could probably shave off a few nanoseconds by typing "hu" instead of humanize.

35

you’re absolutely right Nige, thanks for that information and the correction, it was a fairly widely documented general issue in the past. When I wrote the bulk of my UI scripts. There were lots of posts on forums acknowledging the issue and it was considered a bug, that’s obviously since been ironed out. I’ve just used click repeatedly in several different apps without any delay issues. Ironically the issue still persists in logic. So the best way to mouse click in a logic automation script would still be to use cliclick. Re logics ability to handle the command; I wonder if it has something to do with the absence of an AppleScript dictionary in logic.

36

No, because I don't have that issue in Logic.

Could you please post a script (or macro) that you're having issues with so we can see if there's something else at play?

37

Scripts do fail and the less chance of numbers being entered outside of a script resulting in window rearranging r record button being pressed things being solo’d etc it’s best to use as little text as possible.

38

I’m not having any issues. Because I don’t use click command in logic. I sell complex pro app automation scripts and am not in need of scripting advice. Thanks though. You’re being a bit pernickety Neil :slight_smile: I know because I just tried the click command in logic in a script debugger script. Ui scripts do fail maybe your scripts are just very basic. Not necessarily a bad thing.

39

I can't remember the last time a UI script failed in Logic without simply aborting... but as I already said, I agree. Anyway, enough nitpicking...

Well what do you mean by this then?:point_down:t3:

How would you know if you don't have any scripts with that issue?

40

I'm not offering advice. I'm trying to clarify whether there is indeed an AppleScript 'bug'. If there were one, I'd like to know how to recognise it so that I'm able to work around it myself.

Please humour me and try it using KM's Execute an AppleScript action instead. Script Debugger is great but I find that it tends to interrupt UI scripting in Logic by stealing focus before the script has completed during testing:

Humanize.kmmacros (20 KB)

Macro screenshot

Keyboard Maestro Export
Keyboard Maestro Export1410×1318 123 KB

Or maybe I just try to write them very carefully...?