Problems with new Press a Button action functionality in KM11

[karlfranz]

In playing around with this action to control the Unify music application, I came across a couple of issues:

1. The button press is sometimes too quick for the application to respond. For example, I created a macro that presses the Init button in the application when I press a hotkey trigger. The Init button should call the "Untitled" preset when pressed. When the macro is triggered, I see the Init button flash briefly but the untitled preset is not selected. Pressing the button manually with the mouse works as expected.
   .
   note: I have witnessed this in the past in other scenarios such as when performing a mouse click action to recall the next or previous preset on a plugin. The way I got around this back then was by splitting the mouse click action into individual mouse down and up actions and inserting a brief pause in between the two
   .

2. On buttons that call up a popup menu, the menu appears wherever the mouse pointer was located on the screen rather than next to the button itself.

Tested this on my Mac Studio M1 Ultra running Ventura 13.6

[noisneil]

I've noticed that if I scan Logic Pro for buttons, it sees a lot of them, which is encouraging, but very few are clickable. If I attempt to click the Stereo Output button, I get this area:

Press Button "Description:Stereo Output" failed because the button is disabled.
Macro "Press a Button" cancelled (while executing
Show Menu of Button "Description:Stereo Output"

It would also be pretty helpful if the scanned list was sorted alphabetically.

[karlfranz]

@noisneil

It would also be pretty helpful if the scanned list was sorted alphabetically.

If you hold down the Option key before displaying the list, it will be sorted alphabetically.

[noisneil]

Oof! Good knowledge!

Be nice to set that as default, but no biggie.

[Alexander]

I was about to create a new topic on this subject matter when I stumbled across this one, and decided to post my observations and questions as a comment here instead. But do tell me if this brings only clutter and no contribution to this topic, and would be best of posted in a topic of it’s own. (And anyways sorry for my long-winded ellaborations here.)

Because I have also been trying this awesome new KM11 feature to scan a window for buttons to press, and I have quickly realise how powerful and (by me) underutilised this Press a Button-action really is. But I’ve sadly also found some weird button pressing behaviour within Davinci Resolve, the app I’ve been directing most of my macrofying efforts towards.

Doing a scan of Davinci Resolve makes me realise how incredibly close the Press a Button-action is to giving me control over allot of this apps UI (without relying on Found Images, something many of my Resolve-macros now does) — All kinds of buttons not available as Menu Items, or being assignable for keyboard shortcuts, show up in KMs list of available buttons: Group Pre- and Post clip; selection of the different tools menus; switching from Keyframes to Scopes view, etc.

Only problem: Many of the ‘virtual’ button presses through the Press a Button-action are only being registered as weird halfway presses within Resolve. As shown in the GIFs bellow.

Trying to select different tools menus:

Trying to select Group Pre- and Post-Clip:

In both GIFs I try to demonstrate how a “real” button press, clicking the buttons with the mouse, gives desired result; but ‘pressing’ these same buttons through KM, using the Press a Button-action, only depresses the button, leaving no result apart from this. In the first GIF I also show how the previously selected tools menu button (Primary Color Wheels) sometimes even stays depressed at the same time. This is different than the very quick blinking behaviour described from Unify, but still in the same ballpark, I’d say.

What could be causing this weird half-pressed behaviour? I have the feeling that the answer to this question in this case is Blackmagic Design, and that the problem lies within Resolve. But it would be interesting to hear what makes a Press a Button button press different from an actual button press/click? And of course if anyone have any idea how to counteract this half-press behaviour?

I’ve checked numerous other apps to see if this behaviour shows up anywhere else, but so far I have only encountered it within Davinci Resolve here.

And lastly: I actually never tried the Press a Button-action in Resolve while on KM10, and have no reason to belive this is a bug having first showed up with KM11, but if anyone on KM10 could try Resolve or Unify to see if it is the same behaviour there or not, this would be interesting information. (And I am still on Ventura 13.2, by the way.)

[notsteve]

The button press is sometimes too quick for the application to respond.

This may not be helpful, but it sounds like a problem with the application. The app may need to prepare data for when the button is pressed—but only prepares this data after the button is pressed. Apps should never do that—they should always be one set ahead (if the data is static, which it sounds like it is here). Even if using JS and WebKit, the developer can do stuff like create invisible DIVs in a low-priority thread, and just have them ready to go if the user invokes them.

[karlfranz]

Perhaps but, like you said, not really helpful because we have no control over that.

I have noticed that his problem seems more prevalent in applications that were written using toolkits that allow for multi-platform (Mac/Win) development.

One solution, however, might be for @peternlewis to provide an option in the Press a Button action, that allows for a user-defined delay between the mouse click and release.

[peternlewis]

@karlfranz

The button press is sometimes too quick for the application to respond. For example, I created a macro that presses the Init button in the application when I press a hotkey trigger. The Init button should call the "Untitled" preset when pressed. When the macro is triggered, I see the Init button flash briefly but the untitled preset is not selected. Pressing the button manually with the mouse works as expected.

The accessibility API for pressing is a single call. There is no “press” and then “release”. So there is no way to control this. So the application needs to be fixed, or you need to use simulated clicks (which you can control the speed of).

It is concerning, since the accessibility API is also used by those folks who have accessibility needs, and poor support for them means poor support for people who really need it, not just minor challenges for Keyboard Maestro users.

@karlfranz

1. On buttons that call up a popup menu, the menu appears wherever the mouse pointer was located on the screen rather than next to the button itself.

Again, showing the menu is a single Accessibility API call. Keyboard Maestro has no control over what happens after that.

[peternlewis]

@noisneil

Be nice to set that as default, but no biggie.

Sorting destroys information (in this case, typically the rough position in the window), which is why it is not the default.

[peternlewis]

@Alexander

But it would be interesting to hear what makes a Press a Button button press different from an actual button press/click?

The action is an accessibility perform action, literally the API call:

AXUIElementPerformAction( element, action )

where element is a reference to the button, and action is a string (for Press button, it is “AXPress”).

[peternlewis]

@karlfranz

One solution, however, might be for @peternlewis to provide an option in the Press a Button action, that allows for a user-defined delay between the mouse click and release.

No such option exists because there is no “click” for the Press Button action - the mouse is not involved at all.

[Alexander]

@peternlewis There couldn’t by any chance be any possibilities for extracting the location on the screen of these UI elements/buttons? Something that could be implemented in a future version of KM as something like a Function that returns the x, y coordinate of the button? Something that could then be used for move and click actions and such, and in certain instances be a very robust alternative to click found image

[noisneil]

Give this a try:

Create Tell Block for UI Interaction - "UI Browser Lite"

This is a work-in-progress attempt to replicate the functionality of UI Browser that I find most useful. I started work on this last night, and am quite excited about it. It builds on a clever script by @CJK, which returns information pertaining to the UI element under the mouse pointer, and uses this data to automatically create a functioning AppleScript tell block. Usage: Hover your mouse pointer over the element you'd like to interact with and trigger the macro, which comes with its trigge…

[Alexander]

Wow, this is a really cool and useful macro!
I've tried it with different apps here where it works wonderfully, but sadly Davinci Resolves UI elements does not play well with this tool either.

Interestingly in both attempts, demonstrated bellow, it leaves two buttons depressed, as also happens in my first gif above. But this half-pressing of the second button occurs already after running 'Generate UI Element' within Resolve. Running the Apple script with Click functionality only seems to activate Resolve and nothing more. With the Coordinates funtion it weirdly does not even return a window. I also tried this 'Generate UI Element' macro on buttons that KMs Press a button-Action works with, but these also does not return any result with the 'Generate UI Element' tool (only activating Resolve, not clicking, not returning a window.)

Trying the 'Generate UI Element' Click function on Resolves Curves tools menu

This is the script it returns (for it not to error I had to manually fill in "Davinci" before "Resolve")

activate application "Resolve"
tell application "System Events"
	tell process "Resolve"
		click checkbox 15 of group 1 of window "DEMONSTRATION"
	end tell
end tell

Trying the 'Generate UI Element' Coordinates function on Resolves Curves tools menu

This is the script it returns (for it not to error I had to manually fill in "Davinci" before "Resolve")

activate application "Resolve"
tell application "System Events"
tell process "Resolve"
try
	tell checkbox 15 of group 1 of window "DEMONSTRATION"
		set {xPosition, yPosition} to position
		set {xSize, ySize} to size
		set midX to xPosition + (xSize div 2)
		set midY to yPosition + (ySize div 2)
	end tell
	return midX as string & "," & midY as string
end try
end tell
end tell

Is there anything to learn from how your 'Generate UI Element' macro Interacts with this Resolve UI button?

[noisneil]

I've fixed the differing app/process name problem with the Tell Block macro, in case you'd like to re-download it. Thanks for bringing that to my attention!

In your gif, you changed the application name (which was the right thing to do), but you also changed the process name, which was unnecessary.

Using the new version of the macro, I generated this, which works for me. Of course it may not work for you right away if our Resolve Workspaces differ.

Click Button in Resolve.kmmacros (21 KB)

Macro screenshot

I would add that, for this kind of thing, I might still default to using found images for one reason: it's much snappier than AppleScript; almost instantaneous by comparison. Ideally, if you can use a menu item or set a shortcut for a particular function, those are far preferable to either, as I'm sure you're aware.

[Alexander]

Okay, this is amazing!
It is so fascinating to be able to resize and move the window around and this tool still always just knows where to move the pointer. It just works!

So far it seems very robust, and in my standard of snappy, this is well within spec for allot of use cases. I will continue to test and compare it to found image clicking, but I have a feeling that I might choose this tool over found images in many cases where there are no menu items or shortcuts available, and where the KM Press a button results in the described half-pressing behaviour.

Fantastic that you've fixed the issue differing app/process name! One further issue I met with the tool was that the window name in Resolve takes the name from the project name, so it would initially only work within the project the UI coordinates was set from. But I managed to fumble myself forward to this slightly modified version of the script your tool outputs, that works for all projects:

Select Curves Tools Menu in Resolve.kmmacros (4.4 KB)

Macro image

Modified Get Coordinates Applescript

activate application "DaVinci Resolve"
tell application "Finder"
	set frontWindowName to (name of window of process "Resolve") as string
end tell
tell application "System Events"
	tell process "Resolve"
		try
			tell checkbox 15 of group 1 of window frontWindowName
				set {xPosition, yPosition} to position
				set {xSize, ySize} to size
				set midX to xPosition + (xSize div 2)
				set midY to yPosition + (ySize div 2)
			end tell
			return (midX as string) & "," & midY as string
		end try
	end tell
end tell

————————————————
Do tell if my modifications could be written better as I am a total novise with Applescript (and all other scripting really)
————————————————

---

Another potential challenge within Resolve is that the UI element (in my examples checkbox 15 of group 1) refers to another element on other pages than the Color page. This can however for instance be resolved by adding a check to the macro to see if the Color page is the current page (at the expense of some snappyness, of course (had I been better in Applescript I guess would have tried to combine the two Applescripts into one, to see if this would be more snappy than runnig the two scripts separately)):

Select Curves Tools Menu in Resolve w-PageCheck.kmmacros (6.4 KB)

Macro image

---

Lastly I want to direct an apology to @karlfranz for kind of hijacking your thread here. But I would argue that it is still kind of on topic, and I would wholeheartedly recommend you trying @noisneil amazing tool to see what it can do for you within Unify.

---

EDIT: I took a chance and seeme to have been able to combine the two separate Applescripts into one, and the macro as a whole is indeed runing slightly faster:

Select Curves Tools Menu in Resolve w-PageCheck v2.kmmacros (5.8 KB)

Macro image

Applescript that returns value of Color page button in combination with coordinates of Curves Tools Menu

activate application "DaVinci Resolve"
tell application "Finder"
	set frontWindowName to (name of window of process "Resolve") as string
end tell
activate application "DaVinci Resolve"
tell application "System Events"
	tell process "Resolve"
		set buttonvalue to value of checkbox 5 of window frontWindowName
		try
			tell checkbox 15 of group 1 of window frontWindowName
				set {xPosition, yPosition} to position
				set {xSize, ySize} to size
				set midX to xPosition + (xSize div 2)
				set midY to yPosition + (ySize div 2)
			end tell
			return (midX as string) & "," & midY & "," & buttonvalue as string
		end try
	end tell
end tell

————————————————
Again, do tell if my modifications could be written better
————————————————