How to choose this UI element?

Questions & Suggestions
1

In Digital Performer, I need to be able to click this pulldown menu, and then choose a specific item from it.

Is there a better way to do this, vs. simply automating a mouse action to a specific x/y coordinate?

(I know I could also use typing to type the characters of the one I want, however, I’m concerned that could be error-prone when this macro runs thousands of times, if the timing of the typing isn’t right.)

Very short YouTube demo that describes it specifically:

2

So it’s obviously not a menu-item that can be seen by KM? Apologies for the silly question – but I often find myself missing the obvious solution.

If not – this would involve some UI scripting, which is way above my head. No doubt someone would be able to provide actual assistance in this regard.

Personally, I usually get around this problem by relying on a little utility called ShortcatApp. It’s very useful for interacting with KM in those cases where I need to trigger a UI element, since it almost always picks it up – but it’s quite bad in that I’ve put off learning scripting, since I’m almost always able to use it as a workaround!

Probably doesn’t help your specific situation, but if this is something prone to happen frequently (needing to interact with UI elements), it might be worth spending the $18. I’ve constructed some very nice workflows around it, that would otherwise not have been possible in my world. Figured I’d mention it.

3

Thanks for the suggestion about ShortCat.
It looks very interesting.
So, do you find that it works well with KM7?

4

I find it to work quite well. I also use it with ActionsApp for iPad (soon to be Quadro) – and it always performs consistently.

Essentially, the steps would be as follows:
1.) I invoke Shortcat app. I type “.” – which forces Shortcat to highlight all the possible UI elements it can interact with on the screen. It then overlays a short letter-combo over each UI element, such as AA, AB, AC etc. – depending on how many there are. [The alternative procedure would be to invoke Shortcat, and start typing the name of the button – i.e. like “next” if there is one down below, on a forum page – the actual button will then be highlighted].

2.) Once I “see” that the correct element is selected (it gets highlighted in blue) – you simply hit enter. If you start with “.” instead, which throws up the overlay, and several UI elements are selected, you simply use CTRL+AB (or whatever the relevant overlay letter combo is) to select the correct UI element, and then hit Enter – whatever is quicker.

3.) Having noted what letter-combo activates the relevant UI element – or, what combo of letters I can type (e.g. “next”) to have that UI element selected, I then simply get KM to do this for me.

4.) So KM invokes Shortcat by calling up its search–bar with its keyboard shortcut (like one could use KM to invoke Alfred), then [types] “next” (or “.” and CTRL+AE) to have Shortcat select the correct UI element, and then hits ENTER to “press” it.

Basically, KM simply does what I would have done manually – I simply do it the first time, to check what I would need to do in that particular programme or whatever.

By way of example, in one of my frequently used programmes, the various search functions had keyboard combos for advanced search, or text search, or group search etc. But there was no keyboard combo for getting the cursor into the “quick-search” field of the main window. I use the latter far more frequently than the others.

So I’m inside Scrivener or Devonthink – I select text, invoke KM – which copies the selection to my clipboard, jumps over to the other App, invokes Shortcat to run the UI search, selects the Quick-search window UI element, hits enter – which places the cursor into the Quick-search field, then pastes the clipboard, and hits enter again to invoke the search.

Without Shortcat in the middle, I couldn’t get “into” the Quicksearch field, in the absence of some UI script. And all the above happens in split seconds, to the point of not being noticeable…

Apologies for the lengthy off-track explanation, but hopefully that explains things properly…

4 Likes
5

Thanks. That was very helpful.

6

ShortCat, which I’d never heard of, looks to be worth trying (the trial is free).

Note that you pose two questions:

  • how to click the disclosure triangle to reveal the drop-down menu (this is the question in the video), and
  • how to select an item in the drop-down menu (this is the question you expand on in your post).

You might have KM perform each click-action differently.

In KM, I have had repeatable success clicking UI elements with the “Found Image” feature. It will take several tries to set the degree of fuzziness (and, I assume, that is screen-dependent, which would make the macro not portable without re-setting the degree of fuzziness). I use Skitch to make a small (but as big as can be used) screen-capture, and then drag the created JPG and drop in in the KM found image box.

There are features in the KM Action that make this easy to test. One, it’s live (if you are one a single-display system, move the KM window so that the sample image is not shown, but the desired found image is shown in the program in which you want to find it). Two, you can click “Display” to have found elements highlighted. (I don’t know what the displayed percentages mean.)

Of course, if the disclosure triangle is always in the same location, you could just use coordinates. Note the options for the reference point and screen.

Selecting from a drop-down menu can be done, as you point out, with either a mouse click at specific screen coordinates (not usually reliable enough) or by having KM insert by typing enough text to select the menu item, and then executing the {return} keystroke. IME, once I have put enough pauses in, and made them long enough (varies by program and site), this works reliably (I’ve used it thousands of time with several different programs). You could also use {up-arrow} and {down-arrow} to move the selection cursor to the desired menu item (of course, will not work well if the drop-down menu is dynamic).

TL;DR: Clicking the disclosure triangle by coordinate, selecting the drop-down menu item by inserting text by typing, and executing the selected item with {return} works reliably on my system once I have included pauses and set their duration.

Here, simply, is something I use all the time (bringing up the context-sensitive menu):
Right Click
At (0,0) from the current mouse location.
Stop macro and notify on failure.
Pause for .4 Seconds
Notify on failure.
Insert Text by Typing
QED
Type the Return Keystroke

1 Like
7

Wow. Thanks everyone for the help, and in particular, Kirby for that very comprehensive round-up and advice.

I just tested clicking relative to a found image, and it works perfectly. (This is crucial to me because the location of that drop down menu does change depending on various factors.)

Thanks!