Mouse Click-Drag Not Working on Certain Controls in Certain Applications (Cubase)

[fossie]

I've been using the Move or Click Mouse action with dragging enabled to simulate click-and-drag gestures on UI controls. This works perfectly in standard macOS apps like Freeform, and I've also confirmed it works within Cubase itself — I can successfully click and drag regions in the Project window without any issues.

However, when trying to drag certain controls in Cubase — specifically a slider in the Pattern Editor — the macro moves the mouse to the correct position but the parameter doesn't respond. The mouse moves as expected but the slider value doesn't change, whereas manually clicking and dragging the same control with a trackpad works perfectly.

I've confirmed the image recognition is landing on the correct target, so this isn't a positioning issue. Has anyone run into something similar with certain controls not responding to simulated drags, and if so how did you resolve it?

[Alexander]

UI interaction can be quirky at times, and for certain apps it is required to separate the action of clicking from the action of dragging, and add a small break in-between.

Does a set-up, split up like I've done below, work for clicking and dragging of your Cubase slider?

You would of course need to change the values in the green action for this to work:
The first green action should be the coordinates for your slider's drag handle (alternatively the finding of the image, as I understand you plan to use in your full macro). The second green action should be the coordinate of where you want to move the drag handle to.

If this works you can do further tests while reducing the pause lengths, now currently set to 1 second for initial test. Once you've identified where in the chain the problem lies, you can probably get away with only very short pauses where the shoe pinches.

Click and drag test.kmmacros (20 KB)

[fossie]

Thanks @Alexander but I think I’ve already tried all of this (separating the move/click, separating the click/hold/move, adding pauses) and none of it seemed to work . I didn’t go as far as 1second but I think .5secs and yeah, I still couldn’t get it to work.

[Alexander]

Do not know if it will make a difference, but you could try doing the clicking and dragging via the command-line tool cliclick that can be installed via Homebrew:

brew install cliclick

If it works, it could be set up something like how I've done it below, where you first position the mouse using Move mouse to found image, and then do the clicking and dragging using cliclick.

Move mouse to found image- then click and drag using cliclick.kmmacros (21 KB)

Macro image

[kevinb]

I don’t use Cubase but I wonder whether in this case the slider can be adjusted using mouse clicks. For example, would clicking at A slightly decrease the value, and would clicking at B greatly increase the value?

Also, would double-clicking on “0.0” allow a value to be pasted in?

[fossie]

@Alexander I install homebrew and downloaded it github file for clickclick but man I have no idea how to implement this and I'm not sure if it's gonna be worth an effort of learning how to be honest. I appreciate the thought though. Of course if you're willing to kind of explain how to do this I'd be more than willing to give it a go but I'm sure you have better things to do with your time.

@kevinb yeah this is a thought but the starting point won't always be in the center, it may be offset and if it is offset to one side or the other the image is unreliable. Plus I'm not sure how accurate mouse clicks would be for making this kind of adjustment.

One thing I am working with though is BetterTouchTool. BTT actually does work for the click and drag but setting up loops and such that cancel (on say key up or different key command) is a bit tricky

[Alexander]

@fossie

Of course if you're willing to kind of explain how to do this I'd be more than willing to give it a go but I'm sure you have better things to do with your time.

Just explain where in the implementation you're getting stuck, and I'll be more than happy to try to help. And since the question is about implementation, it would help a lot if you could also upload the macro you are working on, or a simplified version of it

---

@fossie

@kevinb yeah this is a thought but the starting point won't always be in the center, it may be offset and if it is offset to one side or the other the image is unreliable. Plus I'm not sure how accurate mouse clicks would be for making this kind of adjustment.

If you've been able to successfully identify the slider drag handle position for your original click at the found image then drag attempt, you could just horizontally offset from that position. Here's an example that would click 20 pixels to the left of the found image:

[fossie]

@Alexander Wow thanks. I’m just here really. This gets my mouse to location I need to be at in order to start the drag. I don't know of any other way to click-drag an item other than via the click action and setting the option at the bottom to “and drag to”

I've got homebrew installed now and have installed clickclick as well if that helps anything. Thanks again

µS + Swing Hold.kmmacros (39.0 KB)

[fossie]

As far as implementing clickclick, I have no clue as to where to start really. Are the coordinates something that I need to enter into Terminal and if so how do I get KM to communicate with clickclick via Terminal? Sorry for the noob questions here

[Alexander]

The way I read your macro, you are having a slider with an icon to the right of it bearing this image?

And that you've identified the position of the slider drag handle to be 92 pixels to the left of it? I guess this would then be the slider's default position?

The problem with this, if I read it correctly, is that you with the current setup would not know the position of the drag handle if the slider is moved away from its default position. And as I've read you earlier as wanting also to be able to use this macro when sliders have already been moved, it needs to be done slightly more sophisticatedly.

But before we go further into that, can you first run the following test for me and report back to me the results you are getting? (Crossing my fingers that the test will run as intended in your system, as it got a little bit more complex than I had planned.)

µS + Swing Hold — SLIDER MOVE TEST.kmmacros (50 KB)

Macro Image

---

For reference: Your Macro Image

[kevinb]

@fossie

Plus I'm not sure how accurate mouse clicks would be for making this kind of adjustment.

That is of course true, but then again, dragging a slider value isn’t as accurate as, say, entering a numerical value into a field or using keyboard shortcuts to decrement/increment–but doing things in those ways is only of practical use in certain situations. I really don’t know, but I wonder if more ideas from your fellow Cubase users might perhaps be stimulated if they knew exactly which slider you want to move, and in which scenarios.

[fossie]

Hmmm… @kevinb got me thinking about how I could type in the numerical values instead and I rigged this thing up. It’s kinda slow, but tbh that’s probably at about the speed I need it at. I might just run with this for the time being but if I windup getting annoyed by the clunkiness I might come back and revisit this topic later.

BTW, I failed to mention this but I'm using this via Metagrid Pro which allows for momentary latch buttons, so two macros are required. One main macro for the HOLD portion and one simple one for the RELEASE (just to switch the variable back to the starting point).

µS + Swing Hold test.kmmacros (43.0 KB)

µS + Swing Release test.kmmacros (2.7 KB)

Do appreciate the help and if you have anymore suggestions on how to clean/speed it up lmk

[fossie]

But as Steinberg (Cubase) is extremely inconsistent in its UI and under the hood build there are knobs that have value inputs but some that do not. For those that don't I might need to come back to this thread and think about how to perform a drag gesture, if I can be bothered…thanks again @Alexander @kevinb

[fossie]

and @Alexander sorry that I forgot to mention that, yes it's a slider but, it has a numerical value beside it that can be dragged up or down (despite the fact that it's a horizontal slider) and that's primarily what I was thinking of using as the drag starting point so it would've worked regardless of where the point was on the slider itself. If I started dragging from the numerical value it would always start at the current numerical value. In the macro I’ve provided I’m adjusting the SWING (top) slider

[Alexander]

As you've gotten @kevinb's suggestion, of numerically inputting an increased value to work within your requirements — I believe it makes sense to go for this solution, as it will probably be more reliable than click-dragging sliders or knobs (all though that stationary knob would make it a lot easier to achieve than with a shifting drag handle).

I might be misunderstanding some parts of your macro, and the type of interaction Cubase needs, but below are some general notes on how I believe you might be able to give your macro a slight speed up:

---

To speed up the loop:

• I do not believe you need to continuously copy, and set the value of IncreaseStepSwingValue to %SystemClipboard% while looping. As the value is continuously stored in the variable there's no need to re-check the value set in the Cubase environment for each loop.
• Also, that triple-click might be slowing things down somewhat. Maybe you can get away with adding a return/linefeed at the end of the text in the Insert text by typing, and then following that up with a single click, for re-selecting it instead. (By the way, some apps allow for using arrow keys up and down when an input field like this is used, something I would also test to see if Cubase is supporting here.)

Something like this

---

For general snappiness before reaching the loop:

• I would also suggest not searching for that image twice; you already found it in that If, and can extract the value from that find using the %FoundImage%-token. To calculate that found image's center, the easiest way is using calculation dot-notation .MidX and .MidY.
• Also, I am not sure you'd need to split the Move and Click actions here.

My suggestion would be something like this

---

• Generally, I would also suggest swapping out all global variables (not needed for persistent storage or communication between macros) with local variables, but I do not believe it would do much for speed in this case.

---

For reference: Your macro image

[fossie]

Awesome way snappier! Thanks! And thanks for the tip about local_vars. Will definitely start implementing that as well as learning a bit more in general about how variables work.

µS + Swing Hold.kmmacros (26.1 KB)

[Alexander]

Great to hear that you got it operating snappier!

One last tip to speed up the finding of images – if your layout and screen arrangement is somewhat consistent – is to shrink the area the image is searched for. Especially for high-resolution monitors, this can offer quite a substantial speed-up.

Defining the area is most easily done by pressing the 'Get' button and drawing a rectangle around your area of interest. If your secondary display is arranged to the left of your main display, you might get negative values; if to the right, a value of more than your main display width resolution; etc.

If your layout is not that consistent, even drawing a very generous rectangle over half or quarter of your screen will help.

---

The pixel values of the area parameters are by default absolute values, relative to your main display's top-left corner. But the values can be made relative to, for example, the focused window, by using the WINDOW()-function. This way, you could move the window around, and even change display arrangement, so long as the layout within Cubase does not change relative to your selected window edge/corner.

As an example, relative to the top right corner of the focused window, could look something like this:

This will define a rectangle of 300x150, hugging the right edge, 500 pixels down from the window top edge.

---

While setting up more convoluted areas like this, a "pro-tip" can be to use the Highlight Location action as a tool to check where your defined area is pointing to. Set it to rectangle mode, and the same x, y, width, and height (→, ↓, ↔, ↕) values as your area. Placed in a loop, you can even see how it follows the focused window while moving it around!

---

(P.S.: Defining a tight area relative to the window frame like this can take you dangerously close to simply defining the exact coordinate you want to click — something that could take finding of the image out of the equation entirely!)

[fossie]

@Alexander Yup. I do implement the area from time to time but might try doing it more often here.

But damn, I tried basically copying and pasting the working macro from yesterday to create another macro for an element (Pattern Randomization) that reacts in a similar way where the field input able to be clicked into and set manually, but for some reason Metagrid is not triggering the release macro nearly ever. Only when I tap briefly on the Metagrid button is the release trigger ever executed but if I hold for any length of time (and sometimes even when tapping quickly) the loop just continues until it reaches the three second time out.

There are two main differences I can notice between them. ONE is this one requires opening a drop-down menu (the added first click action + pause) but as the loop starts, and occasionally finishes, correctly I have somewhat ruled this out at this point. TWO there is a secondary UI element that updates (a knob for the individual selected step lane) in real-time to reflect the change. However this knob can be hidden with a side panel hide/reveal button and even when hidden the issue persists.

I’ve add a short 0.05sec pause in the loop, experimenting to see if perhaps this is the cause and though the over movement feels more fluid, I haven’t found a placement that works to help it trigger the release macro.

µP + Pattern Randomization Hold.kmmacros (23.5 KB)

µP + Pattern Randomization Release.kmmacros (3.0 KB)

I’ve been testing things for the last 3 hours can not figure it out…

Any tips would be greatly appreciated

[fossie]

I've done some more testing and I'm now less certain the issue lies in KM. I have two momentary buttons in MetaGrid Pro that should be working identically — one for a Swing Hold function and one for a Pattern Randomization Hold function. Both are set up the same way with a press action triggering a hold macro and a release action triggering a release macro. The Swing button works perfectly but the Pattern Randomization button is not reliably triggering the release macro from MGP.

When I test both macros using direct keyboard hotkeys they work correctly, which suggests the KM macros themselves are functioning as expected. The problem seems to be that MGP is not consistently sending the release trigger for one button but not the other, despite them being configured identically.

At this point I'm genuinely not sure where the problem lies — it could be something subtle in the KM macros that only manifests when triggered via MGP, or it could be something on the MGP side. I'm posting in both forums hoping someone with experience in both tools might be able to spot something I'm missing.

In case anyone here is familiar with MGP and would like to cross-reference — the button files and more details are available at the link below.

Momentary Switch Button Not Firing Reliably

Cheers