It would be great if Keyboard Maestro could create mouse move events on a lower level to allow automations in e.g. games which apparently handle the mouse pointer in a non-standard way.
Using KM to script some actions in a game played on Google Stadia, running inside of the Chrome web browser, I encounter the exact same phenomena that fellow user @Slothbear described 3 years ago in Automating Minecraft “mouse” movement: Scripted key actions (press, hold, release) and mouse button clicks just work fine, but attempts to change the mouse pointer position programmatically just don't take any effect unless the mouse is also moved physically.
While the very same happens with similar tools like e.g. cliclick, the macOS accessibility feature "Mouse Keys" and the "to.mouse_key" feature of Karabiner-Elements can move the in-game pointer through keystrokes just fine. I assume that's because they introduce their mouse events on a lower level.
Unfortunately neither "Mouse Keys" nor Karabiner-Elements react to keystrokes generated by Keyboard Maestro, so I'd like to see the ability to create such "low level" mouse movement events to be added to Keyboard Maestro in the future. Or is there some other way that works already now?
So perhaps my thinking about "lower or higher level mouse events" was all wrong and this little modification would make things work.
Could it be that simple @peternlewis? Can you try?
(I already checked the cliclick source code and indeed cliclick does not set delta x and y, so in case this modification should work for Keyboard Maestro it would very likely also work there.)
Addition: Perhaps I should have mentioned that I was trying to automate relative mouse movements à la "move mouse 200 pixels to the right", i.e. I would place the pointer into the proper position and would then invoke my macro with a hotkey. The macro was supposed to press and hold the left mouse button for 1.8 seconds, release it, wait for 0.2 seconds, and repeat this 500 times. Then move 200 pixels to the right and start all over, for five times — i.e. a total of 2,500 mouse clicks distributed over five horizontally adjacent in-game buttons. The clicking worked fine, but the movements did not take place, so all 2,500 clicks occurred on the same button.
Apologies if I'm missing the point, but relative mouse movements are already easily achieved. For example, this action moves the mouse 200 pixels to the right from the current mouse position:
Clicking and holding for a set period is something you're already doing by the sounds of it.
Yes, but as stated in my original post a Google Stadia game (in my case: Destiny 2) running in the Chrome web browser does not react to this, and someone else has written the same about some other game (Minecraft).
I.e. these games apparently require some other kind of mouse movement event than KM can currently generate. Perhaps filling in additional fields (the delta x and y ones) might help, i.e. developer action is needed.
I will take a look. I don't have access to Destiny 2. I do have Minecraft = do you have a repeatable example of it not behaving as expected in Minecraft/Mac or some other case where I can test it?
If Keyboard Maestro already sets them, the cause must be different, but if it doesn't, here might be an opportunity to make it more compatible with some games.
I could try to make changes to the code of cliclick and see what happens if I add CGEventSetIntegerValueField(event, kCGMouseEventDeltaX, delta_x) and CGEventSetIntegerValueField(event, kCGMouseEventDeltaY, delta_y) there. (I have never developed on the Mac yet so this shall be an interesting endeavor.)
Testing with a "real" application/game is probably best, but I see the same behavior with this browser implementation of classic Minecraft. Another interesting angle on this: however this browser implementation captures the mouse might lead to more clues. When in the game, there's a banner at the top that says
Your mouse pointer is hidden. Press Esc (escape) once to dismiss this banner.
After I start this browser game, I use a hotkey trigger to run one action: Move and Click at (360,350) from the Top Left Corner of the Front Window
(The actual coordinates are not important as long as they don't match wherever the mouse is currently pointing.)
Nothing appears to happen until I move the physical mouse, then the cursor jumps to the desired location.
Testing notes:
This one-action macro shows the same behavior in the Java Minecraft program on MacOS – this is the repeatable test case I use.
Point your cursor at open air before running the macro. Clicking while pointing at a block breaks the block and I had inconsistent results.
In real Minecraft the physical mouse movement seems to reset the system and I can repeat the macro + physical mouse movement sequence many times. In the browser classic Minecraft version, after one run the mouse doesn't jump again after macro + physical mouse movement — until I pause the game and continue (ESC, then click "Back to game").
Thanks for all the investigation. Getting this to work would still be super useful to me. @KeMa if you haven't already tried modifying clickclick I could give it a go.
I have added support for setting kCGMouseEventDeltaX/kCGMouseEventDeltaY in the next version.
OK, well, this changed the behaviour. Previously, this web app ignored the mouse move events. However after making the change and using this action:
It honours the movements for a while, but then I think the mouse ends up too far and then it stops working at all, whether I use the hardware mouse or the simulated events. So I don't know how useful it is going to be, but maybe it will depend on how other applications behave.
I can't wait to give it a try as soon as the update becomes available. Thank you very much 