Automating Minecraft "mouse" movement

Questions & Suggestions
1

I know that Keyboard Maestro can automate anything (:smirk_cat:), but I haven't figured this one out.

Inside Minecraft (I'm using the Java edition on MacOS), the mouse is captured by the app. While in the application, you don't see the mouse cursor move around the screen. Instead, a crosshair cursor stays in the center of the screen (the Target for any action), and mouse movement turns your character – towards the left, right, up, or down.

I've mostly experimented with Move Mouse (Just move) — since the normal action by the user is moving the mouse. I don't see any effect of the action when executed. But when I shift the mouse just a tiny bit once the macro is complete, the view shifts more than I move, which makes me think the mouse movement macro was registered... but doesn't take effect until the Real User acts.

I'd love any suggestions for doing this kind of "mouse" movement in Minecraft – or ideas for other tools if Keyboard Maestro isn't the right one for this job.

2

If you are not familiar with this kind of "mouse" movement – you can try it in Minecraft's recently released "classic" edition. This is not a way to test a real macro — since this Minecraft is implemented in the browser, totally different from the native Java client. But it is a good way to see the kind of mouse/cursor character direction/movement I've described.

no registration or installation required:
classic.minecraft.net

Move the mouse to change your direction, press "w" to move forward.

3

Hi, i'm interested.
Have you been able to get this to work?

4

I've tried a dozen or so mouse movement utilities and they all have the same behavior. I'm guessing they use the same hooks into the mouse universe.

I should refine the statement from my original post. After registering a Mouse Move Request via KM (or any other utility), the screen jumps to the desired location on the next physical movement of the USB mouse, not when the macro is complete.

Where does that leave us? I saw another (new-ish?) automation utility a couple months ago that I haven't tried yet. I have a book on MacOS internals I haven't read yet. And I'd like to find a Mouse Expert who can suggest a tidy workaround.

Until then, I do have a proof of concept I've used successfully for mouse movement automation in Minecraft with Keyboard Maestro. Don't get too excited unless you're a fan of Rube Goldberg or perhaps Joeseph's Machines.

:chipmunk: There's a 2021 Digital Rube Goldberg Machine Minecraft Contest.

Since movement of the physical mouse completes each macro request, just move the physical mouse after each macro-driven request. It would be crazy to do that manually, so more automation is needed.

I built a robot using LEGO Mindstorms™ that jiggles the mouse ever-so-slightly. The physical mouse movement is added after the macro request. So the key is to jiggle as little as possible. Most Minecraft targets are large enough that a few extra pixels won't matter. Or I suppose you could pre-calculate to compensate for the jiggle — but your jiggler would have to be much more precise than my prototype.

I started with a Mouse Mover design I found at Instructables. I simplified the design and program (using Swift Playgrounds for iPad.) View the jiggler in action (iCloud shared video).

There are tons of other ways (hardware and software) to move the mouse a bit, since mouse movement is often used to Guarantee The Human Is There. My favorite was the guy who used a Vibrating Football Game, a "favorite since 1947."

My next step is to learn enough to record a video of the machines at work inside and outside Minecraft. Then I'm hoping someone who sees it will say "Why didn't you just…?"

background
If you're wondering "why do this at all?", the goal is to automate stuff in Minecraft that cannot be automated on an unmodified Minecraft server. For instance, my proof-of-concept is an item sorter for unsortable items. Another program (in Java) recognizes the item on screen (by captured text or picture). Then it requests mouse movement to set the destination chest for the item via lectern or item frame.