Mirror click across several application windows

[Limewolf]

Let's imagine I have a grid of windows, of the same application.

Window 1 | Window 2
-------------------
Window 3 | Window 4

Is there a way that I could click in a specific spot in my window 1, and then mirror that click in the same relative spot in the other 3 windows?

I tried an AppleScript that errored out, which seemed needlessly complicated.

(Applescript for reference, although I imagine this can be done with KM built-ins)

-- Get the position of the click relative to the frontmost window
tell application "System Events"
    set frontWindow to first window of (processes whose frontmost is true)
    set windowPosition to position of frontWindow
    set clickPosition to position of mouse relative to windowPosition
end tell

-- Store the click position in Keyboard Maestro variables
tell application "Keyboard Maestro Engine"
    setvariable "clickPositionX" to item 1 of clickPosition
    setvariable "clickPositionY" to item 2 of clickPosition
end tell

[griffman]

Is each window the same size? And if so, by relative location, do you mean that if you click at 100,150 from the top left corner of window 1, then you want to also click there in windows 2, 3, and 4?

-rob.

[Limewolf]

They are indeed the same size. And yes, it would be relative to the position of the top left corner of each respective window.

If it helps, the initial click will always be in the upper left window. But it can be dynamic within that window.

[Airy]

@Limewolf

Is there a way that I could click in a specific spot in my window 1, and then mirror that click in the same relative spot in the other 3 windows?

Is your mouse programmable (or at least have some extra buttons that send USB device key codes)? If so, it is possible to make KM trigger on a mouse click. But if not, would you be willing to use a keyboard key, instead of the mouse, to trigger the action to click the mouse in all four windows? (KM does not have a trigger that goes off when you click the mouse.)

Either way, I think it can be done in three actions: two nested loops, each going from 1 to 2 (or 0 to 1), and a single click mouse action inside the two nested loops, which will do a multiplication and an addition in order to convert the 1 and 2 into proper mouse coordinates. One possible solution would look like the following, with different constants to account for your window positions...

Even though you need to do some investigation and some arithmetic to replace the values of 80, 100, 800 and 1000, with new values to reflect your display, the idea seems sound. If, instead, you want to do calculations based on relative positions within windows, the math may not be any easier.

Press COMMAND-5 while in the KM Editor to get the Mouse Window which will be very helpful to assist you in replacing the four constants above.

[griffman]

Can you tell me which application this is for? I have a (very!) hacky solution that works, but I think it could be much better depending on what app it needs to work with.

-rob.

[griffman]

Actually, scratch that question—ChatGPT and I figured out how to get the window list from any application :). So, here's my solution in action...

Animated GIF hides here

The first version I came up with was really ugly, as it tried to move the mouse around the screen. This version is much better, and has some notable features:

• It will work with any number of windows. However many are open in the frontmost app when the macro is activated is how many it will attempt to click in.
• As seen in the GIF, the windows don't need to be the same size, and don't need to be aligned (and don't even need to be on the same display).

Here's the macro:

Download Macro(s): Click in all four windows.kmmacros (46 KB)

Macro screenshot

Macro notes

• Macros are always disabled when imported into the Keyboard Maestro Editor.
  • The user must ensure the macro is enabled.
  • The user must also ensure the macro's parent macro-group is enabled.

System information

• macOS 14.4.1
• Keyboard Maestro v11.0.2

Briefly, here's how it works. You invoke in the window you want it to begin in (and that window must be active). Position the mouse where you want to click, then invoke the macro (assigned to Shift-Control-P, but change it to whatever you need).

The macro then gets the name of the active app, and generates a list of open windows in that app (and removes the active window from the list). For the current window, it saves the mouse's location, then calculates how far that is from the upper left corner of the window. Finally, it clicks the mouse—that is, once you disable demo mode. In demo mode, it simply highlights the mouse location. To disable demo mode, disable the first action in the macro.

Next, the macro cycles through the window list. It activates each window in the list, pauses to let it become active, then moves the mouse to the same relative location as in the first window, and clicks.

Repeat until the loop is done, and that's that. As noted, this should work for any number of windows of any size, in any location. The only requirement is that the same relative location from the top left of the window must be visible in all windows.

You may have to adjust the two pause actions in the macro, depending on the speed of the Mac and/or the app in question.

Let me know how this works for you; I recommend first leaving it in demo mode, just in case the clicks don't go where you think they'll go.

-rob.

[Limewolf]

Thanks @griffman, this is amazing.

Works for most processes it seems, just need to figure out why some applications don't show up as available windows to be looped through at all. Not even when fetching all processes with AppleScript.

[griffman]

Which app(s) don't show up?

-rob.

[Airy]

@griffman

Actually, scratch that question—ChatGPT and I figured out how to get the window list from any application

That is amazing. AI is coming, and is already here. But the reason I may not use AI is that I actually enjoy coming up with solutions myself. To me, programming is like physical exercise, something that you like to do because you know it's good for you.

[griffman]

I do what I can, then if I get stuck and still want to solve a problem, I ask for help. Given I'm not a coder by training, I get my mental exercise just from coming up with the macros themselves, and coding the bits I can cobble together when I can. (I know just enough Javascript, PHP, Python, Perl, shell, etc. to get little things done, but almost always need to dig for more complex projects.)

Without its assistance, I never would've finished the AppleScript that makes the macro possible. I had all the other parts done, but it was a real pain trying to find the windows. With the script, it's magically easy :).

-rob.

[Airy]

For our benefit, can you share the actual question that you have to AI? I suspect you had a phrase like "Find the Windows" in the question. I would like to note that the original question did not use any such phrase. Instead, it spoke of windows in fixed positions.

[Nige_S]

Do you actually need AppleScript? (I can't believe that's me saying that!)

If you want to cycle through all open windows, loop WINDOWCOUNT() times, activating the -1 window (furthest back in the stacking order). If you only want to cycle through the front four windows, repeat 4 times activating window 4.

If you don't need to target windows by name or similar you only need to manipulate the stack order.

Example for every window. As with @griffman's, activate your "start" window, position pointer, trigger hotkey. You'll end up with your original window frontmost again:

Click in Every Window.kmmacros (4.0 KB)

Image

[Limewolf]

@Nige_S would it be possible to limit that to n occurences, in case I have windows open that I don't want to be affected?

[Nige_S]

@Limewolf

@Nige_S would it be possible to limit that to n occurences, in case I have windows open that I don't want to be affected?

Yes, you just repeat n times, activating the nth window each loop.

Think of your windows like a stack of cards -- when you pull one from the middle to the top, the cards that were below (behind) it stay where they are while all those in front drop down by one. If you always pull the nth you'll cycle through the top n windows.

So the main question is -- how do you determine n? Always the same number? A fixed fraction of the currently open windows? The window whose title is "Pick Me!"? Ask with a dialog?

So many choices...

[griffman]

@Nige_S

Do you actually need AppleScript? (I can't believe that's me saying that!)

Apparently not :). This is a great example of (a) I'm not a coder, and (b) there are still things about Keyboard Maestro that I don't know: WINDOWCOUNT() and that you can bring windows to the front based on index. Very nice solution!

@Airy

For our benefit, can you share the actual question that you have to AI? I suspect you had a phrase like "Find the Windows" in the question. I would like to note that the original question did not use any such phrase. Instead, it spoke of windows in fixed positions.

Here's the ChatGPT side of things:

ChatGPT

You'll notice there's a big logic break near the end, where it proposes a quite-long script that didn't quite work. At that point, I was frustrated so I headed to Bing Chat, which helped me come up with a more-functional script. I wasn't logged in to Bing Chat, though, so I don't have any history I can retrieve from there.

I then pasted that functional script back to ChatGPT, as Bing Chat was having trouble with the last step for me—which ChatGPT handled properly (encoding as a line-break-formatted list). I then tweaked the code in Keyboard Maestro to work with a passed-in application name, and that was that.

But @Nige_S' solution is so much more elegant and functional, there's no reason to use mine!

-rob.

[Airy]

@griffman

Here's the ChatGPT side of things:

Yikes, that was a lot of work. I think it makes my 3-action solution much more appealing. I guess I won't be out of a job before I'm dead.