How to Find the Position of the Mouse on an Image

When I have the Click at Found Image action, clicking the GO button, doesn't do anything. I can hit Space and see the image, but that doesn't allow me to get my mouse location.

Is there a hidden setting for this? Or a workaround? If not, how do you usually find the mouse position on an image? Usually I paste that image in a new Photoshop document and use the Properties, but this is way too much work.

Any tips?

Try turning on KM's mouse display found in the KM Window menu:

KM 0 2022-06-14_12-05-58

1 Like

I was wishing this was the solution, but it doesn't give me the mouse position relative to a found image. For example let's say you pasted an image to the Found Image action. If you click it and hit Space, you can preview the image (Quick Look type thing), but it's centered on the window. The Mouse Display is always related to the corners of the window the image is being previewed on, not the image itself, or the corners of the screen. You know what I mean?

1 Like

It should place your mouse over the found image. Does it not do that?

Why would it?

What do you mean by this?

Can you try to rephrase your question? It's a bit unclear what you're after.

I think it's better if you try it yourself. Grab an image and use it for the Click At Found Image. Click the GO and see what happens. It will show the notification that it couldn't find the image.

When you hit SPACE to preview the image you just pasted, it would be good to have at least the option to turn some setting ON that would show, based on the image, not the window, what the mouse position is.

As I mentioned, if you have an image that you want to find on a screen and click on a particular place on that image, you need to have the coordinates for the action, right?
So how do you do this right now? My only option is to paste that image into Photoshop and check it there, which is a pain. I don't always have PS open and it takes a long time for the whole thing when KM could have that option.

Is it clear? Let me know.

1 Like

I can’t get mine to fail at all. Let’s see the action image to access the settings.

I'm confused now, because maybe I'm missing something, or you are all confused about my question, so here it goes. Let's say I have this image:

Screen Shot 2022-06-14 at 1.18.33 PM

Now I add it to a Click on Found Image action:

Screen Shot 2022-06-14 at 1.18.56 PM

I want to know the coordinates of the part where it says 1/8.
Clicking the GO button, gives me this:

Screen Shot 2022-06-14 at 1.19.15 PM

If I click the small square where the image is and hit SPACE, I can preview it in a screen size window:

There's no GO button here to show me the mouse position in the image.

Activating the Mouse Display as suggested, will give me the mouse position relative to either the full screen or the position relative to that preview image. Since the image preview window takes up pretty much the whole screen, I get these values:
705 and 445, which is not the values from the corners of the image but from the corners of the screen/window.

Hope this helps.

Works fine here: Here's my Found Image action:

KM 2 2022-06-14_13-19-40

This is what happens when I click the Go button:

KM 1 2022-06-14_13-18-49

You can see the pointer has been positioned over the centre of that part of my desktop that contains the image KM is looking for. I just so happen to have positioned KM's mouse tool next to it so you can see the coordinates.

This is the part of the desktop that contains my target image:

KM 3 2022-06-14_13-22-23

It seems to me you're confusing the image in the KM action itself with the actual image that you're searching for. BTW, it's easy to get coords relative to the target image's corner: just place your pointer at the target image's corner (the actual target, not the one in the action); remember the x and y coords; then move your pointer to a desired position and remember those x, y coords. Then simply subtract the first x,y from the second x,y to get the relative position.

Then you haven't found the image -- play with Found Image's settings until you do.

Then you can use the Mouse Display -- click the padlock and you have 5 seconds to click the action's "Go" button before the coordinates lock. You can either write them down or copy them. Unlock Mouse Display, move mouse to your "offset position", note coordinates, do maths.

But -- hang on. We've got KM! Set the image you want, tweak the settings so it's found, run macro, move mouse to your offset location during the pause, await results...
Mouse Offset.kmmacros (28.1 KB)

Summary

Nope, works fine. Check KM has screen recording permissions in security prefs. Also, it's often a good idea to restrict the search to the Front Window (rather than "all screens"), to make finding a unique image less prone to error.

Right, so you want to press the Space bar to show the image, then mouse over it to see what the mouse position displacement would be from the image's centre or corner? This wasn't very clear. It also wouldn't be practical, as found images are not displayed at actual size when previewed in the KM editor.

I have a method that will work once you've figured out why Go isn't working for you:

  • Hit Go to send the mouse to the found image
  • Without moving the mouse, hit your screenshot hotkey
  • Click and drag to the position on the image you'd like to click, but don't release the mouse button
  • Make a mental note of the X/Y coordinates shown onscreen by the screenshot tool
  • Hit Escape to cancel actually taking a screenshot
  • Enter these coordinates into the Click at Found Image coordinate fields

I would add that you don't need to use a found image for this. Just use a click relative to the front window's corner.

I'm still confused how KM knows what window it needs to open when you click GO?
What if the image is not even on any window, but it's from a closed app, for example?

Also, look at the coordinates from the Mouse Display. Your image is super small. How can you have 2497 and 821? That's the coordinates relative to the mouse position on your screen. If you use those coordinates on your action it won't work, because it will look for 2497px to the right from the image's top left corner, which your image doesn't have. The same for the 821 px down.

No confusion at all. I want to know the coordinates of the image I used when I pasted it. A 100px by 100px image, if I'm using the top left corner as my anchor and I want the cursor to click right at the center, then it would give me 50 and 50, regardless of where the image is on the screen.

"Easy", you say...? It's almost the same time and work as using the Photoshop method. :slight_smile:

What if the image is from an app that's not even open?
To me it would make more sense that KM would have this implemented.

Yes, both Editor and Engine.

Thanks for the tip. I never messed with that and I never had issues, specially because I only have a screen anyway. I will try that from now on though.

Exactly.

This could be something that the software could work around. For example, if the image was too big, bigger than the preview window, we could set it to 100%. Then based on the image's total size, it would calculate where in the image the cursor was. Not based on the cursor's position on the screen, but on the pixel count of the image itself. I believe this would be possible to implement. I mean, it's 2022... we already have nearly perfect deep-fake technology. What can't be implemented, right? haha

We go back to the same issue: what if the image is not even visible on any of the apps? My issue is not that the GO isn't working. I think the GO should work inside KM, not outside. What if the image I used is from a dropdown menu that only open when you click with the mouse? The moment you go to KM, the menu closes.

The workaround you mentioned is a good one (without the GO button in the process). The only issue with that, is the font and color Apple uses for that. Try to use that method over a text image, for example. It's a mess.

This was just an example. I'm looking for a method that works in any scenario. I wasn't even creating a macro for Logic, actually. It was for something else.

Notice how Photoshop, regardless of the size of the image on the screen, keeps the coordinates relative to the image. This could be the same for KM
Kapture 2022-06-14 at 14.59.33

Then you would programatically open the app and wait for its windows to be open before searching for a found image in one of them. A Keyboard Maestro action ordinarily does one thing; in this case, it's to search for a found image in a designated area. It won't open an app for you beforehand. It searches for images the same way your eyes do, using what it can see on the screen, so obviously if an app isn't open, it can't search for images in it. This is fundamental stuff.

The reason I try to restrict my image searches to a specific window (and more often, to a specific area of that window) is that KM then has less screenspace to search. This means that not only is it easier to discern a unique image, but the image search will be a little faster.

There are easy ways to achieve this that don't involve rewriting how parts of the editor work in order to satisfy a very niche use-case.

Yes, that's right. When you go to KM you're switching apps, so rather than do that, test a larger part of your macro. For example, test opening the dropdown menu and then clicking the found image. Another option is to have Try Action set to a hotkey like this:

In Photoshop, you're dealing with 1:1 image resolution, and coordinates are relative to nothing but the image itself.


Anyway.... Try this. Select your found image action and trigger this macro with a hotkey:

Window Mouse Position v2.kmmacros (23 KB)

Macro screenshot

Similar to @Nige_S's suggestion, move your mouse to the desired location and wait for the coordinates to pop up in a window.

It does. If you "Quicklook" the image pasted in "Move Mouse" action, KM can find it. You might have to ramp up the fuzziness though. But I'm still not sure how this benefits you?

A method that works to do what? To determine a pixel offset between a found point and a future click-point? Again, the macro I posted above should be easily adjustable for most situations -- even if it's something hidden in a menu, just add some "Click menu" actions at the beginning to get to where you want.

I have no idea what you mean, but this is what I get when I QuickLook the image I pasted:

Where are the coordinates?
I did this using the Move or Click Mouse action with the fuzziness all the way to the right.

That shows me the mouse coordinates based on the image I'm using on my action.

I guess what you are probably confused about, is that this has nothing to do with the macro when it's running. I'm looking for a way to, when I'm building a macro, to show me the coordinates of the mouse so I can use them here:
Screen Shot 2022-06-14 at 3.31.55 PM

Using another macro, adding the Click Menu action, etc, this is extra stuff that doesn't make my life easier. The Mouse Display option would be what I need, if it wasn't limited to the position of the mouse relative to the whole screen or a window.

@Nige_S and @noisneil this is what I am doing. Hope this helps understand exactly what I'm facing. All I need is see the mouse coordinates when I'm hover over the image that I'm using inside the action. Pretty simple task.

You're clicking the Go button while the KM Editor is obscuring your found image. If you can't see it, neither can KM, so of course it won't find it. :man_facepalming:t2:

But why do you need to find the mouse coordinates inside the image? What do you do with them after that?

Also, mouse coordinates are relative to something -- what should your (0,0) be?

Here's another way to get relative coordinates, more generally. Trigger the macro, click once at your starting point, then again at the end point and the relative coordinates between the two will pop up.

KM - Get Coordinates (Between Two Clicks).kmmacros (48 KB)

Macro screenshot