How to Find the Position of the Mouse on an Image

[iamdannywyatt]

@noisneil

It searches for images the same way your eyes do, using what it can see on the screen, so obviously if an app isn't open, it can't search for images in it

Exactly my point! So if you're building a macro and you need to find the coordinates of a specific point in an image that's not even there, how would the GO button help me, if it worked? The image is not visible, even if the app is open. If KM is taking my full screen and the other app is behind it, how would KM find it? As you said, if my eyes can't see it, how could KM?

Also, even if KM found it by "magic", how would that give me the coordinates of the section of the app I just selected and copied?

@noisneil

There are easy ways to achieve this that don't involve rewriting how parts of the editor work in order to satisfy a very niche use-case.

To be honest, I don't think this is a very niche-case. If you have the button that says GET so you can get the coordinates of the screen, that means that you need to know coordinates of something, right? If we have the option to click a specific place of a found image that requires coordinates, I don't know how that is a niche-case?

@noisneil

When you go to KM you're switching apps, so rather than do that, test a larger part of your macro. For example, test opening the dropdown menu and then clicking the found image. Another option is to have Try Action set to a hotkey like this:

All I am looking for is a tool that shows me the coordinates of the mouse. Why all the extra actions, extra steps, extra macros, extra testing? If I use the Try Action, I will have to keep playing that action until I see where the mouse is, adjust the horizontal and vertical pixels, go back and hit Try, etc. Might as well just open Photoshop and do it

@noisneil

In Photoshop, you're dealing with 1:1 image resolution, and coordinates are relative to nothing but the image itself.

And why can't KM have the same functionality so we can easily paste an image and find the coordinates of the mouse of a particular place in that image?

@noisneil

Anyway.... Try this. Select your found image action and trigger this macro with a hotkey:

Didn't work for me...

Anyway, this became more complex and confusing than it needed to be, to be honest haha
My brain is about to explode, because it seems that it's "impossible" to do it natively without extra steps, macros and actions, that it's just easier to open Photoshop and find the coordinates.

Thanks anyway for the time you spent trying to help with this issue

[iamdannywyatt]

@noisneil

You're clicking the Go button while the KM Editor is obscuring your found image. If you can't see it, neither can KM, so of course it won't find it.

That's exactly what I was saying from the beginning
That's why I said that it would be better to have something that works inside KM without moving windows left and right. But again, even if the image was visible, that would not give me the mouse coordinates of a certain position in the image relative to the image itself, only relative to the screen.

[tiffle]

@iamdannywyatt

how would the GO button help me

@iamdannywyatt

how would that give me the coordinates of the section of the app I just selected and copied

KM gives you tools to do things. As for getting coordinates and so on, it does whatever it can to help whenever it can and sometimes it just can't help much. I mean - if you can't see the image on screen then you won't be able to copy it into the Found Image action, will you? So how do you expect KM to do stuff with something it can't see?

@iamdannywyatt

All I am looking for is a tool that shows me the coordinates of the mouse

Since your best tool is

@iamdannywyatt

about to explode

why not try a screen ruler? The one I use when KM isn't helping me is xScope; there are plenty of them - even free ones.

[iamdannywyatt]

@Nige_S

But why do you need to find the mouse coordinates inside the image

First reason: if KM gives us the option to enter mouse coordinates on a found image, that means that some of use actually use them... otherwise, why would the option even exist? I used it a few times and it's useful.

Second reason: Sometimes I want KM to find a unique section of the screen, but then I want to pick where it clicks.

@Nige_S

Also, mouse coordinates are relative to something

Yes, if I have a painting on a big wall and I paint a dot in the middle, I can say that the dot, relative to the wall's top left corner is at 1978cm to the right and 200cm to the bottom. But relative to the painting itself, it's just 21cm to the right and 13cm to the bottom.

[noisneil]

@iamdannywyatt

how would the GO button help me

Make the KM Editor window smaller so you can see the image in question and then press Go. Orrrr.... test a part of the macro that includes making the image visible.

@iamdannywyatt

Also, even if KM found it by "magic", how would that give me the coordinates of the section of the app I just selected and copied?

Via the %FoundImage% token.

@iamdannywyatt

And why can't KM have the same functionality so we can easily paste an image and find the coordinates of the mouse of a particular place in that image?

Because KM wasn't written according to the whims of a particular user. It's a great tool that can achieve a lot, and we're demonstrating that right now by providing solutions for your problem.

@iamdannywyatt

Didn't work for me...

Elaborate with screenshots please. You must have misunderstood the instructions. Or perhaps you're still trying to find an image that isn't onscreen, in which case... stop doing that.

What about this one?

[iamdannywyatt]

@tiffle

So how do you expect KM to do stuff with something it can't see

Thing is: I don't!
I just shared this video, because you guys were saying that the GO button was working for you, while it wasn't working for me. I was confused about that, because my goal was not to find the image in it's original location. Again, if the image is a small part of a full app, how would I get the mouse coordinates of a specific position in the image? If the image I pasted is 100px by 100px and I want to click at 73px and 14px, from the top right corner of the image, how would the GO button help me even if I have the original window with that portion of the image visible on my screen?

@tiffle

Since your best tool is

The way I've been using is pasting the image into Photoshop and hovering over the image, I can see the coordinates. I shared an animated GIF showing it.

@tiffle

why not try a screen ruler

This wouldn't solve the issue completely. The only option that could help in certain situations, but most of the time it's a mess, is what @noisneil suggested, with the screenshot shortcut. But as I mentioned, in certain background, I can't even read the numbers.

[noisneil]

@iamdannywyatt

If the image I pasted is 100px by 100px and I want to click at 73px and 14px, from the top right corner of the image, how would the GO button help me even if I have the original window with that portion of the image visible on my screen?

You'd set the click point relative to the top-left corner and then put those numbers in the coordinate fields. The Go button would help you test whether you got the numbers right by showing you where the pointer would land.

[iamdannywyatt]

I must be saying that's so out of this world...
Here's an image that I want to use on that action, for example:

Can you see that on the O in HOW, there's a small red dot?
Well, I want to find the coordinates of that dot, relative to the boundaries of the image itself. Photoshop tells me this:
X= 70
Y= 17

It doesn't matter if KM finds the image in the context of the KM forum, on the website, because it would give me the coordinates of the red dot, based on the screen or the browser's window, which would be something like this for the position relative to the browser's window:
X= 149
Y= 135

And if the window was not taking the whole space and it was moved somewhere else on the screen, it would show me something like this (based on the window position I have now):
X= 683
Y= 357

So the solutions you are providing (unless I make your macro work and it does what I'm looking for), will not give me the X and Y of the red dot inside the image, relative to the image itself (70 and 17).

Is this really just a "whim of a particular user" to know the mouse coordinates on a particular image, when KM gives us that option?

Now, if this isn't possible with KM, that's one thing. But what you guys have been suggesting, has nothing to do with what I am trying to know. The option of using the screenshot feature would work, but as I said, sometimes I can't even ready the numbers.

[noisneil]

@iamdannywyatt

So the solutions you are providing (unless I make your macro work and it does what I'm looking for), will not give me the X and Y of the red dot inside the image, relative to the image itself (70 and 17).

Both of mine do what you want to do.

@iamdannywyatt

Is this really just a "whim of a particular user" to know the mouse coordinates on a particular image, when KM gives us that option?

Yes because it isn't built in the way you wish it were, but KM will give you that information using macros designed to do that. A Swiss Army knife isn't a worse product just because it doesn't do the whittling for you.

@iamdannywyatt

But what you guys have been suggesting, has nothing to do with what I am trying to know.

That's not true. Be patient and try to get the macros we've taken time to make for you working. If they're not working, discuss that with us so we can try to help. You already have the solution; you just need to take the time to get it working.

The second macro I gave you (the one where you click the mouse twice) is the same as using the screenshot tool except the numbers appear in a window.

[Nige_S]

@iamdannywyatt

Second reason: Sometimes I want KM to find a unique section of the screen, but then I want to pick where it clicks.

@iamdannywyatt

But relative to the painting itself, it's just 21cm to the right and 13cm to the bottom.

OK -- now we're getting somewhere. You want to click at an offset you determine (somehow), relative to the top-left corner of the found image. We just need to find an easy way to do "somehow".

One way would be to paste your image into a new Preview document and then, with the rectangle tool, drag from the top-left corner to the point you want to click -- Preview will show you the size of the selection which, in this case, will equal your (x,y) offset in KM. Obviously, that only works when your desired click-point is within your image!

If you don't mind maths, even quicker (and more versatile) is to ⌘-⇧-4 into Screen Capture and make a note of the image origin and then your desired click-point. Hit ESC to exit capture mode, then calculate the offset you need. The downside would be that you're "guessing" your image origin compared to the image you copied in to KM, so good for most things but not when pixel-perfection is required.

So, back to KM. Here's another version of the macro. You can paste your desired found image into action 1, then run it. It'll wait until the image you want can be found on-screen, giving you plenty of time to ⌘-Tab to get to your app, trigger menus, or whatever. It'll grab the top-left coords of the found image, then flash a notification telling you to click at your desired offset "click-point". Do that and it'll throw a dialog showing the X and Y offsets to use in your macro.

Is that closer to what you are after? If not, why not? As @noisneil says, easy enough to iterate over an "almost" until it works...

Mouse Offset v2.kmmacros (38.6 KB)

Summary

[iamdannywyatt]

@noisneil

e

This could be close to that, but the issue is that if the places you click are for example buttons, or areas that behave a certain way when you click then, such as a drop down menu, you wouldn't be able to make it work. But it's pretty close. If it was possible to somehow make it as if you put a "layer" on top of the screen that blocks any mouse actions that affect where it clicks, then it would work. Think of this as you having a piece of paper and if you tap it with a pen, it leaves a mark. But if you put a glass on top and tap it, the paper is intact and you have the dot on the glass, not the paper.

[noisneil]

@Nige_S

The downside would be that you're "guessing" your image origin

Not if he uses my macro, which "tries" the found image action to place the mouse first. Of course he'll need to make the KM editor smaller so the image can be found first.

The two-point-click macro doesn't need the KM editor to be visible, so there's that.

[noisneil]

@iamdannywyatt

if the places you click are for example buttons

Then use the other macro, first temporarily changing your found image action to "just move" rather than "click".

[iamdannywyatt]

@Nige_S

One way would be to paste your image into a new Preview

Yes, I read about that option as well somewhere. That's something I could try using more often which is faster than opening Photoshop.
I never use images bigger than my screen anyway, so this is ok.

@Nige_S

If you don't mind maths, even quicker (and more versatile)

I think this would be slower and more complex than the Preview option, in my opinion.

@Nige_S

Here's another version of the macro

I will check this later tonight, because I have to leave now. I will let you know if that worked. Thanks!

[iamdannywyatt]

I will check this later tonight. Have to leave now.
My suggestion if it's even possible:

Instead of clicks, what if the macro waits like 3 seconds and after that it takes the first "screenshot" of the mouse position. Then waits another 3 seconds (so we can move the mouse) and takes the second screenshot, similar to clicking the second time?

Would this be possible? Not sure if that's what you meant?

[Nige_S]

@noisneil

Not if he uses my macro

Agreed -- but the aim was to show him a couple of quick'n'dirty non-KM methods, along with some of their limitations.

[noisneil]

• Make the image visible by reducing the size of he KM window.
• select your found image action.
• trigger my macro. It will place the pointer on your found image.
• now move your mouse to the point you'd like the relative coordinates for and wait. They will pop up within 3 seconds of running the macro.

[Nige_S]

@iamdannywyatt

I never use images bigger than my screen anyway, so this is ok.

You misunderstand -- this method would fail if, for example, your desired click point was to the left of your found image because it would be outside the bounds of the capture you've put into Preview.

We're going for as versatile a solution as possible here!

@iamdannywyatt

I will check this later tonight, because I have to leave now. I will let you know if that worked. Thanks!

It does work, for me. But you'll almost certainly have to tweak it for your use. It uses left-click to set the second point, so if you need to open menus first you'll want to change that to right-click -- unless you also want contextual menus, in which case swap the "Mouse button:" condition for some whacky "Key" condition that you aren't otherwise using.

On using "waits" instead of events -- certainly doable but, IMO, if you are going for the most versatile solution it would be better to base on something you trigger when ready. Then it doesn't matter if it takes you 1 second or 1 hour to set things up -- no time wasted and no "you weren't ready in time" errors.

[noisneil]

@Nige_S

some whacky "Key" condition that you aren't otherwise using.

Good idea. A multi-modifier mouse click would probably work nicely. (Edit: Nope, still presses buttons.)

@iamdannywyatt

Hitting a single modifier works nicely:

KM - Get Coordinates (Relative).kmmacros (50 KB)

Macro screenshot

[Moosae]

@noisneil

Yes because it isn't built in the way you wish it were, but KM will give you that information using macros designed to do that. A Swiss Army knife isn't a worse product just because it doesn't do the whittling for you.

I agree with what you said