Click at Found Image Action Detects Button but Fails to Actually Click It

This is an instance of a problem that occurs intermittently in multiple scripts. I know that KM detects the button, because the Pause Until action says that the condition is currently true. But the following click action does not work. Since I've enabled Display, it puts green shading over the button. I've adjusted fuzz until this button and no other is detected. Any ideas?

macOS 10.14.5
Safari 12.1.1
KM 8.2.4

Hey @dp_,

Turn OFF “Display” in the first action.

Change “Click” to “Move” in the second action.

Turn off all other irrelevant actions and play around with this to see if/where the cursor is moving.

I've occasionally seen cases where buttons/images had internal blank spots, where clicking failed – and I had to adjust the position of the cursor a bit to get a valid click.

-Chris

Thanks, Chris. I've tweaked the coordinates, and have moved the pointer all over the button, without effect. All parts of the button are active when manually clicked, but none when I use the macro.

I have a few ideas.

Although Find Image is my favourite action of all time, it can be finicky and can completely fail in some cases. Especially on solid colours. Yours isn't solid, but it does have a lot of solid, especially around the edges. In a case like this I usually add a little bit more to the image, perhaps your button has a white edge around it, In that case, capture the white edge on the TOP of the image. It works better with asymmetrical images.

Another possibility, which we can't see, is whether you managed to put the exact same bitmap into the two boxes. They look the same, but form this side we can't be sure. Did you know you can click in the image boss and from there use macOS's copy and paste to copy it to the other box?

An even better idea is to use the FoundImage token. That token, according to the documentation, should contain the actual location value of the image in the first Pause statement. So you don't have to relocate it the second time.

Hey @dp_,

Well, that’s frustrating...

What happens if you set up a pure click at coordinates action to click on that button?

If you want/need to experiment with an alternate means of clicking you might check out this command line utility:

https://www.bluem.net/en/projects/cliclick/

-Chris

The image has no visible border. I did copy and paste from one action to the other to guarantee identical copies. The word Home appears next to the button, so I tried using the button and a slice of the H, then adjusting the relative coordinates so that it clicks on the button and not in the middle.

21%20PM

This didn't work, either. I inserted the Find Image on Screen action, but the subsequent Click action still does not work. I read the manual about the FoundImage token. It says that the Find Image on Screen action will generate it, but it doesn't tell me how to access it. It's not one of the options in the Click action. Can you please elaborate?

Okay, we tried. As for your question, after the Pause statement, insert this statement:

I haven't used it with the Pause Until statement, so I can't say 100% certainty that this will work, but it may. What it does is fetch the actual location data of the last Find Image. I use this approach a lot, but I haven't used it with the Pause Until Image Exists macro. If it doesn't work directly we can make it work by modifying your Pause Until statement.

So here's how you could modify your code to ensure that any image found in the pause statement will be findable by the next statement:

I know the wizards on this forum won't like the fact that I didn't actually test something that I uploaded here. But I just know this should work because I do this so often. :slight_smile:

Basically what that does is test for an image in a loop. If the loop finds the image, you don't have to find it again after the loop completes. You already have the image's location in the variable LocalPauseImage.

1 Like

Hey the Pause Until Image Found actually does modify the FoundIMage token, so an even simpler solution is this: (which I did test to be sure!)

Instead of displaying the image location, you can assign it to a variable and click on it or do whatever you want with it.

Okay. This:

Failed with this error:

28%20PM

I'm moving on to your next suggestion. Thanks for taking the time for this.

You were close. I've had that error too. You just need to modify it a little. You can't use the USE action because the extra parameters from the FindImage token confuse it. You have to extract the components of the FindImage token. There may be more than one way to do that, but the cleanest way is probably to assign the token to a variable first then use the first and second numbers in the variable. Let me create an example for you. But you may already see what I mean.

This should patch you up:

You may be wondering why the "+5" is there. That's because the token always returns the upper left corner of the image which often is far from the center of the image. So usually when I add 5 to it, it can reach the button's area.

Okay, here's what I've got now. It doesn't throw an error, but it still fails. And it doesn't abort. It will keep running until timeout. Pause Until Conditions are Met returns True, but the Click statement just sits there like it doesn't know what to do.

Hey @dp_,

Set the mouse location with a variable expects x,y coordinates:

image
So to use your original image info you have to strip off the excess.

Here's one method:

Keyboard Maestro Actions.kmactions (12 KB)

I normally use @Sleepy's method myself, but this works.

-Chris

We must be getting close. But I'm not sure what you mean by "the Click statement just sits there". I don't think it's possible for the Engine to "just sit there" when running that particular action. So I'm perplexed. Can you open the debugger? Does the last action sit in the debugger window for a long time? I don't see how that's possible. I think you might mean it's sitting on the Pause Until action. The debugger can clarify this.

And you mention a timeout but you didn't say where you set the timeout. Is the timeout on your Pause Until action? If so, then that means the Pause Until action never found the image. That wasn't your problem earlier. In any case if that's the problem now, that's a different problem. It means the image is never found. We can fix that too, but that wasn't the initial problem.

Thanks, Chris. I'm going to follow through with Sleepy's method, and return to this if that fails.

Chris's method is a variation on how to extract the data. It's another way. Personally, I'd rather see the Use action updated in KM to allow your use case to work properly. I always find it a bit annoying to have to take these extra steps for no apparent reason. I think it's time for me to get some breakfast. I come here to wake up and you've helped me do that. :slight_smile:

Well, it's time for me to get some supper and some sleep. I'll start debugging when I wake up. Thank you very much for your help.

For humans who sleep 8 hours per day, there's still a guaranteed minimum of 8 hours per day that two people in different time zones have the ability to interact while awake. That's the same amount of time as I spend with coworkers when I go to work with someone in the same time zone.

I've run into the "click doesn't work" issue. I sometimes can get around the problem with Applescript and System Events.

tell application "System Events"
   click at {750, 550}
end tell

OR

tell application "System Events"
   tell process "processName"
      click at {100, 200}
   end tell
end tell

I have a gripe for the KM staff. Why is it that a button that can be manually clicked anywhere within its borders require us to jump through so many hoops to click using KM? I haven't had this problem with other macro programs.