Click at Found Image Action Detects Button but Fails to Actually Click It

[GregT]

The button click problem has been reported numerous times. Most
of the responders to the reports had experienced the problem
themselves. When I first had problems with button clicks, I
searched and found previous cases. The cases were either unsolved
or had very complicated workarounds. The workarounds differed. I
used QuicKeys for eighteen years, and I can't recall a single case
of not being able to click a button within a macro. KM has
something inherently wrong with button clicking.

  Macros are supposed to save time. Looking through my macros to

find the ones that needed button click workarounds will not save
time. For what it's worth, my fixes were to use the Applescripts I
posted or to avoid the button altogether by using a sub-menu item
or a different approach that generated a different window that
could perform the same function with a different button that KM
could click. For example, many of the items in Thunderbird's
preferences window can also be accessed via Account Settings, and
the windows and buttons are quite different. KM might not be able
to click a button in the Preferences window but have no problem
with a similar button in the Account Settings window.

  Something else of note. Apple can prevent buttons and menus from

responding to anything but true mouse clicks. Screen Sharing is
the best known case. Screen Sharing also prevents the creation of
keyboard shortcuts. I can't imagine third-party application makers
deliberately blocking "clicks" from macros, but I suppose it's
possible.

GregT

[Sleepy]

Your post taught me something I didn't know, about an alternative to the button click action, so thanks.

Yes, I too have noticed anomalies in KM's behaviour, including the mouse. I make a list in a macOS Notes file and occasionally I dip into that list and post a message on this forum from my list. But even in those cases where KM is behaving improperly I just find my workaround and live with it. That's one off the beautiful things about programming, i.e., the computer is never frustrated that it's having to perform "workaround" code every minute of every day. So even in those cases where KM is defective, I don't feel the need to gripe because there's always a workaround, and there's a decent community here who can help you find it.

[dp_]

Although I'm aware of all of the input, something has come up, and I'm not going to be able to re-engage until Monday. I appreciate all of the help, and don't wish to appear as though I've dropped off the face of the earth.

One quick response: The page in question is part of the FACTS (previously RenWeb) SIS (Student Information System). You can't get to the page without login credentials. So direct testing is not an option. But thanks for being willing.

Dropping off now.

[dp_]

Short Version

It works now. Thanks very much to all who helped.

Tedious Long Version

Since multiple people are now involved, and since I don't wish to end up wasting everyone's time with something that turns out to be a stupid assumption about something that I thought to be irrelevant, but which a more expert user would see to be a fatal error, I'm going to step back and be extremely pedantic.

• The context is the web front end to a student information system built on an MS SQL database. It's behind a login screen, and cannot legally be accessed by people who are not employed by my organization.

• I use "click on found image" triggers elsewhere in my interaction with this database. Most of the time, they work as intended.

• The physical context includes a MacBook Pro laptop and an LG Ultra HD monitor. The latter functions as my main screen. The former functions as my second, reference screen. For example, as I'm working this problem, Safari is open on the main screen, and is where the action happens, while the KM Editor is open on the adjacent laptop screen. So, I try to be meticulous about specifying screens in my macros. I've considered undocking while troubleshooting this problem, but paradoxically, that would introduce a variable that would be absent from the operational context (and would also be tedious).

• I've created a new test macro that contains only the actions related to this problem.

• I've assigned a hot key trigger, rather than triggering it from the KM editor, to avoid the issue of whether the target app is actually active when I trigger the macro (especially given that the KM Editor is on a different screen from the browser).

• In my test macro, I created a "click on found image at absolute position" action, and it worked correctly, which proves that the image does function as a button, and that KM is capable of clicking it. Of course, I can't use absolute position routinely, because the location of the image varies slightly depending on the position of the browser window.

• I took a fresh screenshot of the image, and replaced the single step in the test macro with a "Find image on Screen" (main screen) action, with Display enabled. When I triggered the macro (from the active browser window on the main screen) it briefly highlighted the button in green, and threw no errors.

• I added a "Click on Found Image" step

... and it worked.

ARG! Normally, I am happy when things work, but the euphoria is dampened by not knowing why it works now, but not before. I will now add one action at a time to my test macro until it either breaks, or my test macro becomes my fully functioning operational macro.

I want to thank everyone who helped with this. From my perspective, it was not a waste of time. If nothing else, I now have a better understanding of variables. And clicking on found images is an action that I must use throughout this interface. Time honing that procedure is well spent.

[ccstone]

@dp_

• I've assigned a hot key trigger, rather than triggering it from the KM editor, to avoid the issue of whether the target app is actually active when I trigger the macro (especially given that the KM Editor is on a different screen from the browser).

Hey @dp_,

Never trigger found image macros from the Keyboard Maestro Editor.

In fact – the Keyboard Maestro should be hidden when testing these types of macros.

The macro you're testing CONTAINS the image you're looking for, and this can greatly interfere with testing.

So, I try to be meticulous about specifying screens in my macros.

You've tested to see if the screenshots on each screen differ?

(I don't have two differing monitors to test this.)

-Chris

[dp_]

I'm ignorant. How do I tell the difference between two images that are apparently identical visually?

[ccstone]

@dp_

How do I tell the difference between two images that are apparently identical visually?

Hey @dp_,

Take your screenshot holding down the Control key to copy it to the Clipboard.

Open Preview.app.

Type ⌘N to open the Clipboard image in a new document.

Type ⌘I to Get-Info on the image.

You'll get a pop-up window that looks similar to this:

There you can check the image size and DPI.

---

My paranoid self would also test screenshot files...

-Chris

[dp_]

Thank you.

[Sleepy]

Since I've been helping you I should tell you that I'm leaving soon on a little vacation so you'll need to rely on others here.

[dp_]

I think that my current crisis is resolved. Thanks very much for your help, and enjoy your vacation.

[JMichaelTX]

@dp_

This is an instance of a problem that occurs intermittently in multiple scripts. I know that KM detects the button, because the Pause Until action says that the condition is currently true. But the following click action does not work. Since I've enabled Display, it puts green shading over the button. I've adjusted fuzz until this button and no other is detected. Any ideas?

I'm not sure if you problem has been completely resolved, but in rereading your OP, it occurrs to me that you may be encountering the same KM design flaw that I did.

The design flaw that I recently found is that the current (as of KM 8.2.4) Pause Until Action has a design flaw that does NOT allow you to identify which image was found, and, even worse, does NOT always return the proper FoundImage token, WHEN you have multiple Found Image Conditions, and you want to match the first one that appears.

If you are still looking for a solution, then this macro may help:

See:

MACRO: Simulate Pause Until Image Found with While/IF Image Found

Please feel free to ask any questions about this.

[dp_]

Developments at work have pulled me off of this project for a few days, but I will explore this when I return to it. Thank you (and others) for your input. This is a remarkably helpful forum.

[dp_]

Hmmm. I've run across a variant of this problem in a different context. I created Click at Found Image macros to allow me to use the keyboard to switch between mailboxes in Mail. The macros all worked as expected, then failed a few minutes later. The image on my Desktop had changed, which subtly changed the color in the Mail navbar. I went to System Preferences > Accessibility > Display and enabled Reduce Transparency. Now the macros work again. I think that the translucency of apps in Mojave may be messing with macros that depend on found images.

[ccstone]

@dp_

Now the macros work again. I think that the translucency of apps in Mojave may be messing with macros that depend on found images.

Hey @dp_,

Unequivocally.

This is a known problem – translucency regularly changes what an image looks like – so how could it not be a problem?

Hmm... It doesn't look like we have this documented in the wiki. @peternlewis

The wiki is locked until KM v9 is release, and I've made a note to update it.

-Chris

[JMichaelTX]

@ccstone

This is a known problem – translucency regularly changes what an image looks like

Another issue, with Mojave+, is OS and/or app theme. Obviously images can appear differently in Mojave Dark mode than in Light Mode. Some apps/web pages autodetect/adapt, others don't. Chris, I wonder if @CJK's method of getting/setting Sys Pref would work with SP > General > Appearance? Or might there be a KM token for this, @peternlewis?

[ccstone]

@JMichaelTX

Chris, I wonder if @CJK's method of getting/setting Sys Pref would work with SP > General > Appearance?

Hey JM,

What setting?

-Chris

[JMichaelTX]

Chris,

As I said

@JMichaelTX

SP > General > Appearance

But I'm sure you don't have that setting since you don't have Mojave.

[ccstone]

@JMichaelTX

But I'm sure you don't have that setting since you don't have Mojave.

That's right.

tell application "System Events"
   tell appearance preferences
      set dark mode to not dark mode
   end tell
end tell

[August]

I found my way to this current thread from the linked thread below, referenced above.

@JMichaelTX

If you are still looking for a solution, then this macro may help:

See:
MACRO: Simulate Pause Until Image Found with While/IF Image Found

Please feel free to ask any questions about this.

I have been wrestling for days with a simple problem of finding a Chrome Extension Icon on my Chrome toolbar and clicking it. The macro has worked for days and now has stopped working. I think it was because I tried to use the "frontmost window" in Chrome because the macro was sometimes finding the icon in the wrong window. (I don't remember, I don't keep notes of every change, especially not ones that seem "obviously right".) I have tried everything I could think of, including putting a pause in between moving to the button and clicking on it. But I couldn't figure out what the issue was.

What I found on that other thread, the one above about "Simulate Pause Until Any of Multiple Images Are Found...", was the notation that

As of Ver 75.0.3770.142, Google Chrome has had a bug in it for ~1 yr+ that causes the KM criteria for "FrontMost Window" to fail.

THAT seems to be my problem. I had been presuming -- for accumulated HOURS of debugging -- that the Frontmost Window was the natural way of identifying where I wanted to find the image of the button icon. And yes, in the macro, the Go button would indeed go there, and yes, when I ran the macro with Display on, I could see the green highlight saying "1%" for less than 1% fuzz needed to make a match. Everything has been looking right. And it has not been working.

Knowing that there is a glitch where KBM has trouble locating the frontmost window in Chrome let me change it to "screen of frontmost window" and that works -- AS LONG AS I don't have any other Chrome windows in that screen where that icon might be visible. If that's the case, then I get orange highlights on the buttons indicating that more than one image has been found. I want the image in the frontmost window, but I can't set that, it seems, because of the bug, so we go around the circle again.

What seems to be working for now is to use "frontmost window's screen"; it seems Chrome will allow KBM to know where the frontmost window is, even if it won't share the size, or whatever the bug actually is.

For the future

What I would really love is for bugs like this to not be buried as a comment deep inside an answer to a problem that I may not have. I would love for the Wiki page on the actions that are affected to have links to a post that describes the bug. Then if it's ever fixed, just that post needs to be update even if removing the links gets overlooked.

For example, in the thread above, JM comments that KBM has a "design flaw" in the logic around setting the %FoundImage% token. There is a link to his bug report where Peter explains the design logic, JM argues his point again, Peter explains more deeply, and then Peter announces a fix that looks like it it will solve JM's issue, which JM confirms, at least for his test case.

The fact that JM linked his "design flaw" comment to the bug report is what made it possible to figure out that the problem explored in this thread has been (mostly?) solved. Thanks for that thoroughness, JM.

What I'm asking for is the same kind attention to detail whenever bugs are discovered -- in this case the bug (or "design flaw") in Chrome that limits the usefulness of the "Frontmost Window" option. For something like this, that severely impacts KBM's ability to function as designed and described, it would be very useful to have a reference in the Wiki, wherever "Frontmost Window" is defined, to the bug report that tracks the current state of this conflict with Chrome.

I'm remembering the words of a VP of Engineering that I worked for a couple of decades ago: "If it's not accurately documented, you've wasted your time building it."

[BernSh]

@dp_

Of course, I can't use absolute position routinely, because the location of the image varies slightly depending on the position of the browser window.

I’ve found locate by image too finicky and stick with absolute positioning. What makes this work is to have window positioning macros to easily and reliably position windows. This is an important part of my daily actions.

Putting a window positioning action for when an app activates in the global group gets things going from the start and having a shortcut or palette on screen to relocate if needed or put in a positioning action step within a longer sequence has served me well.