When I use Pause Until with Image recognition and use display I get multiple boxes on the screen. The match is a green box but there are multiple orange boxes (presumably similiar but not exact match). I'd image KM is scanning the entire screen for the image match. How do Is there a best practice for doing image recognition that's fast and consistent?
Orange boxes? That's news to me. Green boxes are for exact or similar matches, with a percentage usually displayed in the upper left corner of the green box. I'm very intrigued by the observation that it's showing orange boxes. Is there a macro you can upload that I can use to see orange boxes?
So I think I have it narrowed down on the orange boxes.
-
I had the pause until on all screens not the main screen. So it might be picking up similiar images on one of my other two screens.
-
the fuzzy slider seems to really be the cause. the more I slide it to the right the more orange boxes I get. I'm assuming the fuzzy slider more to the left is more an exact match and more the right is less?
like below:
with a match image of
Yes, that's right.
That's interesting. I have two monitors, and now I must test that.
EDIT: On my second screen I still get green boxes, not orange. So I'm still intrigued regarding how to get what you're getting.
Best practice for image recognition is always to avoid it, if you can.
( Processing-intensive by nature, and inevitably a bit fragile )
If the element which you need to interact with is on a web page, then there will probably be snappier and more reliable ways of selecting or clicking it.
3 Likes
Yes, and by avoiding the use of images in your macros, your "KM data file" will be much smaller, and since auto-save is always working after every change, then using something other than Images will speed up the KM Editor.
1 Like
That looks like an enormous image that you are trying to match ā image recognition is expensive and fragile even with tiny bit-search patterns.
Perhaps worth standing back from the XY problem
to look at the problem which you are actually trying to solve ...
You are trying to achieve something on a web page ?
1 Like
This only holds true if you embed the image in an image well. You can keep the image in an external file instead, which solves this problem.
I think this statement is too absolute. I have plenty of macros that work just fine using image recognition. Running image recognition constantly is probably not a good idea, but if it's in response to, say, pressing some hotkey, it can work just fine.
And as for being fragile, this can definitely be true, but I have some Final Cut macros that I wrote years ago that still work fine.
Here's something that I think is kind-of a must when using "found image" actions. Don't rely on "Failure aborts macro" to catch errors. Do this instead:
- Turn off "Failure aborts macro" in the gear menu.
- Then check the value of %ActionResult% immediately after the action, and if it's not "OK", display an error message that will help you debug the problem.
If you start out doing it this way from the very start, it will save a lot of heartache. Setup a Favorite Action, something like this:
2 Likes
Yes, but then you have to deal with image scaling. Apps like Preview will rescale the image without your permission.
@DanThomas, great suggestion, but interestingly on my system the logic does not function as expected when using the Find Image on Screen action.
Conversely, when using the Move or Click Mouse action it does work as expected.
I discovered this when attempting to apply the Try or Catch action as seen below.
Iām using:
- Keyboard Maestro 11.0.3
- Sonoma 14.6.1 (23G93)/MacBookPro18,2
@peternlewis am I missing something, i.e., is there a reason these actions behave differently? Thanks in advance.
If possible/practical, try to limit the size of the search area. e.g. instead of using Find Image in main screen, try to narrow it down to search in front window or, even better, a portion of the window or screen. There's a huge difference in CPU usage when you're searching an entire screen vs a small area.
The same goes for the image you're trying to find, as other people have pointed out.
That said, on an M1 Mac, at least, I have lots of macros that use lots of image recognition, and they work extremely well.
1 Like
What does that have to do with anything?
Your example uses Try/Catch. Perhaps that does work differently. But if you check %ActionResult% instead, after the action, it always works.
One of those threads that have lost their focus on helping the OP solve a problem, and become a chat between experts on something else ?
Perhaps scope for a split to a new thread on:
- whether,
- how, and
- when
to bypass the fragilities and resource costs of image search ?
How when and why to avoid image search ?
@DanThomas, that's just it, on my system %ActionResult% does not work. With the Find Image on Screen action, %ActionResult% is always OK.
Again, on my system I need to check local_Result. (See the third group below.)
So I don't think it's a Try or Catch issue, I think it's an unexpected behavior with Find Image on Screen action.
1 Like
If an app (like Apple's Preview) "damages" your app by changing the DPI setting (as it does) then you will never be able to find your image when you re-load it from its file. It's extra work to figure this out, and I was just warning new users about this issue,
1 Like
Oh, OK. I just put it in a KM clipboard and saved it to disk. Problem solved.
BTW, I have some new code I'll be releasing "real soon now" that takes images out of KM actions, saves them to disk, and alters the KM action to use the newly-saved file. Hopefully it'll eliminate that whole issue.
Be sure to check it out when I release it, to help me make sure I don't mess up this type of thing.
2 Likes
It is not an error for the Find Image on Screen action to not find the image. It returns and empty string as the result in that case.
So the Catch will not run if the image is not found (it could fail in other ways, such has having no image, or specifying a source that does not exist, but not simply not finding the image).
2 Likes
Okay; thanks @peternlewis.
Initially I mistakenly expected Find Image on Screen and Move or Click Mouse to behave consistently if the configured image is not found, but now I know why my testing indicated that the former was always returning an %ActionResult% of OK.
Yep, that's what I observed and check for in the green group in my test macro above.