The image finder has trouble finding text in a screen full of text, am I the only one that has this issue?

Questions & Suggestions
22

We don't take it personally if you call something a bug. I agreed with that word when I said "we want to replicate your bug."

Months? Wow, I sure would like to have had a chance to help you during that time. I'm willing to spend hours helping you.

Is this an Intel Mac? If so, are you using a video card? If so, which card? Are you using multiple Desktops or Spaces?

Here are the possibilities as I see them:

  • 1% chance of a font issue (eg, anti-aliasing)
  • 20% chance of a GUI issue (transparency, shadows, animation)
  • 20% chance of a problem in the code that the KM Engine used to implement the Find Image action
  • 1% chance of a bug in the KM Engine
  • 10% chance that there's a bug in the video card driver you are using, if you are using one.
  • 20% chance of a misuse of an action in your macro (eg, area restriction, action flag)

You can always switch to another approach, like the OCR or the file folder approach. OCR should be faster than Find Image (if you can reduce the area where you are looking for the image) and be much more reliable, even not being fooled by font changes.

You could also consider my offer to review a video of your "bug in action", as long as you can sanitize your code and data so that the world doesn't see anything proprietary. It's not only to avoid seeing something proprietary, it's also very helpful to write the smallest amount of code that illustrates the bug. In your case, the bug includes something on the screen, so we have to ensure that our screens contain similar data.

1 Like
23

many thanks.
6 different Intel macs (same issue across multiple computers) - some are 5,1, some are 6,1. Either High Sierra or Catalina, issue persists across all machines, they have different monitor types, different OS and different video cards. So that rules those out, I think.
And as mentioned, I tried the 'reduce transparency' function even though it's just a plain white non-transparent background and that doesn't solve.

24

I may agree with you, but they are all Intel, so we haven't ruled out Intel code as the issue. I do have both an Intel Mac Mini and an Apple Silicon Mac, which may help.

26

As a test, did you try the OCR method I suggested above? Just to see if it's more reliable?

-rob.

27

ok. I made a video of the bug and uploaded it here https://ipb.life/IMG_3344.MOV

28

Not to dismiss your original question, but if your goal is to accomplish the task your macro is setting out to do, and not prove that the found image system is broken. I would strongly suggest you just try using a different method.

I think the case here is that the "found image" system is not the best tool (by far) for what it appears you are trying to do.

If you are looking for text rendered on a screen - Try OCR, as @griffman has pointed out.

If you are looking for text in the name of a file in a filesystem - I'd try using "find" in a shell script (see my previous post), or if you prefer KM's built in actions, you could use the "for each" loop, with the "contents of folder" collection to check for the text you are looking for.

3 Likes
29

Perhaps this would be a good time to ask (actually, that would've been 20 replies ago)...

What are you trying to do with this macro? Based on the video, there's a very good chance you won't need found image or OCR, and possibly not even a complete search of a directory.

Can you explain the task you're trying to automate?

-rob.

30

I am trying to find an image on the screen, reliably, like it is supposed to be able to do. I appreciate the efforts to find a workaround, but I would like to be able to use the image finder. Is it known to be broken? Have others reported the same issue? Did anyone review the video that I was asked to make and send? Upon reviewing the video, does it seem that there is a bug?

31

It is not known to be broken. I have never seen the issue you're experiencing, nor have I heard any one else reporting it. What version of macOS and Keyboard Maestro are you on?

And we're not trying to find workarounds, we're trying to see if there's a better solution to what you're trying to do that doesn't rely on finding an image onscreen.

Your demo was with text in Finder, and if that's what you're really after, 100% there are much better ways to get it done. But if it's an actual image and not text, then yes, you'll have to use Found Image, which is known to be less reliable than methods that don't rely on matching a screenshot of a section of a screen with the actual screen.

-rob.

32

Home - The XY Problem

33

KM 11.0.2. Using either high sierra or catalina, various different machines, same issue.

I am confused, first you say it is not known to be broken then you say "known to be less reliable". Not sure if you can have it both ways. Is there a known issue?

When you buy a car, and find that one of the components doesn't work, lets say the windows won't roll down, and you take it to the garage and they ask "what are you specifically trying to do with this car".... It's not that I am trying to do any one particular thing; I want to be able to drive it anywhere I want and as with any paid product, you purchase it so that you can use it as advertised. Either it works or it doesn't. I have scores of different macros that all do different things, so giving one specific example (which I already did) is not really relevant.
"Image finding is very rarely the right solution for any problem" also contradicts with the other person who said they use it thousands of times a day and that it is very reliable!

34

I can understand that you are frustrated, especially if this has been going on for months, but people on this forum genuinely want to help you find a solution.

Please try re-explaining exactly what problem you are trying to resolve and that is NOT the same as the proposed solution you are trying to debug, please look at the bigger picture.

35

The bigger picture is extremely simple.
I want to be able to use the image finder to find images.

I am not attempting to do anything exotic, unusual or left field with it.
It is not working. I uploaded a video where it is hopefully demonstrated with perfect clarity that it is not working.
The problem I am attempting to resolve is to understand whether this is a) a known issue b) an unknown issue c) user error.
If it is a), then I will seek a workaround
if it is b) then I will await a bug fix
if it is c) then I will learn something.
That's the bigger picture. Until we have crossed that rubicon, I am not interested in workarounds, because we have not yet reached that point. I don't have time. I bought the car and I want to drive it from A to B.

36

en route to something else, but clearly the wrong route.

37

I disagree. The "XY error" is in this case a red herring and avoids the actual issue, which is to ascertain whether the software does or does not function correctly.

38

Broken does not equal less reliable. Broken would mean it didn't work at all: The car's windows never went down when you hit the "Down" button. Less reliable means they go down, but maybe only 99.5% of the time. For me, and I think others even including @Airy, finding images onscreen is not 100% reliable, not in the way that 1 + 1 = 2 is reliable. Certain images in certain situations can approach 100% reliability, but in my experience, that's not every image.

But that doesn't mean Find Image is broken. It just means you may sometimes find it just doesn't do what you want it to do.

So when I'm writing a macro, I use found images as the last solution, only approached if I can't solve the problem another way. (OCR, on the other hand, which also involves looking at the screen, is something I've found to be incredibly reliable.)

And that's why we asked what you were trying to accomplish.

As for the specifics in your video, as I stated above, I've never seen anything like that on my Macs, and I don't recall anything like that being reported here in the forums.

Watching the video, it really looks like a timing issue, like the found image is taking a really long time to run, and when you clicked the Finder window and moved it, something got released.

How many screens are on the test Mac? Does it behave differently if you use window instead of screen, which limits the search range? If you enable the debug logger, does it run differently if you're stepping through it—that is, is it more reliable? What about making the slider less fuzzy, so it may not find as many possibly matching images it has to deal with?

-rob.

39

OK that all makes sense.
I am not only trying to accomplish one thing. I have many macros and many objectives for them. I want to be able to accomplish anything that I set my mind to, with regards to finding images.
When you buy a car, you have not yet planned out all the journeys you are going to make with it. There will be some occasions on which you are driving down new roads to new destinations.
Not all of the images I wish to find, contain text.

I have some macs with 4 screens, some with 1. Issue persists on all.
I have tried various amounts of fuzziness. Error persists.
If it was finding other images, I would see green squares appearing somewhere else with a lower % - and I am not seeing those. I also get an "image not found" popup that breaks the macro. It really just cannot find the image, especially when the finder window is touching the top of the screen, which is absurd but it is what it is. Move the window by a few pixels and then it finds it. With no other changes.

Re: Screens - I just tried it with "window with title containing" instead of screen. Error persists.

Re: Debug logger - I isolated the specific step for the video and used the "go" button to demonstrate it, and the fault persists in exactly the same way as when running the macro. But I've never used the debug logger.

40

I was away from my computer for a few hours. I see others have responded to your post, mostly trying to persuade you to solve the problem a different way. I don't object to you wanting to get this to work using this method.

I might have an explanation for your problem. If I'm right, I want some applause. Here we go...

Do you remember when I said that Find Image didn't work when searching for all white squares on a chess board? This may be very relevant to your situation. That's because the image you are searching for is two things: (a) low contrast, mostly white and grey, and also (b) occurring on many locations on the screen at the same time.

I think what is happening is that at the beginning of your video, it's not actually failing, it's finding so many occurrences that it (the code KM uses to find images) just refuses to return any results. (This is probably the Find Image API which KM uses to encode this functionality, which I said had a 20% chance of being your problem.) Then when you move your window, the number of matches DROPS a lot (notice how much text disappears when you move your window.) You came to the conclusion that it started working, but actually it was finding TOO MANY results before you moved the window. By increasing the fuzziness, you are making the problem WORSE by causing even more matches. Do you see? This is the same thing that was happening to me when I was searching for white squares on a chess board. It's possible that on the part of your screen that we can't see, there may have been more occurrences of the same word at the same font size, which would back up my theory. If you had filmed the entirety of your screen, I might be able to make better conclusions.

So indeed, I have seen this sort of problem in the past, and I already mentioned above what caused it, a bland/colourless object that occurs many times in the screen. So I already had nailed the cause of your problem in my first post, a few minutes after you asked your question. Indeed, I would call it a bug, but it doesn't show up if you search for less common items than "simple grey words."

So now, how do we take steps to either prove my theory or fix your problem, while keeping Find Image working?

  • One thing we might be able to do is stop using a word as an image, and use an icon instead. I'm sure you have noticed that macOS supports custom icons for files, so maybe we can do that. If we create a custom icon with some red, green and blue, that should always work, as long as you search for the icon instead of the word.
  • Another thing is to use a larger font. This might help a bit.
  • Another thing is to have less text on your VERY busy screen when you do your searches. This may help a lot.
  • Another thing is to use an emoji character in your file name and that should let you find it with great reliability. (This is what I strongly recommend. It's simple. It's fast. It's easy.)

This whole situation reminds me of the discovery of the Van Allan radiation belts around the earth. Your Find Image action was "saturating the detector," as explained in some videos online like this one:

P.S. If you want to solve this by adding an emoji to your file name, I recommend an emoji that contains a rainbow (like the Van Allen image above?) because that contains many different colours and therefore offers the most reliability in any image search.

P.S. Upon reflection, if you LOWER the fuzz factor, it might instantly start working. That would definitely prove me right!!

41

Sadly, he said he tried various levels of fuzziness with no improvement. And he's also said that it's not necessarily text he wants to find, just that he used text for the demonstration.

I tried so set up a similar looking window as in the video, and it worked perfectly here:

image

I searched for the image "custom plates," and Keyboard Maestro reacted instantly, and highlighted the match.

So I have no idea what's going on, and can't replicate the exact issue shown in the video.

-rob.

42

Yes, I know, but he didn't say that he dropped it to the bottom of the scale.