I've been using the "click at found image" action with mixed results and I am looking for some advice on how I can improve the accuracy of my found result(s). I've noticed that I don't always have the same result if I am looking for an image on one browser vs another or on one mac vs another. I'd like to improve my baseline knowledge on how this process works.
A few questions that I have are:
What factors are used to determine a match? Is it matching pixel colors and seeing if adjacent pixels match the expected value? If so, do other factors play a role, such as how the screenshot was obtained, what computer/monitor it's being displayed on, or in what program/browser I am looking for the image in?
How do the dimensions of the image factor in? For example, if I have a full size reference image, but the images being looked at are smaller (because they're thumbnails, screen resolution is different, app is zoomed out, etc), will this have a significant effect in what is being matched?
I would bet my bottom dollar that KM doesn't really have anything to do with the search mechanism, because KM probably calls an Apple API to do the search. Perhaps a better quesiton would be where is the documentation for this API. I'd like to read that too.
You are better off just using screenshots from the same monitor that you want to do the search in. And if you change your screen's resolution, all bets are off. However I have found that if you change things like "Accessibility Colour Filters" this does NOT change the search results, because the API sees only the screen before the colour filters are applied. I was hoping to take advantage of this idea, but it doesn't work.
Yes, it certainly does matter which browser you are in, because each browser has its own zoom level. If you so much as even modify the zoom level in the same browser, it won't work at all.
I think I've answered most of your questions. Find Image is very sensitive to changes on the monitor. Maybe someday Apple will introduce a new image search API that can find images regardless of scaling, but until then, we are stuck with what we have.
And even with its limitations, Find Image is astoundingly fast. It has changed my life using my computer. I still don't understand the math behind how it works, but I think there's a branch of math called Discrete Fourier Transformations that may be behind the wizardry.