How to click on two or more identical images on the screen?

[Airy]

@Willis

Let me say I’m sorry before I keep going. I tried following the instructions you gave me. But I must have misunderstood something or messed up some basic stuff because I still couldn’t get it to work

No worries. I feel bad that I didn't explain it clearly. I see a couple of mistakes in your latest macro. In this case, I will show you a screenshot of exactly what you need to do, instead of giving verbal instructions indicating how to fix it. Note "each setting in each action" that I'm using.

Notice that this time (to help you debug & understand it) I replaced your simulated keystrokes (and your click) with a simple Move action and a Highlight action which will help you to understand what the macro is doing. You don't really have to do that if you don't want to, but I think it will be fun and enlightening to see it work this way.

P.S. I omitted the Assert action for this moment. We can put it in later, before the For action.

[Willis]

@Airy

No worries. I feel bad that I didn't explain it clearly. I see a couple of mistakes in your latest macro. In this case, I will show you a screenshot of exactly what you need to do, instead of giving verbal instructions indicating how to fix it.

Thank you for being so patient with me!

Yup! Finally got it working! I’m so happy hahaha.

@Airy

Highlight action which will help you to understand what the macro is doing. You don't really have to do that if you don't want to, but I think it will be fun and enlightening to see it work this way.

It certainly is fun to see how it highlights the images! This also gives me an idea to highlight the area that I selected first, which helps me visually see that I have selected the correct area before triggering the next macro! But ehhh..if I may ask one final question about this before we call it a day…

How to get the highlight action to show up only after I press both Page Up and Page Down? Right now, it would temporarily highlight an “incorrect area” after I pressed Page Up and before I press Page Down (this happens because it retained the previous LowerRight coordinate). If this can’t be fixed easily, then I don’t want to bother you! This isn’t really a real problem per se.

CleanShot 2025-10-13 at 18.28.35@2x1812×586 100 KB

[Willis]

I know I said that’s the last question I was going to ask, but I just ran into another potential problem that I can’t fix by myself. Sorry!

There are some variation of the Image (same icon symbol but different color). I figured I’d use a If clause to fix that! But again, not working! What did I do wrong this time??

[Nige_S]

@Willis

There are some variation of the Image (same icon symbol but different color). I figured I’d use a If clause to fix that!

Looking for that tiny image in "all screens" can easily give a false positive, putting you into the wrong part of your "If" action. And your first Collection match has lowest fuzziness possible, so might not detect anything.

Before going any further -- can you not build your Collection from both images? In the "For Each" action click the "+" button next to "The found images:" and set the area to the same as before while changing the image and fuzziness to suit the second image:

Note that the Collection is built in that order so the "For Each" will loop through all the found first images, top to bottom, then all the found second images, top to bottom -- if you've a mixture of images it doesn't work down the page processing image 1, image 2, image 1, image 1, image 2...

[Airy]

@Willis

How to get the highlight action to show up only after I press both Page Up and Page Down?

I'm not 100% certain I understand, (there's more than one way to interpret your statement) but if I do understand, the solution is simple: directly after the For Each macro, set the variables (UpperLeft and LowerRight) to an empty string. That way, your Assert action will notice that they are blank and stop your macro. And at this point you may want to turn off the Assert macro's flag "Notify on Failure," but that depend on whether you want to be notified or not that you made a mistake.

You asked another question, and that may take me more than 5 minutes to respond, so I will respond separately.

[Airy]

@Willis

There are some variation of the Image (same icon symbol but different color). I figured I’d use a If clause to fix that! But again, not working! What did I do wrong this time??

You didn't really say what "Not Working" meant. Nige gave you a good response that will probably work. That's the same response I think I would have given you. His approach is the smart approach, but if you really want to use an IF statement, at least modify the IF statement by replacing "in all screens" with "in area" the same way you built your For Each loop.

And your image is so tiny and so uniform in color that you're probably going to get a lot of false positive search results which will mess you up, especially with your low fuzz factors. But at least with your Highlight action you can easily debug whether or not your fuzz factors are working correctly (although for debugging purposes, you should replace your "Move and Click" action with "Just Move" and disable your "Type Key" actions.

[Willis]

@Nige_S

In the "For Each" action click the "+" button next to "The found images:" and set the area to the same as before while changing the image and fuzziness to suit the second image

Ok that works! I didn’t do that at first because I thought I would get an error if I had two or more “found images in the “For Each“ action and not every image was found when the macro is executed. Turns out it can sort of work like “if then else”.

Thx!

[Willis]

@Airy

I'm not 100% certain I understand, (there's more than one way to interpret your statement) but if I do understand, the solution is simple: directly after the For Each macro, set the variables (UpperLeft and LowerRight) to an empty string. That way, your Assert action will notice that they are blank and stop your macro. And at this point you may want to turn off the Assert macro's flag "Notify on Failure," but that depend on whether you want to be notified or not that you made a mistake.

Yes that was exactly what I meant. Setting variables to an empty string works! Thx!

[Willis]

@Airy

Nige gave you a good response that will probably work. That's the same response I think I would have given you

Yeah it worked great!

@Airy

And your image is so tiny and so uniform in color that you're probably going to get a lot of false positive search results which will mess you up, especially with your low fuzz factors.

Actually, it was only when I turn the fuzziness to the lowest point that I would stop getting false positives.

@Airy

But at least with your Highlight action you can easily debug whether or not your fuzz factors are working correctly (although for debugging purposes, you should replace your "Move and Click" action with "Just Move" and disable your "Type Key" actions.

Yup Yup Yup!

Thank you both so much!

[Airy]

@Willis

Thank you both so much!

Glad I could help. It's always a risk when I give verbal advice instead of uploading a macro, but for some things, like Find Image actions, I can't really upload a working macro, since I don't have your images. And I can't get the fuzz factor settings correct unless I have access to your environment. So these things need to be solved at your end, and I'm glad you solved them.

[Nige_S]

@Willis

Ok that works! I didn’t do that at first because I thought I would get an error if I had two or more “found images in the “For Each“ action and not every image was found when the macro is executed. Turns out it can sort of work like “if then else”.

No, no, and again no!

Have a read of the "For Each" action page on the Wiki. You'll see that it iterates through the items in a "Collection".

This'll be easier to show with a "Number Range" Collection. We'll use the action

First, the Collection is created -- a list of numbers from 5 to 7 inclusive. Then the action iterates through the items in the Collection, displaying 5, then 6, then 7.

Now consider this:

...which, when run, gives:

From that we can infer that:

1. The first Collection, the numbers 5 thru 7, is created
2. The action iterates through the items in the first Collection
3. A second Collection, the numbers 1 thru 4, is created
4. The action iterates through the items in the second Collection

We know it must happen like that -- if both Collections were built before any iteration began then Local_i would still be 3 and we wouldn't display 1 or 2.

(I'll admit that, until now, I had thought it made all required Collections then iterated through the lot -- so thanks for making me dig into this!)

So for your images the Engine will create a "Found Images" Collection for the first image, iterate through that, then do the same for the second image.

It doesn't "interleave" the finding of the two images -- so if you wanted to "work down the screen from top to bottom, clicking on either image1 or image2" you'd have create the first Collection and save it to a variable, create the second and save that, then combine the two, then "For Each" the combined result.

Each found image is stored as the matching rectangle followed by the fuzziness of the match -- left,top,width,height,fuzz -- and to work "from top to bottom" we need to sort by the top value. Easy with a quick shell command to "sort numerically on the second column, where columns are delimited by a comma":

sort -n -k2 -t,

Putting it all together:

[Airy]

@Nige_S

It doesn't "interleave" the finding of the two images

I knew it wouldn't do that, but I didn't know it worked as you uncovered. That's really good work, Nige.

[Willis]

@Nige_S

Have a read of the "For Each" action page on the Wiki. You'll see that it iterates through the items in a "Collection".

@Nige_S

We know it must happen like that -- if both Collections were built before any iteration began then Local_i would still be 3 and we wouldn't display 1 or 2.

This is such a neat little macro to demonstrate and prove how “For each” action actually works when there’s more than 1 collection. learnt something new again!

@Nige_S

So for your images the Engine will create a "Found Images" Collection for the first image, iterate through that, then do the same for the second image.

Ah ha Ah ha!

@Nige_S

It doesn't "interleave" the finding of the two images -- so if you wanted to "work down the screen from top to bottom, clicking on either image1 or image2" you'd have create the first Collection and save it to a variable, create the second and save that, then combine the two, then "For Each" the combined result.

@Nige_S

Easy with a quick shell command to "sort numerically on the second column,

This Shell script doohickey is making my head SPIN!

A little off topic here, but I need to learn shell script among many other things! After some research, it looks like Apple used to use Bash and now uses Zsh? People say Zsh is just an enhanced version of Bash. And most tutorials on youtube are Bash rather than Zsh. There’s apparently a lot of similarity between the two shell script languages? So if I learn from Bash tutorial, it is mostly translatable to using Zsh shell script I suppose???

[Airy]

@Willis

After some research, it looks like Apple used to use Bash and now uses Zsh?

I think that was when Apple changed the default user shell script. That's just the default. Any Apple user can still use any shell. Sometimes it's as easy as typing the new shell's name on the command prompt. In the case of the KM Execute Shell Script action, it's as simple as specifying the shell in the first line of the action. (I won't mention the syntax here.)

@Willis

People say Zsh is just an enhanced version of Bash.

All the shells seem (to me) to have all the (external) commands in common. So if you learn the commands, you can write those commands in any shell. As for the shell itself, many of the internal commands are things like "If...Else" which I don't normally need because KM can perform If/Else statements for you already.

@Willis

And most tutorials on youtube are Bash rather than Zsh.

Again, those tutorials are probably talking about the internal commands, (loops, ifs, etc.) which differ. But KM has duplicates of those commands built-in to its own language (loops, ifs, etc.)

@Willis

There’s apparently a lot of similarity between the two shell script languages?

The commands (at least the external ones) are probably identical. You can learn them and that information will be universal.

The differences, I think, tend to be in relation to how commands are string together (piping, loops, etc.) And this is the part that I prefer to use KM for.

This is my opinion. I'm sure others will differ with me.

[Airy]

@Willis

I need to learn shell script

The external shell commands you should learn (based on my experience) can be split into separate categories: (I've tried to sort them my importance, but this list is subjective)

Command usage: man, history, apropos
File management: cat, grep, head, tail, sort, sed, awk, cut, diff, tee, chmod, chown, echo
Folder management: ls, pwd, cd, mkdir, rmdir, rm, cp, mv, touch, file, find
Disk usage: df, du
Process usage: ps, kill, time, top
Internet usage: ping, traceroute, curl, nslookup

Consider this list to be a good place to start reading up on commands.

[Willis]

@Airy

I think that was when Apple changed the default user shell script. That's just the default. Any Apple user can still use any shell. Sometimes it's as easy as typing the new shell's name on the command prompt. In the case of the KM Execute Shell Script action, it's as simple as specifying the shell in the first line of the action. (I won't mention the syntax here.)

oh okok.

@Airy

many of the internal commands are things like "If...Else" which I don't normally need because KM can perform If/Else statements for you already.

Yeah I agree, I would only use a shell script to perform tasks that cannot be easily done in KM.

@Airy

The commands (at least the external ones) are probably identical. You can learn them and that information will be universal.

That’s good to know! The external ones are what I want to learn mainly! Then, like you said, use KM to do the “loops” and “if then else” stuff.

@Airy

Command usage: man, history, apropos
File management: cat, grep, head, tail, sort, sed, awk, cut, diff, tee, chmod, chown, echo
Folder management: ls, pwd, cd, mkdir, rmdir, rm, cp, mv, touch, file, find
Disk usage: df, du
Process usage: ps, kill, time, top
Internet usage: ping, traceroute, curl, nslookup

THX! This is a great starting point for me!

[Nige_S]

@Willis

A little off topic here, but I need to learn shell script among many other things!

Need? No. But it is certainly helpful, and another tool in your toolbox. But time is limited, and learning something you'll rarely use means less time for the useful things...

To @Airy's excellent summary I'd add:

While Apple's default shell in Terminal is zsh, that doesn't stop you from using other shells. The KM "Execute Shell Script" action's default shell is sh (actually bash in sh-emulation mode) -- but, again, you can shebang to use any shell you have installed.

Many people write in bash, or even sh, for maximum portability since those shells are installed by default in the vast majority of *nix distros.

Many bash tutorials and scripts you'll find on the web will assume bash version 4 or even 5 (current is 5.3) -- but macOS still ships with version 3! So watch out for that... True macOS bash devotees will install the latest version via homebrew or similar -- again, that will reduce portability so you need to mention such if uploading shell script-using macros to this Forum.

Many of the commands @Airy listed are actually built-ins and so can vary by shell -- echo is one that always catches me out Again, something to be aware of rather than write a thesis about.

Most shells have the same constructs, even if they implement them differently. zsh is based on bash so does things mainly the same except where it does them better. If I was starting from scratch and considering a future using macOS I'd go with zsh as it is the "modern default-installed shell" -- if I was using a Mac to learn how to shell script on other platforms (maybe considering a move into *nix administration) I'd grab bash v5 from homebrew and learn with that.

But for occasional macOS command line and KM shell scripting? It really doesn't matter because you only need to know what's available so you know the question to ask Just remember to include the shell in the question -- "How do I do an if in bash", "How do I loop through a range of numbers in zsh" -- in case it makes a difference.

And for understanding scripts you see here you can go the other way -- "what does the shell command "sort -n -k2 -t," do". LLMs are quite good for that, particularly if you break longer scripts into smaller chunks to see what each part does. For example (my second ever use of Copilot -- go me!):

Perhaps most important for us -- bookmark, read, and try to get your head round the "Execute a Shell Script" Wiki page. How to use KM variables, quoting strings, using paths, not having the correct entries in KM's ENV_PATH Global variable -- all "gotchas" that crop up time and again in the Forum, so understanding that page will give you a huge head start.

Sorry, that's a lot of disconnected blather -- hopefully you can pull something useful out of it!

[Airy]

@Nige_S

Many of the commands @Airy listed are actually built-ins and so can vary by shell

Oops. My bad. But as you said with your italic text, they "can" (but may not) vary by shell.

[Willis]

@Nige_S

While Apple's default shell in Terminal is zsh, that doesn't stop you from using other shells. The KM "Execute Shell Script" action's default shell is sh (actually bash in sh-emulation mode) -- but, again, you can shebang to use any shell you have installed.

Many people write in bash, or even sh, for maximum portability since those shells are installed by default in the vast majority of *nix distros.

Great, then learning bash will do just fine!

@Nige_S

Many bash tutorials and scripts you'll find on the web will assume bash version 4 or even 5 (current is 5.3) -- but macOS still ships with version 3! So watch out for that...

Will do!

@Nige_S

Many of the commands @Airy listed are actually built-ins and so can vary by shell

@Nige_S

But for occasional macOS command line and KM shell scripting? It really doesn't matter because you only need to know what's available so you know the question to ask Just remember to include the shell in the question -- "How do I do an if in bash", "How do I loop through a range of numbers in zsh" -- in case it makes a difference.

@Nige_S

Perhaps most important for us -- bookmark, read, and try to get your head round the "Execute a Shell Script" Wiki page. How to use KM variables, quoting strings, using paths, not having the correct entries in KM's ENV_PATH Global variable -- all "gotchas" that crop up time and again in the Forum, so understanding that page will give you a huge head start.

Thx!! I got a better idea of where to start and what to look out for now!

[Willis]

@Willis

I want the option to select an area on the screen first, so that it only clicks the icons within the area I selected. I vaguely remember there’s an action that captures an area on the screen, but I cannot remember what action that is. I suppose maybe I could use that action to capture the area and save it as a variable and then somehow incorporate that variable into the macro I have right now? I’m not sure! How should I go about it?

@Airy

KM does not have an action that specifically addresses this problem - perhaps it should. This would be a good thing to request. I would really like that added to KM.

Hey @Airy, I think I found that “screen-capturing” action I was talking about: Prompt for Screen Rectangle Action

Get Area.kmmacros

“Prompt for Screen Rectangle Action” Macro