Searching for a Trigger to Continuously Scan Screen for Image and Execute Action upon Detection. Any Suggestions?

[KeyFinder]

Hi all, having a bit of trouble figuring this one out. The closest I've figured out is using a periodic trigger for every 1 second with an If Then action (If image is detected, then execute command). I'm using this to speed up some app workflow so it's important that this be fast-acting.

Unfortunately periodically checking the screen every 1 second is still proving to be too slow for me, and I notice there is no option to check every hundredth of seconds.

Am I missing something, or is there a trigger option that continuously scans the screen for an image to act upon, rather than having to tell KM to scan once every 1 second? Any other (faster) approaches to this?

Thanks!

[griffman]

Under a second with a continuous scan of the screen would probably have a huge impact on your Mac's usability. One way you could try to work around this would be to duplicate the macro ten times, and then maybe it's running every 10th of a second? But I don't know how KM handles task scheduling, maybe it'd try to run them all at the same time.

And the other issue with this, of course, is that once the screen does change, you've got to figure out how to make only one macro act on the change. I don't know that that's a solvable issue.

-rob.

[Airy]

I have the highest respect for @griffman, but I have some differing opinions here. As I type this message, to help make my point, I have a "while loop" running that is permanently looking for an image on the screen, and I feel no slowdown whatsoever. Here's a picture of my CPU history over the last two minutes. You can see a bump from about 20% usage to about 60% usage when I turned the infinite loop on. Of course, your results may differ (different CPU brand, speed, etc.) But I can drag windows around the screen even while the infinite image search loop is running without any problems. (The screenshot shows the load on each of my 8 CPUs.) Of course, if you're doing heavy image processing in another app at the same time, no doubt that will suffer while this is running. But the infinite loop doesn't affect my GUI's usability.

And there's another thing to point out. KM v11 has a new option for the Pause Until action. If you look at that action, there's a checkbox there for "Reduce CPU usage." So you can put an image search into that action, and KM itself will automatically insert some pauses between each iteration after a short while, and longer pauses after a longer while. (I've tried to find a post on this website explaining the timing details, but can't find any.) I think this could be a perfect solution for you. (Well, that really depends on the way KM implements its pauses. Perhaps after an hour it inserts a one second check between each iteration. We'll need to ask The Architect.)

And there's a hybrid solution that @griffman didn't mention, which I use all the time. Instead of thinking that it's a choice between a periodic trigger and an infinite loop, you can have an infinite loop with a Pause action, perhaps a third of a second, for example, between your iterations. I do this in dozens of my macros, and if I cancel any one of them, I have them set to auto-restart every five seconds. (And I use a sempahore at the top of the macro to prevent two copies from running at the same time.)

Now let me show you the same CPU chart, but this time, I will turn on the "Reduce CPU usage" flag. Let's see the difference...

Notice that in the bottom four cores, the Performance cores, there was a 30% spike in activity about two thirds of the way across the image, which is when I turned on the loop. It lasted about 10 seconds, and then the CPU usage dropped to about 10%. It's still running at 10% as I type this, and I don't feel any problems.

But I'm confident that my "hybrid solution" above is perfect for you. I have many macros that use this approach. It beats the 1-second increment, but it doesn't run in an infinite loop without pauses.

I can explain much more, but this is long-winded enough for a first response.

[griffman]

You're right, I didn't consider an infinite loop and pause action - nice solution!

-rob.

[Airy]

Thanks. By the way, until yesterday I thought your avatar was a fighter pilot.

I thought that the blue background was the sky, the white bubbles in the sky were clouds, and the black snorkel was a seam between the side glass canopy and the top glass canopy. Like scuba divers, fighter pilots also use air breathing systems and harnesses.

[KeyFinder]

Hey! Thanks so much for this, just a couple questions as I get this set up. Would you be able to attach a screenshot of the macro setup you have within KM that runs this? I am having a little trouble finding the infinite loop trigger and/or action you are talking about, or exactly how you are setting up the "while loop" to work, and under what trigger.

I am running KM 11.0.2.

[Airy]

I'll be happy to answer your question.

Let me say first, just for clarity, the screenshots of CPU usage were just screenshots from the Activity Monitor app.

Secondly, let me say that normally any macro which contains a "Find Image action" will probably not work on anyone else's Mac, due to a variety of issues such as screen resolution. HOWEVER in this case the Find Image action is specifically designed to look for an image that is NOT on the screen, so that means I could upload a macro with Find Image and it will "work" because it is intended to fail to find anything.

Thirdly, if your goal is to see your Activity Monitor, like I had shown above, well that was all done manually. My macro did not have anything to do with setting up those windows. I did that all manually.

Nonetheless I can show you the macros that I was referring to.

FIRST:

This is a simple infinite loop macro that I used, which looks for an image of a tree on the screen, which it will never find. Note that the "CPU" option is turned off in this macro. You should use your own image of something that is not going to be on your screen. I used this approach to generate the heaviest load possible on the CPU, as you can see in my first Activity Monitor screenshot above.

SECOND:

This is a modified infinite loop macro that I used which is similar to the above, but takes advantage of KM v11's new feature to insert pauses in the loop so that the CPU becomes less stressed over time. I used this approach to generate the lighter load on my CPU, as you can see from my second Activity Monitor screenshot, above.

As you can see from my Activity Monitor screenshots above, using this flag caused a significant decrease in CPU usage. That's because the KM Engine inserts some pauses as time goes on. I don't have the details as to how long those pauses are.

THIRD:

Here's an example of the "hybrid" approach that I often use. In this case, I'm searching FOR a specific image. I want the search algorithm to do something whenever it finds the image. In this example, I produce a sound. And notice that this macro will check for the image approximately three times per second. (Because of the Pause 0.33 seconds action.) Also notice that I have set this macro to restart every second, but the extra copy will abort if a copy is already running because of the Semaphore action at the top.

Notice in this example that there is an infinite while loop, while at the same time there is a Pause statement inside the loop to reduce the load on the CPU. If the user should press a key to "cancel all KM macros", this macro will restart itself approximately one second later, but you don't have to auto-restart this macro if you won't want to.

This is the approach griffman was referring to when he said "I didn't consider an infinite loop and pause action - nice solution!"

Search for a Tree on the Screen every 0.33 seconds Macro (v11.0.1)

Search for a Tree on the Screen every 0.33 seconds.kmmacros (91 KB)

You could potentially run this action and it will beep when it finds a tree on the screen, but it will never find that tree on your screen, because you must replace that image with your own image due to screen resolution issues. You can replace the tree by an image of your own choosing, and then this macro will beep whenever your chosen image appears on your screen.

[noisneil]

A periodic image check is indeed a viable option, but one that I would consider after others have been ruled out.

Could you give a bit of context as to what you're trying to achieve more broadly? There may be a more elegant solution right under our noses.

[KeyFinder]

Thanks! I am attempting to find a way to constantly have the "Enable Patch Merging" window in Logic Pro 10.8 open at all times in the library browser window. In recent updates, they have sadly made it so that switching the selected track will 'reset' this patch merging window. There is a key command to open it back up, but the library window must be selected for the key command to go through, and unfortunately with so many patches being loaded, the library window tends to refresh or hiccup quite a bit.

Hence, my plan within KM: A periodic trigger of some kind that scans to see if the "enable patch merging" window is not present (done via image detection). If not, click on the library window and hit the key command. Repeat this in fast succession to hop over any hiccups that may shut the window again.

Sadly, periodic trigger of minimum 1 second is too long a time; I am clicking around quite fast, and quite often have to wait for a) the periodic trigger time and b) the image detection.

In short, I am trying to find a hack for Logic Pro to keep a certain toggle window open at all times, which naturally resets closed whenever a new track is selected within Logic.

[Airy]

@KeyFinder

Thanks! I am attempting to find a way to constantly have the "Enable Patch Merging" window in Logic Pro 10.8 open at all times in the library browser window.

Is there a hotkey in that app which opens the window? And what happens when you press that hotkey if the window is already open?

[KeyFinder]

There is an assignable key command to open the window, yes, which I am using within my custom KM macro.

However, as mentioned, it is unfortunately quite finicky where the window must be selected in order for the key command to register and enable patch merging. Hence, my need to incorporate a 'click' action, followed by the key command, and repeated in quick succession several times.

If the Library window is selected and the "enable patch merging" window is already open, no change occurs; the "enable patch merging" window remains open.

If the library window is not selected and the "enable patch merging" window is already open, a white flash occurs across all displays, suggesting an error (the erorr being that the command is expecting the library window to be selected; it is not in this scenario).

[Airy]

@KeyFinder

However, as mentioned, it is unfortunately quite finicky where the window must be selected in order for the key command to register and enable patch merging. Hence, my need to incorporate a 'click' action, followed by the key command, and repeated in quick succession several times.

Well then, try my macro above called "Search for a Tree on the Screen every 0.33 seconds Macro" and replace the Tink action with this action:

And follow that action with a mouse click action on ImageLocation.x, ImageLocation.y

[KeyFinder]

This isn't quite working for me. I have the library window on a separate display, but it keeps seeming to move the mouse up onto the main display instead of the separate one.

Oddly, hitting the "Go" button for the "Move and Click" action returns the correct mouse position I want.

The ⌃⌥⌘W keystroke at the end is the key command I have assigned to open "Enable Patch Merging" after the library window is selected from the "Move and Click" action above (right above the 0.2 second pause action).

[Airy]

The first problem I see is that you don't have a percentage character in the set variable action. I see you have an ampersand there. Fix that first and then we'll see what's wrong afterwards.

[KeyFinder]

Thanks for the catch. Fixed.

Issue still occurs; my mouse is moving directly up to the top left corner of my main display, when I need this to happen on one of my secondary displays. Again, clicking "Go" returns the correct position I would like the mouse to move to.

[Airy]

@KeyFinder

Issue still occurs; my mouse is moving directly up to the top left corner of my main display, when I need this to happen on one of my secondary displays. Again, clicking "Go" returns the correct position I would like the mouse to move to.

Well, now look at your click action, it says "relative to the MAIN screen's upper right corner." Could that be the issue?

[KeyFinder]

Thanks, actually it didn't seem to be, but it seems something with KM was going awry... I just hit my "Kill all macros" macro to reset everything and now the behavior for this macro is working, with one issue still.

The action still has a delay of about 2 seconds. I imagine this has to do with the periodic trigger as well as KM taking a while to detect the found image.

Any thoughts?

[Airy]

@KeyFinder

The action still has a delay of about 2 seconds. I imagine this has to do with the periodic trigger as well as KM taking a while to detect the found image.

Let me look at your action. Off the top of my head, the only thing that can cause a two second delay is the time the editor freezes the engine whenever you edit a macro. That might be your problem. Or not, let me read your macro in a few moments.

[Airy]

I suspect I know what's wrong. Your semaphore lock action has the wrong setting checked off. You are supposed to use the following settings:

[KeyFinder]

Got it, my settings still appear identical to your screenshot, but it still takes ~2 seconds to trigger.

Again, I suspect is has to do with detecting the image in time, as the evaluation status seems to keep saying [currently false] for the duration of the wait before snapping to true and enabling the action.