Questions about "Click at Found Image", for a Special Icon in a Music Plug-In

[Nyhdroe]

Hello Everyone,

I am blind and use the screen reader VoiceOver. I have registered in this forum because I found some good posts for my work over the last days and now I have some questions about a few macros I created with Keyboard Maestro. Unfortunately, this will be a somewhat longer post. Often, apps or music plugins for VoiceOver are not accessible or only partially accessible. For such cases, I use Keyboard Maestro to create macros to make elements accessible that I cannot reach with VoiceOver.

In the current case, I have created macros in Apple Logic Pro for the Nexus 4 plugin by ReFX. This concerns the Arpeggiator section. Above the Arpeggiator Pattern is a small triangle Icon that can be moved across the entire pattern. The manual says the following about it:

Loop Region

It is possible to create an arp pattern that has a lead-in section (only plays a single time) and a loop section (loops while the note is sustained). This can be handy to create buildups and fade-ins. Above the note editor is the pattern timeline where you find a position head (right-facing triangle icon). This sets the start of a loop where the pattern returns to when the end of the pattern is reached. Drag this with the mouse left or right to position the loop marker. The loop is set from this point to the end of the pattern.

This icon is not accessible with VoiceOver, so I have created some macros to find and move the icon. It works quite reliably, but there are still sometimes problems. When I use the macro to move the icon to the right, it can sometimes happen that the mouse continues to the right more than the icon is moved. Then I have to start a macro that searches for the icon, and when it is found, the mouse is moved on it again, and I can continue moving it. However, for this to work, some variables I have used must have exactly the right entry, otherwise, it doesn't work.
If you want to test, the macros work only correct when VoiceOver is active, because there is an apple script and a jawa script.

Therefore, I would like to do it differently. I will upload the macros and 4 screenshots to explain what I mean. At the moment, I am using the action "Move or click Mouse" to move the mouse to the right or left, and the current position is written to variables. However, I would like to use the action "Click at found Image." It should be possible for Keyboard Maestro to find the icon by itself, click on it, and move it. The problem is also that the Arpeggiator can be zoomed. The more steps the Arpeggiator has, the smaller the icon is. This is shown in the uploaded images. In Image 1 and 2, the step length is set to 4, and the icon in Image 1 is all the way to the left, and in Image 2, it's all the way to the right. In Image 3, I moved the icon to the middle and set the step length to 16. The icon should be smaller. And in the fourth image, the icon is very small because I set the step length to 64. The icon is positioned so that the loop starts at step 4, shortly after the beginning.

Unfortunately, I'm not able to take a screenshot only of the icon to then load the image into the action. Is it even possible for Keyboard Maestro to find the icon, no matter how big and in which position it currently is? If yes, what would I need to do? And if I need one or more images of this icon, would someone be so kind and cut out the icons from the images?

Thank you very much for help!

Ronny

Nexus New Macros.kmmacros (50.5 KB)

[Airy]

I don't have your applications, so I can't do very much to investigate. But I'm good with Find Image in Keyboard Maestro. So I have some suggestions below.

For other people who are reading this, let me point out that your images were uploaded in the following order: 4, 1, 2, and 3. This tip may save some people a few minutes of work.

Firstly, even if I was sitting at your computer, it would be tricky to solve this, because of the variety of possible shapes and sizes of the image you are looking for. I've solved problems similar to this before, nearly identical, but that was because I was doing it on my own computer. Secondly, when you upload images to this forum, this is very helpful for us, but certain things related to the image's resolution, scale and dots per inch get modified, and I'm fairly sure that even if I did extract pixels from your images, they wouldn't work when I sent them back to you. So the best I can do is give you ideas for solving this.

Before I talk about the Find Image approach, I want to ask a question. Is there a shortcut in this app that moves the marker and your data to the left side of the window? If there is, you may not even need to use the Find Image approach. I would think hard about that approach before trying to use a Find Image approach. I don't have this application so I can't look into that.

I did spend an hour trying to figure out if the Accessibility features in macOS could help with this, but didn't find much. There are some features that can change the shapes of some buttons in certain circumstances, but it doesn't seem likely that the feature you are looking at would comply with these features of macOS.

Nevertheless, the Find Image approach is viable for this problem, especially if you follow some of my tips. But if you are blind, I'm not sure how you are going to read my tips, because my tips include images. I'm going to assume that you either have a small amount of vision or you have someone who can help you with the following advice.

Most people search for only one image at a time. But in your case, and in many cases that I try to address, there can be multiple shapes or colours that should trigger a match. In these situations, I do something like this:

Demo 1 of Finding Multiple Images Macro (v11.0.1)

Demo 1 of Finding Multiple Images.kmmacros (26 KB)

What the above macro does is move the mouse to the location of the Wifi button on the system menu bar regardless of whether the screen is in Light Mode or Dark Mode. It does this by trying to find the light image, and if that fails, it tries to find the dark image. Whichever search is successful, the variable Local1 is set to the value of the found location. So this example works with multiple images, and we will exploit that idea below.

I believe this approach could work for you, but you're likely going to need several images. By the looks of your screenshots, I'd estimate you'd need several. But it may not be as bad as you think, because I have another trick or two up my sleeve.

I notice that all the arrows in your example have some things in common. First, they all appear in a very specific region of your screen, or at least that's what it seems to me. Second, they all have a very dark area just to the left of the arrow, followed by either a light brown or a dark brown rectangular area. Third, the Find Image action has something called a Fuzz factor which allows a certainly leeway when searching for images. Based on what I see here, I believe you may be able to solve this in as few as two Find Image actions, exactly as in my example above. If not two, then probably four. I'll give you an image showing roughly what it might look like, but I cannot upload this as a macro because it is absolutely certain that my macro would not work for you due to technical issues that I mentioned earlier. So all I can say is that you can use the following images as guidance.

If you replaced the two Find Image statements in my above macro with the following data, I think you may be able to solve your problem. Please bear in mind that it may require a good hand and good vision to be able to extract the following image data from your own screen:

I am fairly sure that with careful selection of the images, this could achieve your goal of moving the mouse to whatever that icon looks like. Notice that I'm looking only for a rectangle that is only the left portion of the icon you are searching for. Nevertheless, this approach always works for me.

Please note that in this example, I want you to replace the constants (e.g., 555) by the appropriate values that represent the area where the icon can be found. This is important in your situation. You may also have to tinker with the fuzz bar in the Find Image actions to make sure they get the right results.

I hope this puts you in the right direction. I can give more advice, but let me stop here for now.

[Nyhdroe]

Thank you very much for your detailed response. I need some time to try your tips, but I would like to answer your questions first.

Unfortunately, I realized the issue with the wrong order of the images only after the post was published, and I couldn't change it. This didn't work with VoiceOver.

No, there is no shortcut that moves the marker all the way to the left or right.

I can't see anything, but I can get help, although that takes a few days.

Just to make sure I understand correctly, is it necessary to create one or more screenshots where only the arrow is visible, or am I perhaps misunderstanding something?

Thanks again!

[Airy]

It sounds like you understood most of my message correctly. I'm happy to hear that.

However I wasn't asking for screenshots of the arrow. Let me rephrase it. I was asking for you to create just a couple of tiny screen capture rectangles, perhaps 5 pixels high by 10 pixels wide, of the left edge of the arrow, which you will paste into your Keyboard Maestro actions yourself. Allow me to elaborate on that.

In my examples above, each screen capture was about 5 pixels high and 10 pixels wide. Each screen capture contained roughly a 5 by 5 black area from the area just to the left of the arrow, and then a 5 by 5 brownish area of the left edge of the arrow. In other words, each of my two image captures were basically a rectangle whose 5 by 5 left half was black, and whose 5 by 5 right half was brown. From your screenshots, the arrow that we are trying to locate seems to be visible on some pictures as a light brown, and in other pictures as a dark brown, which is why I had to use a two image approach. Usually when something on the screen can be displayed in two differing colors, it takes two Find Image searches to make sure the macro can find both colored arrows.

Furthermore, these screen captures cannot be sent to me, they have to be placed directly into your actions. If you send them to me, they will end up being resized and then become unusable to you. You will have to take the image snapshot and move them directly into your Keyboard Maestro Find Image action.

I have years of experience doing exactly this sort of thing in macros, nearly every day of the week, and so I'm very confident that this approach could work for you. But I have no experience helping blind people in a typewritten message, so I'm not at all sure if we can complete this by ourselves. I don't know how you can do a screen capture of an element on your screen if you are blind. You may need someone with moderate computer skills to assist you with this. It took me several minutes to do this myself, and even then I chose to launch a macOS application called Digital Color Meter which is found in the Utilities folder of the Applications folder. This app shows a zoomed-in area of the screen where I was trying to capture the 5 by 10 areas that I needed. It helped me to make my screen captures be as large as possible.

By the way, when I refer to the size of the area as 5 by 10, the exact numbers that you use may have to be different for you because your screen resolution may change the maximum size that you can use. The larger the area can be, while maintaining its appearance of black in the left half of the image and brown in the right half, the more likely this approach will work, because there will be fewer false positive Find Images searches.

By the way, you have answered one of my own questions. I have often wondered if I could continue with my hobby of programming if I became blind, and now I know the answer is yes. So, thanks for inspiring me.

[Nyhdroe]

Thank you once again for explaining what I would need to do that Keyboard Maestro recognizes what I'm searching for on the screen. I believe I understand that now. Unfortunately, I am quite certain that I cannot implement this because the people who might help me would likely have significant difficulties in handling my computer. I know theoretically how to copy a small area of my screen, but I would still need to move my physical mouse to do so, and I cannot accomplish this with keyboard inputs. It is impossible for me to control what and how much I have currently selected on the screen for copying.
Therefore, I have decided to continue with the approach I posted here as Macros and not use the "Find Image" actions. I believe I would have more fun and success in further developing my Macros as they are and experimenting with different variables to try to restrict the mouse cursor to almost 100% on the area where the arrow can move back and forth, as long as I work with the Macros. The advantage here is that with these Macros, I don't need to know how big the arrow currently is when I position the mouse at a specific location that always works.
Regarding setting a variable in connection with the "Execute Actions Until Conditions Met" action, I have a question, but I believe I need to explain why I am asking for you to understand.
Here is a screenshot to show you how my action is set up. Unfortunately, as I explained earlier, I can only capture the screen of the entire front window, so you see a lot of irrelevant Macro Groups on the left side.

This concerns the variable "VOOut". The "Until" action should execute three actions until the variable "VoOut" is "Group." The first action moves the mouse 1 pixel left, the second action executes an AppleScript that copies what VoiceOver last spoke into the clipboard, and the third action writes what is in the clipboard into the variable. If the word "Group" is in the variable because VoiceOver spoke it last, further actions should be executed.

A quick note: I have another Macro collection that allows me to move the mouse from pixel to pixel on the screen; otherwise, I couldn't do all of this. And when the mouse is on the arrow, which is the focus in this topic, VoiceOver speaks the word "Group." However, this is not the only case where VoiceOver speaks "Group" without anything else. Therefore, sometimes the "Until" action is stopped, and the Macro continues to execute even though the mouse is not on the arrow. This leads to the following question:
Is it possible to program something so that the "Until" action only stops when the variable "VoOut" is "Group" for the second or even the third time? I still need to test how reliably the Macros will work, but something like this could certainly help me.

Thank you once again for your detailed explanations, and I would be very pleased if you understand why I won't try the "Find Image" action. I'm also delighted that my posts could answer a question you've always had. However, even though a completely blind person can program, it works quite differently than with vision. If blindness were to occur, there would be many other things to learn anew because much of what you do now works but under entirely different conditions. I have never been able to see, so it is entirely normal for me.

[Airy]

I will assist you with your request regarding a macro, in my next message, but I have one more question first. Many programs provide a visual indicator when you move the pointer over specific symbols on the screen. Sometimes the visual indicator consists of some words that appear in a pop-up window, which is something we often call a "tooltip." If something like that happens when you move the mouse's pointer over the triangle, then that could possibly be another way to detect where the triangle is. Your assistant wouldn't necessarily have to be skilled to make this observation. Just ask them to move the mouse over the triangle and ask them if they notice something changing in the application window when they move the mouse onto, and off of, the triangle.

I have used this exact approach to solve a problem similar to yours. I had Keyboard Maestro move the mouse horizontally across the screen until a tooltip popped up with a specific word inside it. My macro detected the word and then knew the location of an icon on the screen. Since this worked for me, it may also work for you. In fact, we might not even need to use a Find Image action for this approach, since Apple has introduced an extremely fast Optical Character Recognition algorithm.

[Airy]

You asked the question, "Is it possible to program something so that the Until action only stops when a specific variable contains the word "Group" for the second or even the third time?"

Firstly, let me say that I've read your code. I see you are using a wonderfully ingenious method to detect the location of the icon on the screen. I really admire that solution. I come up with a lot of tricky solutions myself, but I've never thought of that method. I do use the Accessibility features in macOS to solve tricky problems, but your approach is a nice surprise for me.

Actually, a small portion of your code was cut off the bottom of the screen, so I may not have seen it all. But I suspect that I saw the majority of your code.

Now, to answer your question. You said you wanted to repeat something two or three times. Normally, that is a trivial problem to solve. but I've looked at your code, and I see a problem. The problem is that your code doesn't have any way to detect when the pointer has moved past the icon. Your code, which detects "the last spoken phrase to the clipboard," can correctly detect when it has moved onto a new icon, but I don't think your code has a way to detect when it has departed from an icon on the clipboard. But I might be able to solve this for you. We can safely assume from the images that you uploaded earlier that the triangle symbol you are trying to detect has a maximum width. And to solve the problem, we can use another trick. There are several steps to my trick. The first step is to move the pointer about 50 pixels past its current location, in order to go past the triangle. Then we save that location into a variable. Then we move the mouse over some part of the screen that triggers a new Voiceover output. Then we return the mouse to where it just was, and continue moving it until we find another occurrence of the word, "Group," using your detection method. This method should achieve what you want.

I would be happy enough to write this macro for you, but I have one more problem. Your last screenshot was truncated at the bottom, so I'm not sure what the rest of your macro did. But what I will do is take a guess, and upload my suggested solution in my next message to you, below.

Now this last paragraph of this message of mine will contain one more idea. You can completely ignore this idea. It does not improve the functionality of your program. It's just that I would have taken a different approach than you did, and I want to explain that approach in this paragraph. You can take this idea or leave it alone. I think your code can be simplified by putting the AppleScript action into its own separate macro, having it run in an infinite loop, and have it update a KM variable instead of writing to the System Clipboard. That way your code could be much shorter, and cleaner. Some people on this forum are opposed to the practice of putting any code in an infinite loop, but I do it all the time and it does work. It doesn't even burden the CPU, usually, because the KM Engine has enough delays built into itself to prevent most infinite loops from burdening the system.

[Airy]

Okay, I'm pasting my macro below which might solve your problem. I emphasize "might" because your screenshot did not include the bottom or the top of your macro, so I'm doing some guessing here. Also, I'm also guessing some other details about your system, which I won't explain here. I am posting the macro below, in a downloadable version, entitled "Find Third Triangle, Draft Version," which I have placed in a new group called "Nexus Draft."

The purpose of this macro is to let you trigger it using a trigger of your choosing, and it will move the mouse leftwards until it reaches the third triangle, or another number of your choosing. That seems to be what you want.

If you prefer that I describe the macro in English words, I'm willing to do that.

Find Third Triangle, Draft Version Macro (v11.0.1)

Find Third Triangle- Draft Version.kmmacros (8.9 KB)

[Nyhdroe]

Thank you very much for your responses. It's been a few days since I've been working on my macros, and now I'd like to respond again.
There are tooltips in the program, but not at this point.
Regarding my last question and your suggestion with the counter:
I've moved away from that method because I've modified the macro to a point where the Apple Script no longer copies the last spoken sentence from VoiceOver to the clipboard. Instead, I've found out how to create an Apple Script that directly passes what VoiceOver last spoke to the variable. This method is better because with the other script, the phrase 'Last sentence was copied to the clipboard' was repeatedly being copied to the clipboard itself. That's why I came up with the idea of using the counter, but that's no longer necessary now.
I'll upload the entire macro group instead of taking screenshots so that you can take a complete look at the code if you'd like.
If you want, I could also create a screen video showing how I work with this triangle and what I do when the mouse loses the symbol.
Nexus New Macros.kmmacros (87.5 KB)

Referring to the last paragraph of your second message, I've probably implemented the idea you had roughly, without reading it previously, as I no longer need the action with the clipboard as described above. Although I haven't put the Apple Script into a separate macro, but in my opinion, that's not necessary either.

Thank you for writing the macro; I tried it, but it didn't work right away, and at the moment, I'm not exactly sure what I would need to change for it to work. Also, I don't quite understand why you used the 'For Each' action and what it does. Perhaps I don't need this action, but I'd like to understand it. I read about what the action does in the Wiki, but unfortunately, I still didn't fully capture it. Sometimes, I need a bit more time to understand what Keyboard Maestro does with each action.

[Airy]

If that solution works for you, then I guess we are done.

The reason I had a For Each loop was because you used the phrase, "for the second or third time." The loop was intended to let you pick second or third, as you wish. But it doesn't matter now if you have a solution.