I want to click each of the three console buttons in the debugger of IntelliJ. There is no inbuilt shortcut and the click at found image within Keyboard Maestro was not working.
Based on this I have constructed the following AppleScript with the help of ChatGPT but am getting no luck
tell application "System Events"
tell process "WebStorm-EAP"
-- Wait until the application is ready
repeat until exists (windows)
delay 0.5
end repeat
-- Variable to store the found element
set theElement to missing value
-- Iterate through each window
repeat with theWindow in windows
-- Try to find the target element in the current window
try
-- Get all elements that could potentially be the target
set allElements to UI elements of theWindow
-- Iterate through all elements to find the target
repeat with anElement in allElements
-- Check for elements with a title or description that matches
if value of anElement is "Debugger Console" or label of anElement is "Debugger Console" then
set theElement to anElement
exit repeat
end if
end repeat
if theElement is not missing value then
exit repeat
end if
on error
-- Ignoring errors to continue searching in other windows
end try
end repeat
-- Check if the element was found and perform action
if theElement is not missing value then
click theElement
else
display alert "Element not found."
end if
end tell
end tell
If you are unable to solve it using any of the methods above, I'm pretty sure the macro that I uploaded a few days ago called MoveMouseToWord would work. So far, nobody has even commented on that macro.
Josh_P,
you cannot use Applescript or keyboard maestro UI control action (eg Press button action ) to manipulate button in Webstorm because Webstorm is not a native Mac app as it is written in Java using its own UI library (Swing).
you need to rely on Click on Found Image action and make it work somehow.
No luck to this solution, the macro crashes and stops working completely.
Not sure how to view error messages in Keyboard Maestro or the likes. But I can't activate this macro after attempting to find the button. For it to reload I have to turn the macro on and off for about a minute...
The UI browser you mentioned looks interesting however, the one that has been discontinued, but it keeps crashing when trying to select the element I am after as well?... This happens on two different individual machines so I have a feeling it's Webstorm that is being difficult here?... as it does work on native apps like text edit.
@noisneil
Also didn't work, though looks helpful. I tried using the "Press a Button" action but it only found a certain subset of buttons, of which didn't include the one I was after... looks like it scans the menu bar in this case
@Airy
How does your one work, I downloaded it but it isn't an action? Confused on how to set it up
@macdevign_mac
I normally use the Click on Found Image action but it wasn't working for this usecase, I tried with different themes on the IDE and different blurriness. But the macro wasn't very stable and would work only some of the time... So was looking for a better solution. (probably doesn't help that the text is so small in the IDE).
What do you mean you can't use AppleScript? It does work doesn't it since the accessibility inspector can find the buttons? and the UI Browser mentioned in the first comment can see different elements?
Surely most apps aren’t native (considering electron apps are popular). Generally perplexed why it’s so hard to click a button haha,
What you find is the Window ( and its standard buttons like close eg), you will not be able to find button (eg Debugger Console control ) and other UI Controls that specific to the application using Applescript or UI Browser since the app is not built using Mac UI library.
I'm confused because the window just has alot of buttons and static text and other elements.
I find that the button I want is within this tab block but then it just shows this, and will often crash when trying to dig any further. Is this what you meant? Its just because the application is written in Java I am not able to access anything here? using Keyboard Maestro.
Because Window and Menubar, and controls that make up of those are standard items for all application (regardless if it is written in Mac UI library or other UI library), which is why they show up. Other than that, you will not be able to find other UI controls specific to the application like the one you trying to click on.
My solution is a macro, not an action. I think if you downloaded it you will find it in a group called OCR, and the macro name is MoveMouseToWord.
As my macro documentation says, it could take 5 to 10 seconds to find the "word" the first time it runs, but it should be quick after that.
What you do is create an "Execute Subroutine" action in your macro like this:
In this example my macro will search for the word "Keyboard". The number "20" tells the macro the estimated size of the font your word will be (accuracy isn't important, so just use 20 for now.) If it worked, it will store the value "0" in "LocalResult." If it didn't work, it will return "2", if it's not sure whether it worked, it will return "1".
The first time you run it, it will probably take about 5 seconds if your Mac has an M1 chip (less than 5 seconds for faster Macs.) After the first run, it will be almost instantaneous, unless the target word changes location, in which it will take another 5 seconds to find the word again.
One of my common uses of this macro was to run it like this:
Remember, my macro won't be a popular method among people who demand perfectly reliable solutions. My macro was designed to solve really difficult problems, such as when the string "Click to Continue" was written in a humongous, stylized font in the middle of the screen and not in a straight line, such as what happens in some computer games. That's an example of a difficult problem that my macro was deigned to solve.
The click action isn't working. It does move to the correct text, so the OCR works well. So can imagine this being the solution so appreciate your responses so far.
What you find is the Window ( and its standard buttons like close eg), you will not be able to find button (eg Debugger Console control ) and other UI Controls that specific to the application using Applescript or UI Browser since the app is not built using Mac UI library.
I'm not sure about this because when looking at that UI Browser before, it does show the buttons that correlated to the actual buttons in Webstorm. Although I'm not sure if the button numbers change, say button 39 is the run button in the run tool window. I wonder if it changes if I open another tool window such as project...
So confused what you mean by I wont be able to find a button when the inspector says otherwise?
Also the scan button worked but it looks like it just tries on the menu bar to read the webstorm actions
I'm glad that the MoveMouseToWord macro worked for you. I am surprised DBL CLICK didn't work for you. I'll give it a try myself. I'll let you know shortly.
I may have the answer. Before you click on the window you have to make sure it's the active window. Otherwise all the click action does is activate the window. Can you try activating the window before the Click action?
Also, make sure that there are no other occasions of the word "Debugger Console" on your screen, including in the KM Editor, or it might click on the wrong one.
Tried it and had no luck, even when using a delay on both the mouse click and after your macro. But the mouse click actions work by themselves, so I'm wondering if it's the execution of your subroutine that is causing it... any luck on your end?
Only had one occurrence of the "Debugger Console" on my screen, and tried the activate window. All ended with the same outcome
I agree with you now. It does the same thing for me! I'm so sorry. I'm investigating. Will let you know.
EDIT: Aha! I think I found it. Hang on.
Yup, I found it. Near the top of my macro it says "Cancel just this macro" but I had to replace that with:
Return 0
Obviously I will have to replace the one I uploaded.
The problem was when I said "cancel just this macro" it actually cancelled your calling macro too! Ugh. I think I introduced that when I rewrote it from scratch to support Apple OCR.
You can make the change yourself or wait for my re-upload.
I'm not sure how to replace a macro that I uploaded, so this may take some time to figure out.
EDIT: the new v1.1 is uploaded. Sorry for making you a guinea pig.
As an aside what do you use Keyboard Maestro for? and how do you create/debug scripts? During one of the rabbit holes I followed while trying to solve my issue I encountered the UI Inspector / Accessiblity Inspector and Script Debugger 8 (for apple scripts). Do you use any of these tools or suggest anything like this.
I'll probably mark your response as the solution, just testing some further things to make sure it works as intended. If there are two matches of the same text is there a way to differentiate between them?
I use it mostly for the sheer joy of using it. Perhaps that's an evasive answer, but it's very true nonetheless.
I use only the KM Debugger. It's very useful and very reliable. But I wish it had a few more features, like tracking frequency of execution of actions.
Don't worry, I'm not proud. You don't have to mark me for anything.
Excellent question. I've pondered that issue a lot. At this time, my macro makes the assumption that any occurrence of the word that it finds is the only occurrence. In rare occasions that can be a problem, but I generally find that by carefully choosing the search string, the problem goes away. When I'm debugging my macro, sometimes my KM Editor contains multiple copies of the word I'm searching for, and it finds the wrong one. So I have to work around that by picking words that occur only once on the screen. This macro wasn't really designed to find short words like "The" on a screenful of text. It was designed to find words on the screen when there aren't many words in the screen. For example, some games have an "Exit Game" button and my macro can find that button even if the words are in 200 point font, even if the words are in some hand-drawn font! Really, the Apple OCR can read handwritten text! It's absolutely amazing! This macro is an excellent tool for trying to automate macOS games.
