The option to use "Apple Text Recognition" OCR might be my reason to upgrade to KM11. However, I'm having a hard time to get this to work (better than the image matching I was using previously) and could use your help. I'm trying to automate logging in to a service from work, which happens in 3 different steps/windows (username, password, TOTP code) in a specific macOS Application.
Currently I have these questions (though more might follow):
How can I find out the exact window title of this Application, so that I can configure the OCR step to only OCR that specific window?
The OCR result seems to be a list of strings separated by newlines. In the last two steps I want to match 3 strings (company name, user name, and either "Enter password" or "Enter code"). How can I put this in a regex without specifying all 6 permutations of those 3 strings?
I’d rather not share the Application name, but if someone answers my generic question (how to discover window titles) I think/hope that I should be able to manage that part.
The question about the regex is also rather generic (I thought), but these are examples of the 3 strings that I want to match with 1 OCR result (check that all 3 strings are present in the newline separated list):
EDIT: Alternative thought: would using 3 separate OCR (simple) "contains" conditions (one per item in a set above) in 1 "Pause Until Conditions are Met" action be good enough? (Or far less efficient, because text has to be OCR'd 3 times?)
EDIT 2: Using 3 "contains" conditions is indeed a bit slow, but that does seem to work
As for the part of your questions where you want to determine the text of the window using OCR, the following works for me: (you may have to change the number "24" to a different number, because this number depends on certain macOS settings such as your screen's resolution, and you haven't provided those details.)
As for the part of your questions where you want to determine if the OCR result doesn't match three conditions, you could do the following, which will cancel the macro if your OCR result doesn't match your three conditions. I think that's what you want to do, right?
Just perform an OCR of the front window, store the results in MyOCRresult, then do the following command:
In one of your subsequent posts you explained you wanted to "wait until three strings appeared in the window." You didn't mention that in your original question. I'm sure you can adapt my solution to meet that requirement, but if you can't, let me know.
Sorry, looks like I'm really bad at expressing myself (today?).
I want(ed) to use "window with title" (with a hardcoded window title) instead of "the front window" in an OCR condition (to make sure my credentials are only automatically entered in the correct Application), but the main window of this Application does not seem to have a title, so I'm still using "the front window" (which works).
No, I want to wait until a new window appears that contains all 3 strings, so (as I wrote in my EDIT's above) I'm now using 3 OCR/contains conditions in 1 "pause until" action. This works, but I assume this is less efficient, because the window might be OCR'd 3 times instead of only once?
Since I'm not good at expressing myself, maybe this picture helps?
Your method of using three OCR conditions in a single PAUSE UNTIL action is definitely three times slower than using my approach of doing a single OCR, saving the value into MyOCRresult, and then doing an evaluation similar to the one I showed. I would never do it your way, I would always save the result into a variable and then conduct three tests on the variable. E.g.,
Yes I know the order (by using a macro that OCR’d the screen and displays the result). I was thinking of using all combinations in case the OCR result is not deterministic, but maybe I’m just complicating things and should try this single order first and see whether it (ever) breaks.
Thanks for the regex. That flag is something I did not know yet either.
A little bit disappointing (and Apple's to blame?): "Sign in" was consistently properly detected on the internal (Retina) screen of my MacBook Pro, but on my external (non-Retina) Dell monitor Apple always detects it as "Sian in"...
(I can fix that when reverting to regular expressions, but still a bit disappointing)