How to use OCR in Mail to save a ton of typing

The following may seem like a very specific tip, but it's actually applicable to anyone who needs to do stuff with lists of messages in Mail…it just requires some explanation to set up the macro, so that's what's first...

When a customer buys a license for one of our apps, I get an email with the order details. The Subject line looks like this:

Order: FOO123456-4321-02468 - Sam Sample - Moom Upgrade

Some customers have had licensing issues related to things beyond our control, but this means that I have to do the following multiple times a day:

  1. Open two Mail windows. In Window 1, I display the new upgrade licenses in list view, looking something like this:

  2. For each entry in Window 1, I have to type the user's name into a search box in Window 2. If there are two matches, all is good, it means the system worked.

  3. If there's only one match, I switch to an archive app and search for the same name there. If there's no match in the archive, then this is an email I need to flag for follow-up.

I automated step three pretty easily: I just created a macro with a typed string trigger. If I type "zx" in the search box in Window 2 (at the end of the name I just searched for), the macro selects the search text, copies it, and pastes it in the search box in the archiving app. That part works great.

But I wanted to automate the tough part: The typing of the names into the search box in Window 2. You can't copy anything while viewing the message list, so that was out. And I didn't want to write a macro that would have to open and find/select text in the body of each message, as that'd be slow and fussy. After some messing about, I realized I could use the mouse location as the basis for an "area" OCR to get the job done.

Here's what I wound up with, and that can be used—with customizations for your setup, of course—for getting text out of the Subject (or other visible) line in a list of messages in Mail:

Keyboard Maestro Export

I position my mouse over the hyphen between the order number and the name, then invoke the macro. The Screen Capture box captures an area around the mouse location—a box 300 pixels wide and 20 pixels high, starting at the mouse's X location, and 13 pixels up from its Y location. Why those values?

Because that's what testing revealed worked the best for my Mail app. You'll have to experiment with the values to make sure it's capturing only what you need captured; it will vary based on your Mail font size and face, spacing, etc.

After capturing the image to the clipboard, the macro OCR's the clipboard then deletes the clipboard (no need to leave bits of graphic lying around). A search via regex action trims the name out of the OCR'd text, and puts that name on the clipboard. The rest of the macro (not shown) just switches windows and pastes the name in the search field in Window 2.

If I then need to check the archive, I just type zx to run the other macro. When done, I move the mouse up to the next row, click the mouse to make Window 1 active, and run the macro again.

This has saved me so much typing and time! And the ability to use OCR on an area around the mouse location is something I'm sure I'll find more uses for in the future.

So thanks, KM, for saving my fingers once again!

-rob.

1 Like

With the "list" active and an email selected:

tell application "Mail"
   set AppleScript's text item delimiters to " - "
   tell message viewer 1
      return text item 2 of (get subject of item 1 of (get selected messages))
   end tell
end tell

(Edit: Script corrected following @griffman's post below.)

...works for me. You could use the result for the next stage of your macro, then back to this window and down-arrow ready for the next. Or select multiple emails and generate a list of names to work through in the other window.

Assuming you need to use the Find box, this should let you do the "capture" in window 1 and the Find in window 2 -- you could make it more bullet-proof by using viewer/window names rather than indexes. You may have to increase the delay to have the text field active when you set its value. Since it returns the found message count you can decide if all's good (2) and, if not, which branch of step 3 to take (1 or 0):

tell application "Mail"
    set AppleScript's text item delimiters to " - "
	tell message viewer 1
		set theSubject to text item 2 of (get subject of item 1 of (get selected messages))
	end tell
	tell application "System Events"
		tell process "Mail"
			if not (exists text field 1 of group 4 of toolbar 1 of window 2) then
				tell window 2
					click button 1 of group 4 of toolbar 1
				end tell
				delay 0.2
				set value of text field 1 of group 4 of toolbar 1 of window 3 to theSubject
			else
				set value of text field 1 of group 4 of toolbar 1 of window 2 to theSubject
			end if
		end tell
	end tell
    return count of messages of message viewer 2
end tell

Usual caveats on UI scripting apply...

I figured there was an AppleScript solution, but I knew it was beyond me, and I hadn't played much with OCR so I thought it'd be fun :). However, the script doesn't seem to work (Sonoma, KM 11.0.3), as it throws this error with a message selected:

Mail got an error: message viewer 1 doesn’t understand the “theName” message.

So I asked my LLM-of-the-day, and it said that you can't assign theName in that way. It suggested this, which works:

tell application "Mail"
	set AppleScript's text item delimiters to " - "
	tell message viewer 1
		set selectedMessage to item 1 of (get selected messages)
		set subjectText to subject of selectedMessage
		set theName to text item 2 of subjectText
	end tell
end tell
return theName

I have no idea why it fails or why it works, but it does work in the LLM-provided version :).

There are only two disadvantages that I see to AppleScript. The first is that the message has to be selected, which just means I have to click on it before running the macro (not a huge issue). The second is that it's (shockingly, to me) marginally slower than doing OCR, probably due to the overhead of AppleScript.

The AppleScript method takes ~0.17 seconds; the OCR method takes ~0.12, despite having to do a screen capture, OCR, and regex to extract the name. Wild.

Thank you for the AppleScript version—it's probably also much more reliable than my OCR method, but I had fun figuring that one out :).

-rob.

A typo on my side -- sorry about that. The line should be

      return text item 2 of subject of item 1 of (selected messages)

Well, Apple have put a shedload more engineering resources, both hard and soft, into OCR than in to AS over recent years!

TBH, the AS method was looking ahead to what comes next. If you can find a good way (using something like messages in mailbox theBox whose subject contains...) to do the search you should be able to write a mail filter script that flagged incoming messages according to your "found 2, 1, or 0 matches" rules, maybe create Reminders deep-linking the email, perhaps posting a new issue on your ticketing system...

But I'd be worried that "firstname surname" isn't a reliable UID in even a small population set and that two different "James Smith"s each submitting one email would be counted as "has 2 mails" and be ignored. So your OCR and "human overview" is probably a better way to go anyway!

Yea, my first thought was I could get the macro to do all the work. But the matching is so me-involved that it didn't make sense to try. Not only is name a horrid matching candidate, but email isn't much better, as the app has been on the market for 12+ years, and many people have used different emails to purchase various versions.

So then I have to go to a third level of search, which involves yet another system (or two) ... so yea, I decided that simply removing the need to type names out over and over again was where I'd save the most time.

-rob.

The edit since you pointed out my error -- just in case anyone else stumbles upon this.

Biggest bang for the least buck-ing about? Sounds good to me!

Hehe, yea, just figured that out when I looked back at the macro in KM :). However, now it generates a new error:

Mail got an error: Can’t make item 1 of selected messages of message viewer 1 into type specifier.

And see, this is why I don't do AppleScript very well :).

-rob.

Me neither, obviously!

But I'm bull-headed enough to keep poking until things work, knowing that it's often because I haven't dereferenced(?) something. Lists are great, but I do seem to always get the "item of aList" rather than the "contents of the item of aList". So:

return text item 2 of (get subject of item 1 of (get selected messages))

and in the other script:

set theSubject to text item 2 of (get subject of item 1 of (get selected messages))

This is what happens when you're working on one machine and posting from another...

Bingo, that one works!

thanks;
-rob.