In the beta version of Monterey macOS there is a new utility called Shortcuts. It is inevitable that this utility will be compared with Keyboard Maestro. There are some similarities, but there are also many differences. In some cases Shortcuts might be good enough. But Keyboard Maestro is a very refined product with many advanced features. However Shortcuts does have a few advanced features of its own, such as the dictionary feature which is a recent feature of KM. At the moment I cannot find any OCR feature inside Shortcuts, so I am hesitant to consider converting my macros from KM.
A little competition is probably a good thing for both products. I think KM could learn a few things from Shortcuts. For example, I think there are a few actions in Shortcuts that probably would be a good idea for KM to copy. But KM is a very mature product that has more to teach Shortcuts than the other way around.
Although I have written Automator scripts, I havenât tested that question yet. Iâm still examining hundreds of actions that are available in Shortcuts. I did find the OCR capabilities in Shortcuts. It is much faster and more reliable than the OCR capability in KM.
As I said, KM is a mature product with many refined capabilities and Shortcuts is a new product with a lot of âshortcomingsâ. However Shortcuts has a few nice tricks like the ability to control HomeKit devices.
Perhaps the best solution for KM is to provide the ability to call âshortcutsâ by name. That might give us the best of both worlds. I donât know whether Apple provides an API to allow KM to call shortcuts by name.
I've experimented with Shortcuts and I have a sample program (see attached screenshot) which isn't fully functional yet but gives you an idea as to how you might use Shortcuts to capture a screenshot of a Safari window and look for the words "Skip" or "No Thanks" and then (hopefully) to click on those buttons. This will help get rid of the ads in Youtube. I have this working in a KM macro, but this is my first attempt to get it working in a Shortcuts macro. Although this example is missing any "mouse click" action, I'm already amazed by how fast it is. It can perform OCR on the entire Safari window in less than one second, whereas when I try to read the whole screen with KM it takes about 30 seconds on my large monitor.
The main problem I'm facing in Shortcuts right now is its inability to create a loop. There are workarounds but I haven't found a workaround that I like yet.
The $64,000 Question is whether Keyboard Maestro will be able to trigger one of these Shortcuts routines. How are you triggering it to run at the moment? Does double clicking a file activate it? Or is it activated through Siri? I suppose one day there might be something like this:
You are correct that's an important question. However making that possible is more Apple's responsibility than KM's, because Apple needs to make an API to call it. However for people who accept "crazy workarounds", it's already possible for KM to call a Shortcut by doing exactly what I do now, which is clicking on the run button in the Shortcuts app. Of course that's not the "right solution" but for my own testing purposes it's adequate for now.
I haven't tested activating through Siri, but since I have activated a Shortcut through Siri in iOS, I'm sure it's going to be possible in macOS also. You asked if double clicking on a file could activate it: I don't think so because Shortcuts don't seem to be stored in files, but rather inside the Shortcuts app itself. RIght now I click on a "play button" to activate the Shortcut.
Bear in mind that some of these answers may change because Shortcuts is still only in beta.
Let me reiterate, KM is a masterful product, and Shortcuts can't (yet?) replace it. KM is particularly strong in its programming features, where Shortcuts is very weak. But both products have their areas of specialization, and I hope KM finds a way to hook into Shortcuts, like it does with AppleScript and Shell scripts. I'm sure KM's creator is trying to integrate it, but if Apple doesn't provide any API for it, he won't be able to do it.
Apple might provide an API for some of the features in Monterey like Live Text, so even if there isn't an API for Shortcuts, KM might still be able to use some of the new features of Monterey regardless. I'm extremely impressed with Live Text which seems to work about 100 times faster than the OCR in KM.
That is so exciting, Jimmy. You have made my day, week and month. I feel like itâs Christmas right now.
To me, this also strongly suggests that there will be an API for Shortcuts in Monterey. Therefore KM could include Shortcuts as some sort of built in action.
Great. Looks like it is going to work with Keyboard Maestro fine.
â â â â Developer beta 2, which was released Monday, â â â â is the âShortcuts Eventsâ process, which lets â â â â AppleScript scripts run shortcuts, including â â â â passing and receiving input.
In fact, I have already got a prototype working. That is, whenever a âSKIP ADâ pop-up button appears in a YouTube video, as detected by Shortcutsâ own OCR utility, KM will click on the button, resulting in the ad being terminated.
It is ridiculously fast and ridiculously easy. I may need to clean it up a little bit before I publish it here.
I actually had this working in a macro using KM exclusively, but that was ridiculously complicated because KMâs OCR function was very slow, and so I had to create a lot of code to identify the area that needed to be scanned by the KM OCR utility. All that code is obsolete now.
I've got it all working, but there is a performance issue I need to address before I bother to post one of my solutions here. Regardless, I'll describe in a few words one of the approaches that worked.
You will need both a Shortcut (called SHORTCUTNAME) and a KM macro for this solution.
The shortcut itself will contain three sequential statements: (a) "Take Full Screen Screenshot"; (b) Extract Text from Screenshot; (c) Combine Text from Image with New Lines.
The KM macro will contain two actions and a trigger to trigger the macro every second. The actions are: (a) A shell script which executes this string: "shortcuts run SHORTCUTNAME | cat" and save the result to a variable named LocalResult; (b) A conditional statement that will check if LocalResult contains the string "Skip", and if so, send a click to the location on the screen where the SKIP AD button would appear.
If you do these two things, you can have KM repeatedly call your Shortcut to use its superior OCR abilities and if it sees an ad, KM will terminate the ad.
It took me an hour to figure out that you need to add "| cat" to your KM macro. I can't correctly explain why, but it appears to be related to control characters in the output.
The only problem so far is that the screen capture action takes a lot of CPU and causes the video (not the audio) to stutter a lot. This makes this solution unusable at this point, but it's a great start. I tried moving the screen capture into the KM macro, but that didn't really solve the problem.
I'm not sure if there will be a solution to the video stuttering problem. I didn't have this problem when I used the OCR utility in KM. But then again I went to a lot of work in KM to search only a small part of the screen for the text.
EDIT: HOORAY! I was able to eliminate the stuttering effects.
Thank you beta testers. If many of these are developed it will nice to see them all gathered together when Monterey is released. (Of course Shortcuts' implementation may change between now and then). I can barely keep up with full releases much less betas. I have little doubt that Peter is looking at this. When Monterey and the M1 MBP 16" is released (strongly rumored), I'm in.
My goal is to see whether or not I like the new Shortcuts language. I have mixed feelings. It can do some things that KM cannot do, but it also falls short in many areas. For example it has some actions that KM doesn't like have, like HomeKit controls. But it also has pretty poor programming constructs. I get the impression that its GUI and methodology is modelled by someone who is familiar with KM, but it's still way too simple to be considered a competitor to KM. For me the best future is one where KM and Shortcuts can co-exist and work together.
Just did some light test of Dictionaries and Lists, again, all of this on iOS. And have to say the implementation of both is very intuitive, I hope KM steals these features, and more.
Text handling: it is different, it can also be transformed, i,e split, combined, etc., and saved to a Variable.
Have to wrap my head about this, not sure if I like their approach.
And have to say, it outdid KM in one area,
Shortcuts implemented a feature that Iâd like for KM: use different colors for the type of actions. A much requested feature in KM. Helps visualize actions much faster.
(Just as an alternative way to get rid to those click in Youtube: Use «Brave» as your browser, no nasty ads in Youtube anymore at least according to my observations.)
That's a fair point, but Safari has something called Reader mode which gets rid of ads that I doubt even Brave is capable of doing. I won't give up on Safari's Reader mode.
Anyway, I managed to get a macro in KM capable of calling the OCR that's built into Monterey. I'm attaching a screenshot of the Shortcuts shortcut that allows me to do this. It can be made even shorter than this, but I think some of the actions in here will be educational for readers here. Basically, this Shortcut receives the screen coordinates from your KM macro and returns the text within those coordinates using the 100x faster (and much more accurate) OCR built into Monterey.
Now the main problem with the OCR routines in both KM and Shortcuts is that they don't return the X,Y location of the text that was OCR'd, but I've solved that problem too. Maybe for a future post. I blotted out the folder name because that contained my real name. Use whatever folder you want.
