MacOS 27 has a new command called “fm”. I am eagerly waiting for this utility, but I haven’t installed macOS27 yet. This is a powerful new utility that will allow people who write KM macros to let macOS 27’s Apple Intelligence to do some of the legwork that KM would normally do. I will give an example below, but first I have a minor concern about it.
Currently, the fm command can be called from the Execute Shell Script action in KM. For many purposes, this is adequate, especially if the request is just to process some text. But another important capability of the fm command is to process images. Here’s where KM has a small problem. In order to pass an image to the fm command, it must be placed in a file. This is okay for an occasional use, but for frequent use it would be better to pass the image as a clipboard or screen coordinate (just like the Apple OCR function permits, which KM actually supports.) For example, I would like to be able to say, “Return the coordinates of the noses of every face currently on the screen.” I would rather not have to save the screen to a file and pass the file to the fm command through the Execute Shell Script action.
Now the good news is that you don’t have to pass the image as a file if you use one of the SDKs for the fm command. But the bad news is that there are SDKs for Swift, Python and one other language, but not Objective C, which I think KM was written in. So I don’t see any way for KM to be updated (like it was for OCR) to allow the fm command to have direct screen access. This is sad.
BUT in any case, the command will still be powerful and useful, even if images have to be passed as files. And most tasks probably won’t even require passing images.
There will be some overlap between KM’s capabilities, and fm’s capabilities. Both of these tools are good at processing text. For example. If someone wants to rename a bunch of files in a folder so that the filenames contains dates of the form YYYY-MM-DD instead of MM-DD-YYYY, then fm would be really useful at doing this, although fm can’t actually rename the files themselves, that would still have to be done through a macro in KM. Perhaps someday the fm command will be able to do some things, but for now KM’s ability to do things is sufficient. There will always be KM purists who want everything to be done in KM, but I like to try new things.
The fm command, like any language model, can return text to answer any English question. But processing the output in a language like KM could be messy. So to support external languages, the fm command supports schemas that users can use to define the output, and the fm command will return JSON strings that comply with the schema. This is important because a lot of questions can produce a lot of output, such as “Give me a list of all US state names and their capitals.” Schemas are pretty useful for any kind of output.
One thing that I’m extremely eager to experiment with is to take a screenshot, save it to a file, pass the file to the fm command, and ask the PCC model whether it sees any text strings in the image and return the text of each string and their corresponding position. WOW! THIS WORKS! I just tested it. In fact the local model can handle it too!!! I’m shocked and awed. The possibilities here are endless. This is a brand new chapter in KM programming. I will be able to solve problems I have never been able to solve before, or at least solve them more easily.
When I shouted “IT WORKS” what I actually did was wrote a KM script that “moved the mouse to the location of the highest mountain on the screen.” I passed the image to the PCC model which is available in Shortcuts (not the macOS fm command per se, since I don’t have macOS 27 yet.) It actually found the location of the highest mountain, in an image with multiple mountains. This is wild. And this has been available for 2 years, and I never thought of using it for fancy image processing until recently.
I’m tempted to write a KM macro to demonstrate this by allowing you to speak to your computer and move the mouse to the part of the image that you verbally describe.
