Easy UI Interaction with Swift

[noisneil]

Here's a version of my UI Browser Lite macro that uses Swift instead of AppleScript, as it's very handy for interacting with UI Elements whose position in an element tree may change.

It uses a Keyboard Maestro plugin to run the Swift scripts as this is much faster than running a verbose script in KM's dedicated Swift script action.

---

Downloads

1. Here's the plugin. Leave it zipped and drag it onto the Keyboard Maestro icon in the Dock. It will then appear in your actions list.

Run Swift Snippet.zip (101.5 KB)

2. Use this macro to generate, test and copy Swift Snippets:

UI Browser Lite - Swift.kmmacros (150 KB)

Macro screenshot

---

Usage

1. Hover your mouse pointer over the target UI element and trigger the macro.
2. Choose what you want to do with the element using the yellow buttons.
3. You can test the script (with an optional timer) or just copy it for use.
4. Add an instance of the plugin to a macro and paste the copied Swift Snippet right into it.

[noisneil]

For anyone following along, I've simplified installation and use quite a bit. The precompiled binary is now bundled as part of a dedicated plugin, which negates the need for subroutine to run the generated scripts.

[noisneil]

Update: modifier-clicks added.

[fap]

Nice. I tested on a normal app and it works great.

On the menu-bar most of the attempts seem to fail. For example the Keyboard Meastro Engine button return this JSON:

{
  "appName": "Keyboard Maestro Engine",
  "action": "press",
  "path": [
    {
      "role": "AXMenuBar"
    },
    {
      "role": "AXMenuBarItem",
      "subrole": "AXMenuExtra"
    }
  ],
  "bundleID": "com.stairways.keyboardmaestro.engine"
}

and then when running the test I get the error:

Plug In failed with script error: UI element not found. The interface may have changed. Macro “UI Browser Lite - Swift” cancelled (while executing Run Swift Snippet).

Visually nothing happened but perhaps it changed its state to run a macro and that then changes the return JSON

[noisneil]

Good catch! I think it should work now.

1. Delete the Run Swift Snippet folder from:

~/Library/Application Support/Keyboard Maestro/Keyboard Maestro Actions

2. Download the plugin again from the first post and drag the new version to the Keyboard Maestro dock icon.

See how you get on!

[fap]

Confirmed. It works now. Thanks.

[Project]

Maybe I misunderstood the macro usage, I thought I could use it to create macros that make the mouse clicks on some UI elements, but it doesn't work for me. I tried with a tested script that clicked on the share button in the KM application. The test worked but the macro does nothing when triggered

[noisneil]

Please post your macro or at least a screenshot of it. If the test worked, it's likely there's something amiss with the macro rather than the tool. If that's the case, I'll try to explain how to use it more clearly in the instructions.

[Project]

Thanks a lot for your reply. It seems to work now. There was an error caused by my clipboard manager. So I ended up pasting the wrong code.
I use Raycast as my clipboard manager and it seems that whenever I copy the code it ends up as the previous to last entry instead of the very last entry. So, what i was testing in Keyboard Maestro was actually coming from the UI element of another app

[noisneil]

Happens to the best of us. Glad to hear it's working!

[noisneil]

Update:

• Value condition row added, to check state of checkboxes and other UI elements for conditional clicking.

• Now checks if app is already frontmost before bringing it to the front, and then continues once it is, rather than using a 150ms static sleep. TL;DR: consecutive instances of the plugin run much quicker!

[noisneil]

Cosmetic update:

• Value condition made more discreet and only shown for toggles/checkboxes.
• Buttons whose functions will fail for the target element are greyed out.

[noisneil]

Update:

• Added "Wait until element exists". There's no timeout, as this is configurable from the plugin action's gear menu.
• Added "Move Mouse" which will move the mouse to the target element.
• Added optional mouse position recall for clicks.
• Removed conditional clicks. Instead, the Set Value button now intelligently chooses values for checkboxes/toggles/dropdowns.
  • Select the desired value via the dropdown/field below the Set Value button.
  • When the plugin runs the Swift Snippet, it will set this value if it isn't already chosen.

[Nige_S]

@noisneil

The precompiled binary is now bundled as part of a dedicated plugin, which negates the need for subroutine to run the generated scripts.

But will require anyone still on Intel to compile their own version.

Admittedly a rapidly-shrinking proportion of the user-base -- I suspect I'll soon be last man standing!

[griffman]

@Nige_S

Admittedly a rapidly-shrinking proportion of the user-base -- I suspect I'll soon be last man standing!

Hey, I still have an Intel Core 9 27" iMac…of course, it's in its box in the storage room, waiting for me to figure out what to do with it

-rob.

[ALYB]

My main Mac is an Intel Core i7! I still love it.

[Project]

How do I upgrade to the latest version of the plugin? If I try to drag it on the keyboard maestro icon in the dock, keyboard maestro tells me that the action already exist. Do I have to delete the old version first? Of course I'm talking of updating the plugin, updating the macro means just substituting the old version with the new one.

[Nige_S]

@noisneil

Downloads

1. Here's the plugin. Leave it zipped...

Worth noting for Safari users that either:

• The file must be downloaded via right-click and "Download linked file"
  ...or:
• "Open 'safe' files after downloading" must be turned off in Safari's Settings

...otherwise the file will be automatically unzipped.

Similar may be true for other browsers.

[noisneil]

Just to add to what @Nige_S said...

Delete the old version at ~/Library/Application Support/Keyboard Maestro/Keyboard Maestro Actions, and then drag the latest downloaded .zip file to the KM dock icon.

[Project]

Thanks a lot, it worked! I had to authorize again the UX app, but this happened the first time too