Is it possible to select a UI element by the element's help text?
Do you mean on some specific website with some specific browser?
That could be useful ...
Currently I'm thinking of selecting design UI elements in layout mode.
In which app?
FileMaker - and if it would work in FileMaker then it'd probably work in all other macOS apps.
I'm looking to get around verbose and brittle Apple Script hierarchies that rely on UI elements.
Yes, macOS allows you to use AppleScript to get teh help text tooltips for UI elements. Just go to an AI language model and ask this question "In macOS, is there some way to programatically get the tooltip for UI elements?" You will get a sample piece of code that shows you how to get the help text for all UI elements.
The Swift programming language also allows this, and your preferred AI can give you sample Swift code showing you how to do this.
KM supports both Swift and Applescript so it should work for you.
If you have Xcode, there's a way to solve it there too. But I think you want support within the KM environment, so you will have to use one of the first two ideas.
Here are some images of some sample programs I got AI to generate for me...
1 Like
Given that FM is very Apple-ish (understandable given its developers!) then it probably won't work in "all other macOS apps".
That aside -- help text/tool tips are properties of UI elements, so you aren't getting away from that reliance. And there's no "recursive search" down the UI hierarchy -- you can do
tell application "System Events"
tell process "FileMaker Pro"
tell window 1
tell scroll area 1
every checkbox whose name is "Show grid"
end tell
end tell
end tell
end tell
--> {checkbox "Show grid" of scroll area 1 of window "Inspector" of application process "FileMaker Pro" of application "System Events"}
...but not:
tell application "System Events"
tell process "FileMaker Pro"
tell window 1
every checkbox whose name is "Show grid"
end tell
end tell
end tell
--> {}
The only way around that would be to build a "hierarchy tree" that you then search, which is going to really slow things down.
OK, so this is working in FileMaker -- it'll recursively search the UI elements of FileMaker's frontmost window and return a reference to every element whose help property contains whatever you set searchText to be in line one. So for the "Name" field in the Inspector:
...and setting the searchText to "Type a name", it returns:
{text field "Name" of scroll area 1 of window "Inspector" of application process "FileMaker Pro" of application "System Events"}
It isn't fast, but it does mean you don't have to hard-code UI element paths.
Script
set searchText to "Type a name"
tell application "FileMaker Pro" to activate
set theList to {}
tell application "System Events"
tell process "FileMaker Pro"
searchHelp(window 1, searchText, theList) of me
end tell
end tell
set outList to {}
repeat with eachItem in theList
if contents of eachItem is not {} then set outList to outList & contents of eachItem
end repeat
return outList
on searchHelp(theParent, theText, theList)
using terms from application "System Events"
repeat with eachElement in (get every UI element of theParent)
searchHelp(eachElement, theText, theList)
copy (every UI element of eachElement whose help contains theText) to end of theList
end repeat
return theList
end using terms from
end searchHelp
Thanks Nige!
Certainly works and it's slow as you predicted.
Made me realise that the Inspector is still accessible by System Events.
Cheers
Keith
Thanks for this idea @Airy
It took a few runs through with Chat GPT. I had problems with getting Swift to find elements by name so I ended up using a map.
The following example code sets the Name and Tooltip for a selected layout element.
import Cocoa
import ApplicationServices
// JSON-like dictionary mapping logical names to internal AX identifiers
let fieldMap: [String: String] = [
"Name": "_NS:47",
"Tooltip": "_NS:77"
]
// Recursive function to find an AX element by its identifier
func findElementByIdentifier(_ element: AXUIElement, targetID: String) -> AXUIElement? {
var identifier: AnyObject?
if AXUIElementCopyAttributeValue(element, kAXIdentifierAttribute as CFString, &identifier) == .success,
let identifierString = identifier as? String, identifierString == targetID {
return element
}
// Recursively search child elements
var children: AnyObject?
if AXUIElementCopyAttributeValue(element, kAXChildrenAttribute as CFString, &children) == .success,
let childElements = children as? [AXUIElement] {
for child in childElements {
if let found = findElementByIdentifier(child, targetID: targetID) {
return found
}
}
}
return nil
}
// Bring FileMaker Pro to the front
func bringFileMakerToFront() {
guard let app = NSRunningApplication.runningApplications(withBundleIdentifier: "com.filemaker.client.pro12").first else {
print("❌ FileMaker Pro is not running.")
return
}
app.activate() // ✅ Fix: No more deprecated `.activateIgnoringOtherApps`
print("✅ Brought FileMaker Pro to the front.")
}
// Assign a value to a text field
func assignValue(to fieldName: String, value: String) {
guard let app = NSRunningApplication.runningApplications(withBundleIdentifier: "com.filemaker.client.pro12").first else {
print("❌ FileMaker Pro is not running.")
return
}
let appElement = AXUIElementCreateApplication(app.processIdentifier)
// Get the first window
var windowList: AnyObject?
let result = AXUIElementCopyAttributeValue(appElement, kAXWindowsAttribute as CFString, &windowList)
guard result == .success, let windows = windowList as? [AXUIElement], let firstWindow = windows.first else {
print("❌ Could not find FileMaker window.")
return
}
guard let targetID = fieldMap[fieldName] else {
print("❌ No mapping found for field: \(fieldName)")
return
}
print("🔍 Searching for field with ID: \(targetID)...")
// Search recursively for the UI element
if let targetElement = findElementByIdentifier(firstWindow, targetID: targetID) {
print("✅ Found UI element for \(fieldName) (ID: \(targetID)). Attempting to interact...")
// Bring FileMaker to the front
bringFileMakerToFront()
// ✅ FIX: Use `kAXFocusedAttribute` instead of `kAXPressAction`
let focused = true as CFBoolean
if AXUIElementSetAttributeValue(targetElement, kAXFocusedAttribute as CFString, focused) == .success {
print("✅ Focused on \(fieldName).")
// ✅ FIX: Set value using `kAXValueAttribute`
let cfValue = value as CFString
if AXUIElementSetAttributeValue(targetElement, kAXValueAttribute as CFString, cfValue) == .success {
print("✅ Set value: \(value)")
// ✅ FIX: Simulate "Enter" to confirm input
let enterKey = CGEvent(keyboardEventSource: nil, virtualKey: 0x24, keyDown: true) // Enter key
enterKey?.post(tap: .cghidEventTap)
print("🔄 Pressed Enter to confirm the value.")
} else {
print("❌ Failed to set value.")
}
// ✅ FIX: Simulate pressing "Tab" to exit the field
let tabKey = CGEvent(keyboardEventSource: nil, virtualKey: 0x30, keyDown: true) // Tab key
tabKey?.post(tap: .cghidEventTap)
print("🔄 Tapped out of the field.")
return
} else {
print("❌ Failed to focus \(fieldName).")
}
} else {
print("❌ No matching UI element found for \(fieldName).")
}
}
// Test assignment
assignValue(to: "Name", value: "idField")
assignValue(to: "Tooltip", value: "Developer.Tooltip( Self )")
Cheers
Thanks once again Nige.
I change the code to look for elements by name, looked for 'Name', and found elements on the layout.
Made me realise I can select items on the layout via code.
Cheers