Mouse Simulation on UI Elements with Apple Script

Hi,

Is there way for Applescript to dynamically pass uielements to KM Maestro for mouse simulation such as control click, command click, drag and drop to another uielement, and so on?
Applescript has only click method, so it’d be great if we can leverage KM capability.
Thanks,

Hey James,

Many things are possible, but it’s not all that easy.

Search Google for AppleScript UI Scripting.


Best Regards,
Chris

Please correct me if I’m wrong, but it looks like UI scripting through system events doesn’t have much other than just regular click method. Also many people seem to try to find the solutions to I’m looking. That’s why I posted here if we could use KM to achieve this along with Applescript.

Hey James,

You can actually find the location of UI-Elements via System Events. From there you can click to your heart’s content with KM.

An example:

---------------------------------------------------------------------------------
# Auth: Christopher Stone
# dCre: 2014/10/05 11:02
# dMod: 2014/10/14 17:55
# Appl: Finder & System Events & Keyboard Maestro
# Task: Expand Name column to fit
# Libs: None
# Osax: None
# Tags: @Applescript, @Script, @Finder, @System_Events, @Expand, @Name, @Column
---------------------------------------------------------------------------------

tell application "Finder"
  if current view of front window ≠ list view then error "View of front Finder window is not list view!"
end tell

tell application "System Events"
  set quit delay to 0
  
  tell application process "Finder"
    # set frontmost to true
    
    tell window 1
      -------------------------------------------------------------------------
      # Strange discrepency for ~/Documents/
      -------------------------------------------------------------------------
      if image 2 of scroll area 2 of splitter group 1 exists then
        tell image 2 of scroll area 2 of splitter group 1
          set {xPosition, yPosition} to position
          set {xSize, ySize} to size
        end tell
        
      else if scroll area 2 of splitter group 1 exists then
        ----------------------------------------------------------------------
        # Toolbar & Sidebar Showing
        ----------------------------------------------------------------------
        tell image 2 of UI element 1 of row 1 of outline 1 of scroll area 2 of splitter group 1
          set {xPosition, yPosition} to position
          set {xSize, ySize} to size
        end tell
      else if scroll area 1 of splitter group 1 exists then
        ----------------------------------------------------------------------
        # Sidebar-only Hidden OR Toolbar & Sidebar Hidden
        ----------------------------------------------------------------------
        tell image 2 of UI element 1 of row 1 of outline 1 of scroll area 1 of splitter group 1
          set {xPosition, yPosition} to position
          set {xSize, ySize} to size
        end tell
        ----------------------------------------------------------------------
      end if
      
      set {xPosition, yPosition} to {xPosition + (xSize div 2), yPosition + (ySize div 2)}
      
    end tell
  end tell
end tell

tell application "Keyboard Maestro Engine"
  try
    set value of variable "X" to xPosition
  on error
    make variable with properties {name:"X", value:xPosition}
  end try
  try
    set value of variable "Y" to yPosition
  on error
    make variable with properties {name:"Y", value:yPosition}
  end try
end tell

---------------------------------------------------------------------------------

The AppleScript is part of a KM macro:

Window { Resize Name Column }
Triggered by any of the following:
The Hot Key ⌥R is pressed
Will execute the following actions:
Execute AppleScript
Path: ~/Library/Scripts/Applications/Finder/Window { Resize Name Column ; KM}.scpt
Move and Double Click
At (X,Y) from the absolute position.

It’s not real fun, but it can be done.

It’s easier if you have UI Browser from pfiddlesoft (about $55.00 US).


Best Regards,
Chris

2 Likes

Hi Chris, that’s awesome!
Thank you for the code!

Sorry I’m a newbie here and sorry again if it’s a stupid question. How do you access to system events?