After reading the post by @Airy about Sidecar vs Universal Control, I started to wonder if I could do iPad automation via Universal Control.
I’ve tried a bunch of Move or Click actions. I used the Get button to record various positions from iPad. That seemed to work ok, recording different locations for spots chosen on iPad.
When I execute the macro, the mouse only moves as far as the edge of the closest non-iPad screen. So with iPad located below my laptop screen, the Move action only goes to the bottom of the screen then stops. Here’s the cursor stuck in my dock (via Highlight Target action):
I tried locating iPad in various spots. With iPad to the eft of my external monitor, the Move action goes to the far left of the external monitor and stops.
I guess this all points to not being able to click on iPad via Universal Control, but I figured it was worth asking. As Airy mentioned in the other post, it would open up a lot of automation options.
I haven't tested this myself yet. When I got my new M3 iMac, I got absorbed with it.
I'm not yet sure if your idea will work, but there are some oddities about Universal Control with an iPad. Although you can move the macOS mouse onto the iPad, the System Settings Display window shows a physical space between the Mac and the iPad when you click on the Arrange button. Therefore, the macOS API may be treating that display differently from a normal display. If that's the case, then KM's actions may not work properly, if at all, on the iPad. But it's a wonderful idea and I'd love to try it myself. You tested only the Move Mouse action, but there may be some actions that work correctly.
I forgot to test this, until today. You are right, the mouse pointer on the iPad is a second simultaneous mouse with its own independent position from the mouse on the regular screen(s). The pointer on the iPad is completely inaccessible to KM (as far as I can tell.) Screen Capture cannot see the iPad, so it's probably safe to assume Find Image won't see it either.
But there is one exciting development. At WWDC 2024, one of the new features on macOS Sequoia is called iPhone Mirroring, and it will allow you to see your iPhone's screen on your Mac, and you can even press your iPhone buttons with the mouse/trackpad and use your Mac keyboard to type on the iPhone. I strongly suspect that this will be fully compatible with Keyboard Maestro's Screen and Keyboard actions (but probably no triggers will work with iPhone.) I think the beta for Sequoia comes out in about a month.
I have no idea why Apple is limiting this to iPhones' iOS and not including iPadsOS. But even so, this is VERY EXCITING, as it means we can use some KM actions on iOS devices.
Hmm, the first person who can come up with a way for KM to trigger on an event from an iPhone should win a prize. I'm currently thinking along the line of having iOS write a file to an iCloud folder which KM probably can trigger on. That may work.
Screen Capture can certainly see the iPad, at least on my Mac (M1 Pro MBP with macOS 14.5; iPad Air M2 with iPadOS 17.5.1). I can't prove it with a screenshot, but here's a photo of my iPad with a 'capture region' action ready to go:
The room's a bit dark, but you can see the edge of the iPad and the Sidecar controller icons.
EDIT: Found Image works fine, too—I asked it to find the word "Downloads" on any screen (then display the coordinates) as that's what my Finder window on the iPad was showing, and it did so:
Rob, you are talking about Sidecar. I was talking specifically about using screen capture to try to capture the iOS/iPadOS screen using Universal Control. These are completely different ways to use an iPad. For more info:
Th original post in this thread also specifically referred to Universal Control.
No problem. You will always have my respect for your greatness. If I could bow to you, I would. I can't find an emoji here for bow or respect, but if there was I would use it here for you.
