Today, the developer release of macOS Sequoia came out, and it supports iPhone Mirroring. Has anyone got it and tested if you can capture images from it, places those images into a KM macro containing a Find Image action, and check that the action still works? Also, it's a real longshot, but there's a 0.2% chance that the Press a Button action could return the names of buttons and let you press them on the iOS screen. Are there any other KM actions that should be tested?
I notice the app's window has rounded corners at the bottom. If you limit a KM action's search to within the window, I'm guessing it includes the rounded bottom corners that are outside the window. How does KM handle windows that aren't perfectly rectangular?
In order to do this test you have to be outside the EU, as the app is blocked there.
And you have to either have a development phone, or are willing to put your real phone on a beta OS. As of now, I haven't been that brave, though I'd really like to test the feature :).
Hi @Airy
I could test that with a «Click and found image» action. KM did find the image (an iPhone icon) on the virtual iPhone and put the cursor on that icon but it did not click on it.
Kind regards –
Walter
1.KM is not able to finish the clicking action within the iphone mirroring app.
2.KM is able to find images within the mirroring app and move the cursor to it.
3.Any Keyboard shortcuts macros and insert text macros are not available within mirroring app.
4.The MacOS built-in voice input is not able to input content to mirroring app.
5.To input text, copy and paste with in the mirroring app, using a physical keyboard is the only way.
Personal thoughts: None of these problems seem like errors that would only occur in a beta version. There is a small chance these issues will disappear after the official release. Looking forward to a solution coming out.
I've seen this with some macOS apps too. In those cases I was able to force the click by adding another action to "repeat" the click. It could be that iPhone Mirroring requires the same trickery. I've been able to force all apps to accept my clicks, though sometimes it takes persistence and experimentation.
Do you mean the "Press Key" action doesn't work? I'm not sure if that's what you mean here.
That's very, very sad. I would need to try it myself, however, to be sure. Maybe if we resorted to AppleScript or some other utility we might be able to force it.
It's theoretically possible that an update to KM could address this issue, but I wouldn't expect the Architect to be working on this right now, because I don't think he downloads early OS Betas.
I was really hoping that this approach would essentially allow Keyboard Maestro to "run on iOS." At least it's able to read the screen, so that's good. There are some things we don't actually need iOS to do for us, since KM is running on macOS. For example, we can trigger the Send iMessage action on macOS, which means we don't really need to do that on iOS.
I would be very surprised if Peter doesn't install OS betas: Pretty much any developer has to, because we have to make sure our apps don't break in the newest OS release. But there's a big difference between downloading an OS beta and actively developing new features for the new OS.
I have no idea whether Peter is doing that or not, and if he is, no idea where (or if) trying to support iOS screens on the Mac falls in his list of things to do.
I’m not sure about that. If those issues encounter a deep-level suspension in macOS, third-party developers won’t be able to do anything about it.
It’s too good to be true. But I think each KM user will love it!
Yeah, macOS has the power to run every single iOS app easily, however there is still many iOS apps won't let users install them on a Mac. If we could use KM to control them through this mirroring app, that would be a big change in our lives. Right now, I’m looking forward to it with excitement.
Keen to see if there will be any developments on this – I was so excited when this feature was announced so I could finally automate on my iPhone. Thanks for the investigation so far.
Update:
MacOS Sequoia beta3 has been released, KM is still not able to finish the actions mentioned above within the iphone mirroring app as predicted.
Yes it is. I came back here once I found out this exciting information.Now key stroke actions with single key are available as well, but it doesn't work when you try to send combination keys such as Command+A/C/V/X with key stroke actions.
Update:
With MacOS Sequoia beta4 and iOS 18 Beta4, KM is able to finish the copy and paste actions and insert text actions within the iphone mirroring app.
CMD+A means select all when you editing text on your iOS devices.
Instead of using key stroke actions to simulate combination keys, using copy/paste actions is available now. However, iPhone waits for syncing to the macOS clipboard frequently, sometimes it doest not feel smooth.