Avoiding Upgrade Issues with Submacros and Subroutines

[_jims]

The topic discussed below is a bit esoteric, but will likely help some users avoid a common pitfall that can occur when modifying submacros and subroutines. So after sufficient sleep and your morning cup of coffee, consider reading this tutorial.

---

Introduction

If an updated version of a submacro/subroutine is properly installed, the links from the macros that call them will be automatically shifted to the updated version of the submacro/subroutine.

In contrast, if the updated version is installed before the older version is deleted, the caller links will not be shifted.

Let's discuss this possible disassociation and propose some safeguards.

Submacros and Subroutines

In the Keyboard Maestro wiki, submacros are referred to as traditional subroutines. Here's a excerpt from the wiki:

In Keyboard Maestro, traditionally subroutines were implemented with the Execute a Macro action, passing a single parameter to the %TriggerValue% token, and returning results in instance or global variables.

In version 10.0, Keyboard Maestro adds an explicit Subroutine trigger which lets you specify the parameters by name, and whether the macro returns a value via the Return from Subroutine action. Such subroutines are called with the Execute a Subroutine action.

In the remainder of this post, the term sub will refer to both.

Potential Issue When Sharing Subs On This Forum

When creating macros for myself, I often put reusable actions in subs. When sharing macros on this forum, however, I've typically avoided using them because they can be problematic, not when sharing Version 1, but when new versions need to be subsequently shared.

Here's a scenario that causes a problem:

1. You share Version 1 of sub on the forum. This distribution could include one or more calling macros.

2. A user downloads and installs your sub and calling macro(s).

3. The user uses your sub and calling macros. The user might also create their own calling macros that leverage your sub.

4. You share Version 2 of sub on the forum. In this example, there's no need to change the calling macro(s).

5. The user downloads and installs Version 2 of your sub before deleting Version 1. Once Version 2 is installed, the user notices that two versions are installed and then deletes Version 1.

6. The user runs one of the pre-Version 2 calling macros that you shared or the user created. The calling macro fails. Upon investigation, the user notices that the Execute a Macro or Execute a Subroutine does not specify a submacro or subroutine, respectively. Upon further investigation, the user notices that all calling macros have been disassociated.

What Causes the Calling Macros to Become Disassociated?

The Execute a Macro or Execute a Subroutine actions in calling macros appear to refer to submacros or subroutines by name, but actually these actions internally track a unique identifier called a UUID (e.g., 3DD41F1A-6C67-48F3-ACC7-68D930076DB1).

You can find the UUID for any macro by right-clicking, then selecting Copy as > Copy as UUID

When the user installed Version 2 of your sub, Keyboard Maestro checked if the UUID was already in use (by another macro), and since it was, assigned a new UUID to Version 2.

Thus since the calling macros were referring to the original UUID, not the newly assigned one, the calling macros fail.

This problem could have been avoided if the user had deleted Version 1 before installing Version 2. That sounds simple, but it is easy to overlook even if a forum poster includes upgrade instructions.

---

Including Caller(s) Protection in a Sub

Okay, before continuing, get a second cup of coffee.

As I was recently developing a submacro named sm.𝗽𝗮𝗹𝗲𝘁𝘁𝗲⇾MoveMouseToPalette, I was thinking about this issue. Soon after releasing the original version I wanted to share an update. In the post, I included the delete-before-upgrade warning, but I wanted to include some additional protection. After some head-scratching, here's what I created and added to the top of the submacro:

---

Download: Group.kmactions (3.4 KB)

---

The above group, in turn, calls this subroutine (which can be called from any sub that includes the above protection group):

Download: s.𝗸𝗺⇾CheckSubUUID.kmmacros (12 KB)

Macro-Image

---

Macro-Notes

• Macros are always disabled when imported into the Keyboard Maestro Editor.
  • The user must ensure the macro is enabled.
  • The user must also ensure the macro's parent macro-group is enabled.

---

System Information

• macOS 14.2 (23C5030f) PRE-RELEASE SEED SOFTWARE
• Keyboard Maestro v11.0.1

---

Back to the scenario discussed above. If the user attempted to run Version 2 before deleting Version 1, the subroutine (s.𝗸𝗺⇾CheckSubUUID), called from the added group, would generate this dialog:

Whereas if the user ran this submacro after deleting Version 1, this dialog would appear:

---

---

In addition, I've included the following information in the submacro header comment.

PROTECTING CALLING MACROS

The first Group action in this macro includes a call to a subroutine (s.𝗸𝗺⇾CheckSubUUID) that will help ensure that current calling macros will work with this and future versions of this submacro.

If an upgraded version of this macro is installed before deleting the previous version (or if this submacro is duplicated), the upgraded version (or duplicate) submacro will display a warning and instructions for repair.

If the import-before-deletion (or duplication) was intended, the warning can be stopped two ways:

1. The OriginalUUID within the first group action can be modified to match the UUID of the newly imported (or duplicated) submacro.

2. The first group action can be deleted. If this is done, however, new calling macros will not be protected.

WARNING: If either of these changes is made, old calling macros will no longer refer to this submacro.

For more information (including a download link for s.𝗸𝗺⇾CheckSubUUID), refer to this Keyboard Maestro Forum Tutorial: Avoiding Upgrade Issues with Submacros and Subroutines

UPGRADING THIS MACRO

If upgrading from an earlier version of this submacro, it is imperative to first delete the older version prior to importing this version into Keyboard Maestro. Note that calling macros do not need to be deleted.

---

---

ACKNOWLEDGEMENTS

Thanks to @noisneil, @tiffle, and @troy for reviewing this information before I shared it here. As tedious as it reads now, it was much worse before their input.

[cyoungers]

I've been bit by this issue so many times and never understood why. Thanks for clearly explaining what is going on.

[_jims]

Hi, @peternlewis. As I've discussed the above issue with others we've wondered if there might be some change that could be made to Keyboard Maestro during an import.

Brainstorming, i.e., not totally fleshed out...

If there is a UUID collision during macro import, could Keyboard Maestro provide a dialog?

The macro XXX (being imported) has the same UUID as macro YYY (already in the library). You have three options:

1. Replace the existing macro, or
2. Change the UUID of the macro being imported.
3. Cancel the import.

Of course, without alerting, 2. is what is done now.

---

If you hesitate when it comes to replacing an existing macro, how about these three options?

1. Change the UUID of the macro already in the library.
2. Change the UUID of the macro being imported.
3. Cancel the import.

[griffman]

While I'd love to see a solution to this issue, changing an existing UUID would not be my recommended solution. If anyone has built anything based on selection on UUID, it would break. It would also wreak havoc on MacroBackerUpper, as it relies on the fact that a UUID shouldn't ever change—that's how it determines, for instance, that a macro or group was renamed, and isn't new.

I like the first proposed solution much better: If it's going to be newly added anyway, it should get a new UUID. But don't change history!

-rob.

[_jims]

Hi, @griffman. Thanks for weighing in!

@griffman

If anyone has built anything based on selection on UUID, it would break.

It wouldn't break, the imported macro would be the new target. When updating submacros or subroutines, that's often the objective.

(Disclaimer: This could possibly wreak havoc because Keyboard Maestro is doing something internally; only @peternlewis, of course, would be able to shed light on that.)

[griffman]

Ah, sorry for the misunderstanding.

-rob.

[noisneil]

I think those are good suggestions and I can't really see a downside to having that as an option in preferences.

[tiffle]

Since KM allows macros to have the same name in a Macro Group, it cannot always by itself determine whether the macro being imported is

1. a totally new macro, or
2. an updated version of an existing macro

So I would guess the only sensible thing KM could do when it is being asked to import a macro into a Group, and a macro of that name already exists in that Group, is to ask the user which of the above 2 options is the intended one. If it's (1) then the import goes ahead with the new macro and its UUID, but if (2) the existing macro should be replaced (overwritten?) by the new one but keep the existing macro's UUID.

(BTW, If I've misunderstood, apologies.)

[_jims]

Hi, @noisneil. Thanks for weighing in!

@noisneil

I think those are good suggestions and I can't really see a downside to having that as an option in preferences.

If this was a preference/setting, my vote would be that one preference option would be to ask for each import (if a collision occurs).

[_jims]

Hi, @tiffle. Thanks for weighing in!

@tiffle

Since KM allows macros to have the same name in a Macro Group, it cannot always by itself determine whether the macro being imported is

1. a totally new macro, or
2. an updated version of an existing macro

My assumption is that Keyboard Maestro doesn't consider the names, i.e., it's all based on UUID's. Now maybe I'm misunderstanding your comment, @tiffle.

@tiffle

So I would guess the only sensible thing KM could do when it is being asked to import a macro into a Group, and a macro of that name already exists in that Group,

Since calling macros aren't effected if a target (submacro or subroutine) is moved from one group to another, I'm thinking that the the proposed import dialog should be based solely on macro UUID collisions, not the group membership.

[tiffle]

Crumbs - there you go, I imported the wrong version of my post! That G&T must have given me finger trouble.

The summary of my comments is this: what KM currently does by default isn't always what the user nor the author of the macro desires, so it would be an improvement if KM explained the issue and offered an alternative. I wouldn't vote for a preference setting.

[peternlewis]

I don't really have any solid plans to implement any of these changes.

Keyboard Maestro aims to ensure that the links within the imported macros continue to be be correct. But I don't really want to get in to replacing macros because it's virtually impossible for end users or Keyboard Maestro to know whether that is a good idea. If the replaced macro behaves differently, and is used by existing macros that aren't being replaced, then who could say whether replacing it will be beneficial or not.

As it stands, importing the new macros should not affect the behaviour of the existing macros in most cases.

With this proposal, Keyboard Maestro would end up asking the user a question they would not have any way to know how to answer.

Basically, my recommendation would be if you want to distribute macros, and deal with updates, then you will need a documented process for how best the user should handle the updates. Simplest is to put them all in a set of macro groups and have the user delete the existing macro groups and import the macros.

But if you intend to include sub-macros that are used by multiple different macro sets, then inevitably you need an entire package mechanism which is beyond what I intend to implement at this point.

[_jims]

Thanks for the reply, @peternlewis.

@peternlewis

Keyboard Maestro aims to ensure that the links within the imported macros continue to be be correct.

Yes, and in most cases it works very well. Thank you for that!

My OP is addressing the cases when submacros or subroutines:

1. are general in nature, i.e., could be called by a variety of macros unrelated to one another (thus the callers are in unrelated macro groups),

2. might be refined over time, and

3. will be shared on the forum (thus imported).

The example I provided, sm.𝗽𝗮𝗹𝗲𝘁𝘁𝗲⇾MoveMouseToPalette, is such a macro. As I'm sure you recognize, submacros or subroutines like this can be incredibly useful because functionality can be created once and used many places.

@peternlewis

I don't really have any solid plans to implement any of these changes.

I suspect you are referring to my above suggestion. If that's the case, I'm not surprised, and frankly that's why I created a method to provide some protection.

My main objective with the OP was to raise awareness and to provide a thread for discussion. Hopefully it has helped you understand an issue that some users encounter. (I've seen many examples of this on the forum and the solution you ultimately roll out exceeds expectations.)

[peternlewis]

@_jims

My OP is addressing the cases when submacros or subroutines:

Yes. There is no single solution for such cases. For a submacro that is general in nature, used by multiple independent macros, and might change in the future and need updating, when updating it there is no clear way to know whether the changes are compatible with the existing macros or not, and asking the user is not really a viable solution as they will have no way to know.

So the simplest solution for macros that are distributed is probably simply not to do this, or to accept that it will never be possible to update the sub-macro, and instead add a version number to the macro and create a new submacro each time you change the version.

[DanThomas]

I know I'm late to the party here, but I may have a partial solution.

I'm working on a revised version of my "MIM Lite" macro (Macro Import Manager Lite), and I may be able to help with this.

Right now my intention has been to show you what's in a kmmacros file, like this:

You can choose which macros to import, and you can rename both the macros and groups.

I think it would be fairly simple for me to add the ability to show if the macros already exist. This way, you could choose not to import those macros, or at least some of them.

It's even remotely possible that we come come up with a standard way to identify the version numbers of macros, so I could recommend only updating the macros with newer versions.

I haven't worded this very well, but I hope you get the point.

I think there's still a lot of questions here, but I thought I'd re-open the subject.