Macro Repository Suite: Backup and Version Control for Keyboard Maestro

[DanThomas]

Macro Repository Suite v1.1.0

Requires Keyboard Maestro v9 or higher.

Update 2021-06-19: I've been using this for a couple of years now, and I've never encountered a problem, so it's pretty stable. (I know, famous last words.)

PURPOSE

Creates separate source files for each of your Groups and Macros, in a Repository Folder of your own choosing. This makes it easy to back them up, reload them if needed, and optionally place them under Version Control.

The Macro Repository Suite contains two Keyboard Maestro macros that are extremely easy to use:

Macro Repository Updater

Saves all of your Groups and Macros to individual source files, in your Repository Folder.

Macro Repository Importer

Lets you select Group or Macro source files from the Repository Folder, and import them back into Keyboard Maestro.

License Agreement/Disclaimer:

I've tried my best to eliminate bugs, but nothing's perfect. So although I think this Suite works well, please be aware that something could go wrong, and I can't guarantee perfection.

Copyright (c) 2019 Daniel Thomas

Permission to use, copy, modify, and/or distribute this
software for any purpose with or without fee is hereby
granted, provided that the above copyright notice and this
permission notice appear in all copies.

THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS
ALL WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO
EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT,
INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS,
WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER
TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE
USE OR PERFORMANCE OF THIS SOFTWARE.

CHANGES LOG

Requires Keyboard Maestro v9 or higher.

v1.1.0 2019-09-16:
Added the variable Instance_saveAsJson, which lets you save source files as JSON instead of XML. Added links to the online documentation.

v1.0.2 2019-09-15:
Fixed an issue with "blank" Group names. Cleaned up some error-checking. Added the variable Instance_showLog.

v1.0.1 2019-09-13:
Added option to bypass the trash when deleting the Data folder.

FILES:

Macro Repository.v1.1.0.kmmacros.zip (65.1 KB)

Installation

1. Download the .zip file.
2. Unzip the contents, which will give you one ".kmmacros" file.
3. Double-click the ".kmmacros" file. This should cause Keyboard Maestro to import two macros into a group named "KM".
4. If you don't want to read any documentation, at least read the comments at the top of the Macro Repository Updater macro.

Documentation

Full documentation can be found here: http://bit.ly/2lTMXF7

[JMichaelTX]

@DanThomas

PURPOSE

Creates separate source files for each of your Groups and Macros. This makes it easy to back them up, reload them if needed, and optionally place them under Version Control.

Dan, my first thought is just "wow!"

Thanks for developing and sharing such a great, sorely-needed tool.
As you know, because KM uses a single, large, monolithic file in which to store all Macros, there has never been a good way to backup and restore individual macros.

I've read through, but not studied, your excellent document, and I have some questions:

@DanThomas

Change one Action in the Macro Repository Updater macro to point to your Repository Folder

This looks like it uses a Global KM Variable, macrep_repositoryFolderPath. Since that does not follow the convention that you and I were using, which has a prefix of "DND_" for all Variables to Do Not Delete, what about this Variable? Is it a Variable we must retain?

@DanThomas

Replacing Existing Groups or Macros

When you import a Group or Macro, it does not replace existing Groups or Macros. So you can end up with multiple Groups or Macros with the same name.

This does not follow the practice of most backup systems that I have used, which when you do a restore, it offers a primary choice of "replace", with an option to "restore to new file".

In the case of KM, we are, of course, talking about macros, not files.
But, if I want to "restore" an existing Macro to a prior version, so that all other Macros and scripts that rely on its UUID will continue to work, then it looks like I have to manually take other steps before I use your system to restore:

@DanThomas

If you want to replace existing Groups or Macros that are referred to by other Macros , you have to delete the existing Group or Macros first. Renaming is not good enough. Use File->Export Macros if you want to be sure you can restore them if needed.

I would be great if you could provide us with a true restore option that would first delete the existing Macros, and then do the restore from your Repository, so that the Macro UUIDs are retained. Is that possible?

@DanThomas

Each time you run the Macro Repository Updater macro, the Data folder is trashed and re-created, and populated with all your source files.

So, after running the Macro Repository Updater macro, you can update your VCS if you want.
. . .
Because ==every time the Macro Repository Updater macro is run, ALL the source files are recreated==.

So, if I understand this correctly, your backup system is an all-or-nothing, with no option to just backup selected macros, or backup macros that have changed since the last backup. Is this correct?

Also, it appears that I have to manually invoke your backup system, as opposed to having it run periodically and do a classical backup strategy of either "full, differential, or incremental"? I suppose we can add a Macro Trigger to your backup macro that runs on some schedule, right?

Is there a way to know when then last time a Macro was modified?

Well, I think that's all for now. Thanks again for sharing a great system. Don't take any of my questions in a critical way. I'm just trying to make sure I understand correctly.

[DanThomas]

Thanks for the comments, Jim! I'll see if I can answer them.

@JMichaelTX

This looks like it uses a Global KM Variable, macrep_repositoryFolderPath . Since that does not follow the convention that you and I were using, which has a prefix of "DND_" for all Variables to Do Not Delete, what about this Variable? Is it a Variable we must retain?

The macro deletes the variable at the end. It just has to be a "real" (i.e. not a Local or Instance) variable long enough for the JXA script to use it.

But, if I want to "restore" an existing Macro to a prior version, so that all other Macros and scripts that rely on its UUID will continue to work, then it looks like I have to manually take other steps before I use your system to restore:

Honestly, I didn't want to put the work into a UI to handle that. You're welcome to do it, though.

I would be great if you could provide us with a true restore option that would first delete the existing Macros, and then do the restore from your Repository, so that the Macro UUIDs are retained. Is that possible?

I'll see how well this is received. If it looks like it's actually going to be used, then I'll consider doing something about that.

So, if I understand this correctly, your backup system is an all-or-nothing, with no option to just backup selected macros, or backup macros that have changed since the last backup. Is this correct?

Yeah, but that's not really an issue, which is why I talked about using a VCS that compares file contents, not dates.

For example: I just changed a macro, and re-ran the updater macro. This is what SourceTree shows has changed - one file:

So even though all the files were rewritten, SourceTree (using Git) knows that actually only one file was changed. So for all intents and purposes, it is an incremental backup.

Well, I suppose that's not 100% true. If you run something like Time Machine backups, it'll re-backup everything. But my repository is only 30 MB, so I don't think that's much of an issue.

The code is so much cleaner this way, especially if (when) something goes wrong. I can explain in more detail if you need me to, but unless there's a compelling reason not to do it this way, I don't really see me changing it.

[JMichaelTX]

@DanThomas

The macro deletes the variable at the end. It just has to be a "real" (i.e. not a Local or Instance) variable long enough for the JXA script to use it.

JXA can read/write both Local and Instance Variables:

var kmInst = app.systemAttribute("KMINSTANCE");
var kmeApp = Application("Keyboard Maestro Engine")
 
var myLocalVar = kmeApp.getvariable("Local__MyVar",  {instance: kmInst});
kmeApp.setvariable("Local__FromJXA", {instance: kmInst, to: "Set in JXA Script"})

@DanThomas

So even though all the files were rewritten, SourceTree (using Git) knows that actually only one file was changed. So for all intents and purposes, it is an incremental backup.

Well, I suppose that's not 100% true. If you run something like Time Machine backups, it'll re-backup everything. But my repository is only 30 MB, so I don't think that's much of an issue.

I get that. I do see your POV.

I'd like to make use of TM. So I may, when/if I get time, take a look at your system to see if it can easily be modified to only backing up changed macros.

One question here: Why didn't you just use the .kmmacros format, which is essentially XML?

@DanThomas

The code is so much cleaner this way, especially if (when) something goes wrong. I can explain in more detail if you need me to, but unless there's a compelling reason not to do it this way, I don't really see me changing it.

Sure, I totally get that, and don't blame you.

[DanThomas]

@JMichaelTX

JXA can read/write both Local and Instance Variables:

Well, that's pretty cool! I'll give it a shot. Thanks!

I'd like to make use of TM.

I run it every 2 hours. No problem backing up an extra 30 or 60 MB.

So I may, when/if I get time, take a look at your system to see if it can easily be modified to only backing up changed macros.

Go for it. Just remember that not only do you have to save existing macros and groups that have changed, you have to add new ones, delete old ones, and rename existing files and folders when necessary.

And if you encounter an error during that process, and the repository gets messed up somehow, you then have to figure out how to fix it. And if the solution is going to be used by others, it'll have to be able to fix it automatically.

By the way, I just added an option to bypass the Trash when it deletes the Data folder. I'll upload the new version when I've tested it a little more.

One question here: Why didn't you just use the .kmmacros format, which is essentially XML?

I had written an answer for this, but then I realized I might be wrong, so let me get back to you on this.

[DanThomas]

OK, I think this is true. If it isn't, let me know.

As far as I know, a .kmmacros file has to include the group. So this means the group source would be included in each macro source file, and if you change the group, all of the macro source files would have to change.

And yes, I'm already rewriting all the macro files, so what's the big deal? Well, version control system would recognize the changes in all those files, so instead of just saving the changes to the group, it would save the changes to all the macros too.

Also, if I ever do want to support replacing existing macros and groups, then I don't want people to just double-click the files.

But who knows, maybe it's OK to have the group info in each macro file anyway? Let me know what you think.

By the way, the actual "import" process is nothing more than reading the source file(s), sticking them on the clipboard with the correct "clipboard type", then pasting them into KM.

[JMichaelTX]

@DanThomas

As far as I know, a .kmmacros file has to include the group.

Confirmed.

@DanThomas

So this means the group source would be included in each macro source file, and if you change the group, all of the macro source files would have to change.

That's a good point. I don't know of any way of exporting/backing-up JUST the Macro Group without its macros. We might be able to do this with a script.

@DanThomas

But who knows, maybe it's OK to have the group info in each macro file anyway? Let me know what you think.

Well, if it were my design, I'd keep the definition of Macros and Macro Groups separate, but linked, just like in a relational database.

When I restore a prior version of a Macro, I most likely do NOT want to also restore a prior definition of its Macro Group.

[DanThomas]

Sounds good to me. Thanks. And I enjoyed our FaceTime chat. That was fun!

[DanThomas]

Just uploaded version 1.0.1. It adds the ability to bypass the Trash when deleting the Data folder. I also changed to use "Instance" variables instead of regular variables.

And if anyone wonders why I use Instance variables instead of Local variables, it's so I can launch the Updater macro from another macro, and set the variables in the first macro. That way I can leave the Updater macro with the variables un-set on my system, so I don't modify the Updater macro and accidentally upload it that way.

[rolian]

In both versions, I get the following about 25% of the way through the progress bar"

Not sure what is going wrong. TIA.

[DanThomas]

My guess is that you have a macro group called "Untitled Macro Group". If you do, give it a "real" name, and the problem should go away.

Let me know. Thanks.

[rolian]

Bingo! Thanks a million.
As an aside, I got a slightly different error about not being able to make a filename, but once I retitled a couple of macros that started with emoji and another macro that had no title at all, all is good.

Thanks for all the great stuff you've developed and shared with the forum.

[DanThomas]

Thanks! And thanks for mentioning Emojis. I hadn't even thought about them.

I'm right in the middle of cleaning up a few odds and ends, so now's a perfect time to find these issues.

Please let me know if you find any other problems, even if they seem small. Thanks!

[DanThomas]

Can you tell me what emoji didn't work? I've tested with a handful (well, 3) emojis and they all worked as file names.

Thanks.

[rolian]

I ditched the emojis because I thought they were the problem, but it might just have been the blank file name only. I can say that Spotlight Search Macros that start with greek letters pi and omega didn't break anything.

[DanThomas]

Thanks - that makes sense. Looks like emojis are OK in file names, and a Google search even talks about doing it. Good, because the code to get rid of emojis looked kind of ugly.

[DanThomas]

Uploaded v1.0.2. See the change log in the first post.

[JMichaelTX]

@DanThomas

Uploaded v1.0.2. See the change log in the first post.

Dan, I am confused by this update:

1. It contains only one macro: "Macro Repository Updater"
2. The Comment Title says: "Macro Repository Updater v1.0.1"
   • 

FWIW, I would prefer an update that replaces the existing macros.
What do you think?

[DanThomas]

My bad. It's fixed now. Thanks for catching that (both things).

[Shoshanna]

This is fabulous! Thank you for creating and sharing it.