MacroBackerUpper (MBU) is my new macro designed make it simple to create multiple time-based backups of your KM macro library. It functions much like Time Machine, creating a date-stamped set of folders, each with a full backup in it:
───────────────────────────────────────
__ MacroBackerUpper 2.9 Macros.kmmacros (1.2 MB)
────────────────
Feb 12 2024: Version 2.9: For users with macros (in the same group) with identical names, or with leading-dot-named macros and/or groups, this version makes it easier to find and change (if you wish) all such names. The macro now writes the lists of "dupes and dots" names to two files in a temporary directory on your disk, then opens that folder when the macro finishes. (This won't happen again if you disable warnings for dupes and dots when prompted.)
Previous versions
Jan 11 2024: Version 2.8: A maintenance release that updates some naming conventions and re-enables the first-run macro, which was left disabled in the last update.
Dec 21: Version 2.7 fixes a couple fairly small bugs. One caused needless entries in Keyboard Maestro's log file, and the other wrapped some text that shouldn't have been wrapped.
Dec 7: Version 2.6 fixes a bug that @_jims found when the backup folder couldn't be found. Sadly, I caused the bug by changing a routine in the 2.2 release. He also suggested that the macro not tell you it doesn't need an update, if the update checker ran on its schedule (as opposed to you running it). There are a couple other minor changes, but nothing major.
Dec 2: Version 2.5 changes how MBU looks for DefaultFolder X, as it turns out it's not visible to the Keyboard Maestro "is app active?" action if run through SetApp. Using ps in Terminal fixed that. The only other change clears a few large instance variables when MBU is done with them, to free environment space.
Nov 27: Version 2.4 has only two substantive changes, but they're important. First, I was able to work around the Default Folder X issue, as the app is AppleScriptable and includes a very useful enable and disable set of commands. The macro will determine if Default Folder X is running, and if so, disable it before the backup, then re-enable it when the backup is complete. I have tested this myself with the demo of Default Folder X, and it works very well.
The other big change is behind the scenes: I've switched to a more-obscure delimiter for database tasks, which should prevent name-based collision issues a couple users saw. Long story short, you can now use ### in your names again. However, if you happen to use ←🔸→ … well, don't do that. Seriously, just don't :).
Nov 26: Version 2.3 is out with a couple important warnings for Default Folder X users, as well as those who include ### in their macro and/or group names. See the full release notes in the most recent post from me in this thread, or read them onscreen or in the app. There are some other minor fixes, but no new features this time out.
Nov 24: Version 2.2 is out. The macro now includes a diff viewer, so you can see exactly what changed in macros that were marked as modified. (Only for modified macros that have the same name and group as in the last backup.)
The Settings Manager has been rewritten to be easier to use when actually setting settings, and you can do more from the delay box: Cancel, check for an update, or open the diff viewer. I also cleaned up some SQL calls, and fixed a bunch of little things.
Nov 21: Version 2.1 is out. It (hopefully) fixes the two major issues some people had, and cleans up some other stuff I found while going through everything again. The two big issues (macro failing when renamed or leading-dot-named macros/groups existed) were due to me missing one word ("Macros") while editing my existing SQL subroutine calls in three spots across the hundred-odd times I call it.
I also modified the first run experience, so new users only need answer two questions to get going. And I fixed a number of other things, detailed in the release notes post that will soon appear at the bottom of this thread.
Sorry for the issues; I really should have caught those during testing.
Nov 20: Version 2.0. Yes, a major upgrade only weeks after initial release. Why? Due to some structural changes that will make it impossible (without resetting things) to use version 1.4 again after upgrading. But those structural changes enabled some nice new features, including the ability to disable/enable Finder labels, change the delay before backing up, tracking enabled and disabled groups, and more. See the full notes in a separate post below.
Before running version 2.0, I recommend running 1.4 once more, to get your latest changes into a tracked update. The first backup with 2.0 will be a full (no hard links) backup, and without any comparisons to the prior backup. After the first, hard links and analysis return.
Please see the version 2.0 power below for more details on features, upgrading, and downgrading.
Nov 13: Version 1.4: Lots of changes here, but the most important is a typo I found and fixed—I had m where I should have had rm in a shell script command. Not good. I made some changes to reduce the spurious log entries that MBU generates, including one for non-error Type 1 errors from the diff command.
I also removed a routine that tried to manipulate the export window—I think that was causing some of the issues here, so now we'll just have to put up with that window not being centered or sized.
Full release notes in the app, or further down in this thread.
Nov 12: Version 1.3: Big update today. I added some code that will attempt to resolve issues resulting from an incomplete run. To do that, the latest backup is deleted and the database is rebuilt, as that's the only way to get things 'right' for the next backup.
If you use a palette trigger, you should now see the final report if you have it set to show when manually triggered. I also added some code to a key AppleScript to hopefully avoid the spurious -600 errors some users are seeing. Thanks, Chris, for both those tidbits!
Perhaps most importantly, after 3,000+ uses, I managed to find (and fix!) a condition that caused a hard link to fail to be created. As far as I know, this only affected me so far, but please run a new backup after installing 1.3 as it will resolve the issue if you had experienced it.
(The condition: If you had a one-macro group, and you renamed the macro in that group, and you renamed the macro in that group, then the hard link wouldn't be created.)
Nov 10: Version 1.2: Added two new features (let's call it the Chris Release!). You can now control how/if the final onscreen summary dialog appears, and MBU will attempt to put the Keyboard Maestro app back into the same condition it was when the macro ran: Active and visible, active and hidden, or not even running.
I also fixed a few other minor bugs, including one in the update routine, which means you can't use the update routing to update this time. Finally, I added some troubleshooting instructions to the help page.
Nov 9: Version 1.1: A very important bug fix now means MBU should run for everyone, regardless of your number format of choice :). A few other behind-the-scenes updates, too. If you already have 1.0, try the in-macro updater by running the upd | Update check macro. This won't actually install the macro, but it will download it and verify that it hasn't been tampered with.
Nov 7: Version 1.0: Initial release
───────────────────────────────────────
HOW IT WORKS
Just like Time Machine, items in the backup that haven't changed since the last one are replaced with hard links (optional, on by default), which are tiny pointers to the older version of the identical macro. This lets you keep a ridiculously large number of backups in a relatively tiny amount of drive space. After my final round of final testing, I have 24 backups totaling 35MB of drive space—and just one full export of my collection is 24MB.
(Speaking of testing, a big shout out to jims and tiffle, both of whom helped me find and fix quite a few issues with the analysis and reporting in MBU. Thanks!!)
Within each backup folder is a report explaining exactly what changed from the prior backup—MBU analyzes moves, renames, renames and moves, new and deleted macros and groups, and modified macros. Here's a sample of a report:
Very lengthy sample report hides here
The geeky timing section at the bottom of the report is optional (and off by default).
BACKING UP
Run •main | MacroBackerUpper, wait three seconds, and the backup starts. It's as simple as that (after initial setup, of course). But what if you want to change those settings you made during setup? Just press the Control key before the timer in the obnoxiously bright green box expires.
Do that, and the settings manager will open:
Included in the Settings Manager is the ability to relocate all your backups to a new location. The main downside of hard links is that if you use Finder to copy them to another drive, they become real files, and your backup archive is suddenly huge. There are instructions in the macro on how to avoid this in Terminal, or you can use the Settings Manager.
When a backup completes, you'll see an onscreen summary:
The important bit in the summary is the total disk space usage. If you compare that figure to Finder, you'll find Finder's value is huge, as it treats hard links as if they were full files. The value shown in the summary window is the actual drive space needed for all your backups.
THE ANALYSIS
MBU does its best to identify exactly what changed with every single macro and group in your collection since the last backup. I'd say it's pretty good at it, but there are limitations. One of those is that when you duplicate or rename a group, all of the macros in that group are modified in some way by KM. As such, MBU will report all those macros as modified, because they are, even though you didn't modify them.
There are probably other things like this that I haven't run into yet, but overall, I think MBU does a really nice job of explaining what's changed between the last two backups.
For modified macros that are actually modified, and within the same group (and with the same name) as in the last backup, you can open a visual diff tool to see exactly what changed. The macro supports FileMerge (which requires Xcode), Visual Studio Code, and as a fallback, the built-in diff results in a window. Here's how it looks in FileMerge:
RESTORING MACROS AND MACRO GROUPS
MBU was not written as a full-fledged "backup and restore" tool. It is a backup only tool, as I didn't want it to ever have any chance of harming my macros if I had coded something incorrectly.
So how do you restore? Via Finder, because the backups are just files in Finder. Full instructions are in the help, but basically, any macro or group can be restored with a few clicks (and an important renaming step within KM first).
HOW SAFE IS IT TO USE?
It's incredibly safe to use, because it doesn't touch any of your macros (see above, i.e. why it's not a restore tool as well). The only work that happens within Keyboard Maestro is the initial export of your collection, using Keyboard Maestro's own Export All Macros as Folder option.
After that, everything that happens is working with the files on the drive, not the macros in your collection. During development. I have run the macro over 2,500 times, and not once have I had an issue with my macros, nor the backing up of my macros.
You may see issues with the analysis. You may see issues with a Finder label not being able to be applied. You may see some other issue. But none of these will be about the backup itself, which I've found to be perfect (because KM is doing the work!) every time.
So yes, it's very safe to use.
CONCLUSION
I've tried to make MBU as simple to use as possible; there are only a handful of settings on first run (along with a one-time-only intro screen). During each backup, progress bars let you know what's going on, and certain events are communicated via notifications if necessary.
There's a relatively detailed help page for MBU, and the "Info: Read Me First!" macro contains a condensed version of the help file's key points.
If you've been looking for an easy-to-use somewhat-versioned-like backup system for your macros, give MBU a try and let me know what you think. For me, it's already turned into an essential tool, and I used it a few times during development to get an older version of the macro back.
-rob.
... and none of us saw that during our testing, at least not after the early tries. I've sent you a DM with some instructions on getting me some debug info that should let me track down the problem quickly.
