How to archive unused KBM macros?

[August]

I'm concerned that my KBM macro file is getting too big. I have a lot of macros that I rarely use and/or only use in specific circumstances. I really don't need them present all the time.

I want a robust way to archive unused or rarely used macros and macro groups. I've pored over the forum without finding anything relevant.

I know how to export macros, remove them, and re-import them. That's what I've been doing and it's awkward because I have to separately browse the exported macro files to see what's been archived.

What I want is some way, within the KBM Editor UI, to easily see a list or lists of what I have exported. Ideally I would be able to see more info than just the title of the macro.

In my fantasy, in a pattern similar to what @noisneil has done in Applescript: Replace Text Script with File, a macro would first save the to-be-archived macro as text, archive the macro in an appropirate folder, and replace the macro with a placeholder with the same title (maybe with "(ARCHiVED)" appended), and that placeholder would have a Comment action containing the copied text of the macro. That would make it browsable if what I wanted was to find a code snippet in the macro to reuse.

Has anyone done anything like that, or that even starts in that direction, or maybe has come up with a different but useful way of archiving macros?

[peternlewis]

You can sort macros by Date Used to find macros that have not been used for a long time.

Most interestingly perhaps, you can sort macros by Size. Since macros are generally very small unless they contain images, you might find that your macro file size is largely dominated by just a few macros. Deleting them or moving the images out of the macros might be the answer.

Otherwise, you can export macros and delete them and re-import them later if necessary. You can use the Export All Macros as a Folder and then just delete the ones you aren't using.

[noisneil]

@August

Comment action containing the copied text of the macro.

You mean the XML? In that case you wouldn't be saving very much space.

Have any performance issues lead you to the conclusion that your macro library is too large? If so, try disabling all but one of your macro groups and see if performance improves. If it does, then all you need to do is disable your unused macros rather than archive them.

As I understand it, the biggest contributor to a bloated library is embedded images, so you might like to consider referencing files instead where possible.

I wrote a couple of macros that will automatically convert an embedded image to a file reference or vice-versa, and also display the referenced file for convenience:

Found Image - Image ←→ Path.kmmacros (60 KB)

Macro screenshot

Found Image - Display Image from Path.kmmacros (37 KB)

Macro screenshot

[August]

@peternlewis

Otherwise, you can export macros and delete them and re-import them later if necessary. You can use the Export All Macros as a Folder and then just delete the ones you aren't using.

Thanks for the tip about "Export All Macros as a Folder". Can you then import them back in one at a time? That could be useful.

Up to now I've been exporting them one at a time and making up a folder structure that doesn't exactly match the group structure, it's more about how I think about their use.

My issue is keeping track of the deleted macros, to remind me both of their existence and of their content. Often when writing a new macro, I remember having done something similar before, but not exactly where. If I've archived the macro, I can't look at to remind me of what I did before.

@noisneil

You mean the XML? In that case you wouldn't be saving very much space.

No, I mean what you get when you select the whole macro and then click Edit > Copy as Text. Here's an example of a text macro that @_jims posted a while ago that I happen to have open:

Test WhichSpace
No triggers specified.
Will execute the following actions:
TIMING ACTIONS: Position at the top of a macro
Set Variable “local_Milliseconds0” To Calculation “MILLISECONDS()”
MILLISECONDS() Stop macro and notify on failure.

	Timeout Set
	Note that the Settings (gear icon) for the Execute AppleScript action have been changed: 	• Changed Timeout to 2 seconds. 	• Disabled Timeout Aborts Macro 	• Disabled Notify on Timeout. 	• Disabled Failure Aborts Macro 	• Disabled Notify on Failure.    
	Get DesktopNo from WhichSpace
	tell application "System Events"
	    tell process "WhichSpace"
	        set DesktopSpaceList to (title of menu bar items of menu bar 1)
	    end tell
	end tell
	return item 1 of DesktopSpaceList Save trimmed results without errors to variable “local_DesktopNo”.
	  
	TIMING ACTIONS: Position at the bottom of a macro
	Set Variable “local_Milliseconds” To Calculation “MILLISECONDS()”
	MILLISECONDS() Stop macro and notify on failure.
	  
	Set Variable “local_DurationMillisecs” To Calculation “local_Milliseconds - local_Milliseconds0”
	local_Milliseconds - local_Milliseconds0 Stop macro and notify on failure.
	  
	Display Text Briefly
	Space: %Variable%local_DesktopNo%; Duration: %CalculateFormat%local_DurationMillisecs%#,##0% ms

I think that's substantially the same as the text you see when you click View > Stop Editing Macros. Maybe a bit differently formatted.

At one text line or so per action, that's a whole lot less than the XML.

@noisneil

As I understand it, the biggest contributor to a bloated library is embedded images, so you might like to consider referencing files instead where possible.

Yes, that's what Peter suggested above.

@noisneil

I wrote a couple of macros that will automatically convert an embedded image to a file reference or vice-versa, and also display the referenced file for convenience:

Excellent! That should be a big help.

[griffman]

@August

Up to now I've been exporting them one at a time and making up a folder structure that doesn't exactly match the group structure, it's more about how I think about their use.

Given that it's going to take up almost as much room (perhaps more?) to store the text of the macro in your macros as it would the macro itself, why not just leave them there and separate them?

That's what I've chosen to do:

The top (colorful) zzz group are all indvidual unused macros; the folders below are unused macro groups. I preface them (both individual macros and groups) with the date I last worked on them, for relatively easy scanning. But as the whole macros are still there, I can easily search for any terms or actions I know I used in them.

To make the archiving/unarchiving easier for standalone macros, I wrote this:

MACRO: Macro moving and renaming aid

When I post macros here in answer to someone's question, I keep a copy in Keyboard Maestro. But I almost never use these myself; they're around mainly for reference in case I want to remember how I did something. To make it easier to keep track of these macros, I put them in their own group, and change the name to start with the date I created the macro, in YYYY-MM-DD format, like this: [image] I was growing tired of manually renaming these macros, so I wrote a macro to ease the task—it work…

I can hit a hot key to archive a macro, and hit that same hot key to unarchive a macro if I want to bring it back. It lets me insert a customized date (or defaults to today's date), and handles the movement of the macro into my disabled group. To do: Make my newer version work with macro groups as well.

This way, my unused macros are still in Keyboard Maestro, but they're physically separated and disabled.

-rob.

[August]

@griffman

Given that it's going to take up almost as much room (perhaps more?) to store the text of the macro in your macros as it would the macro itself

Thanks, Rob,

If that were indeed a given, I wouldn't have a question, and your renaming approach would work fine. But I'm not convinced that that is the case.

I ran some comparisons using @_jims's Test WhichSpace macro that I used as an example above. The saved text of the Execute AppleScript action (renamed) is simply:

	Get DesktopNo from WhichSpace
	tell application "System Events"
	    tell process "WhichSpace"
	        set DesktopSpaceList to (title of menu bar items of menu bar 1)
	    end tell
	end tell
	return item 1 of DesktopSpaceList Save trimmed results without errors to variable “local_DesktopNo”.

For comparison, this is the XML for the same action:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<array>
	<dict>
		<key>ActionName</key>
		<string>Get DesktopNo from WhichSpace</string>
		<key>ActionUID</key>
		<integer>15977236</integer>
		<key>DisplayKind</key>
		<string>Variable</string>
		<key>HonourFailureSettings</key>
		<true/>
		<key>IncludeStdErr</key>
		<false/>
		<key>IncludedVariables</key>
		<array>
			<string>9999</string>
		</array>
		<key>MacroActionType</key>
		<string>ExecuteAppleScript</string>
		<key>NotifyOnFailure</key>
		<false/>
		<key>Path</key>
		<string></string>
		<key>StopOnFailure</key>
		<false/>
		<key>Text</key>
		<string>tell application "System Events"
	tell process "WhichSpace"
		set DesktopSpaceList to (title of menu bar items of menu bar 1)
	end tell
end tell
return item 1 of DesktopSpaceList</string>
		<key>TimeOutAbortsMacro</key>
		<false/>
		<key>TimeOutPeriod</key>
		<real>2</real>
		<key>TrimResults</key>
		<true/>
		<key>TrimResultsNew</key>
		<true/>
		<key>UseText</key>
		<true/>
		<key>Variable</key>
		<string>local_DesktopNo</string>
	</dict>
</array>
</plist>

Note that this is a single action, not a whole macro. The AppleScript code itself is the same in both cases and has 179 bytes. With the simple text overhead, that increases to 334 bytes, an overhead of 155 bytes. In contrast, the XML for this one action takes 1194 bytes, an overhead of 1015 bytes for the one action.

I ran a comparison on the whole macro, and to keep it an accurate measure of what I'm trying to do, I put the text description of the macro actions into a Comment and counted the size of the XML of that single Comment action. I could paste the text into the Comment as formatted text (RTF) or as plain text, so I tried it both ways and then tried variations.

The whole macro as XML is 5452 bytes/characters.
The text description of the macro, pasted into a Comment action, as formatted text, has an XML size of 118252 bytes!
The text pasted into a Comment as plain text is 3839 bytes. It seems that the XML still includes an encoded version of formatted text, just with a plain format.
The text pasted into a Set System Clipboard to Plain Text action, an action where the XML actually keeps just the plain text and not a formatted and encoded version, the XML of that replacement is 2169 bytes.

So that makes a saved text version, within the KBM file, less than half the size of the original macro.

Since most macros have many actions devoted to managing variables and performing conditional tests, rather than simply running AppleScript, which have complex XML structures while being described with a single line of text, I have been assuming that the general difference would be at least that big, if not more.

I tried it with the largest macro I have, MultiTimer 1.3 by Rob Griffiths. The XML for that macro is 493399 bytes. The text version, saved in a Set Clipboard to Plain Text action, has an XML size of 17448 bytes. That's 3.5% of the original!

[Nige_S]

@August

My issue is keeping track of the deleted macros, to remind me both of their existence and of their content.

Sounds like you need to roll your own database -- no need to go full FileMaker or MySQL, it should be quite easy to write a macro that got the "text" of a selected macro from the KM Editor UI, exported the macro and then deleted it, then created a new note in Obsidian with the captured text along with a link to the exported file (for easy reload) plus any other info (tags?) you might find useful.

Maybe use macro UUIDs for the exported filename so there's no clash when you import/edit/re-export a macro...

Another option that might be worth investigating is the various git setups I've seen mentioned on the Forum. You could, perhaps, make a git archive for all your exports and do a new commit for each, putting your macro text in the commit message.

[griffman]

A very good analysis; I'd forgotten about the XML aspect.

To me, I view disk space as cheap, CPU (and battery) as expensive. So I don't mind just leaving those things in the KM Editor (disk space usage up) as long as they're disabled (CPU and battery usage down).

@August

I tried it with the largest macro I have, MultiTimer 1.3 by Rob Griffiths.

Hey, i know that guy; he's a disk space hog!

@August

The XML for that macro is 493399 bytes. The text version, saved in a Set Clipboard to Plain Text action, has an XML size of 17448 bytes. That's 3.5% of the original!

Let's consider that in a different context. Here's the disk space used by my Keyboard Maestro folder in ~/Library/Application Support (with the smaller ones left off the list and some blank lines added to call out some rows):

$ du -c -sh -- * | sort -r -h
254M	total

109M	Revisions Version 11
 93M	Keyboard Maestro Clipboard History.kmchunked
 31M	Revisions Version 10

 11M	Keyboard Maestro Macros.plist

7.3M	Revisions Version 9
2.1M	Keyboard Maestro Cached Compiled Scripts.sqlite
620K	Keyboard Maestro Macros Saved Version 10.plist
336K	HelpIndex.gz
216K	Keyboard Maestro By Name Sort Order.sqlite
172K	Keyboard Maestro Variables.sqlite

I have 1,046 macros on this Mac (a laptop; my main Mac has 1,200 or so). In total, those macros—the Keyboard Maestro Macros.plist file—take up 11MB (.011GB) of drive space, or .0105MB per macro. The drive in my MBP is a 1TB drive with 994.66GB of total space.

In terms of my drive space, my 1,046 macros are using 0.000000110590554% of my capacity. The entire folder, with the clipboard history and revision files, is 0.254GB, or 0.000002553636418% of my capacity.

Those are tiny numbers. How tiny?

This tiny: Let's say I write some code to scrape every KM macro I can find off the internet and save it. As a result, my macro collection goes absolutely insanely crazy and grows like a weed, to the point where it's taking up 1% (10GB) of my drive space.

How many macros would I have to find to make that happen? Roughly 941,400,000 additional macros. 941 million more macros, and I'm still only using 1% of my drive space.

I'm not saying thinning the collection isn't an important task at some point. But I don't think I'm anywhere near that point yet. Where is that point? For me, it'd be where I noticed slowdowns in the UI just from having a huge collection in the editor. I haven't seen that yet, especially with my practice of putting unused macros into their own macro group.

So before you head into a long complicated project that—at it's absolute best—will put a macro you want to re-use in a state where you first have to search for the text to find it in your database (or whatever) then reimport it—it may be worth considering just how much space your efforts will save, relative to how much space you have.

-rob.

[griffman]

Not withstanding everything I just wrote above, I think it'd be pretty simple to take what I wrote for MacroBackerUpper and make it a macro archiving tool. It would just export the macros to be archived, compress them, and then store them with a description in a database.

Added to my list of things to look at…someday.

-rob.

[August]

@griffman

Where is that point? For me, it'd be where I noticed slowdowns in the UI just from having a huge collection in the editor. I haven't seen that yet, especially with my practice of putting unused macros into their own macro group.

That is exactly the issue that I'm attempting to address. I recall @peternlewis saying that the KBM Macros plist is essentially re-saved with every keystroke in the editor, which provides the feature of not having to explicitly save the file while editing, every keystroke automatically updates the editor's data. But when the KBM plist gets too big, it takes a noticeable fraction of a second to catch up with the keystrokes. In my understanding, that issue would be there whether or not any particular macros are enabled or not, no matter how big they are, because they still need to be rewritten at every keystroke.

And I have been noticing a lag while editing. The editor sometimes cannot keep up with my typing speed. I get tired of slowing down my typing so that the comment I am typing can be complete.

I'm on a 2012 Mac running Catalina. I expect to hit performance limits and I look for how to mitigate them. I do not absolutely know that the lags I am experiencing are due to the KBM Macros plist being too big, but I'm suspicious.

OTOH, I'm nowhere near your 11M Macros file. Mine is at 2.1M. So maybe that's not the real issue.

And doesn't this fit a repeated pattern in this forum? I have a problem, I think I know what the solution is, I ask how to implement my imagined solution, when that is not the real issue at all.

[griffman]

@August

OTOH, I'm nowhere near your 11M Macros file. Mine is at 2.1M. So maybe that's not the real issue.

One cause of editor slowdowns—including an obvious typing lag—is having two editor windows open to the same macro. Even on my almost brand new desktop Mac this will cause painfully slow typing. Do you use multiple editor windows?

@August

I recall @peternlewis saying that the KBM Macros plist is essentially re-saved with every keystroke in the editor…

Only Peter could answer that, of course. Even if that is what's happening, though, mine's rewriting an 11MB file every time and I don't notice slowdowns, even on my slowest Mac, though that's newer than yours. Perhaps there are slowdowns on a 12 year old Mac with a 2MB plist file, but it seems unlikely?

But I guess my general point would apply even more: If just a 2MB plist file is slowing KM that dramatically, it's hard to believe that cutting even 50% of that is going to make a huge difference? And if it's simply the matter of writing to disk, then no amount of thinning is really going to help.

@August

And doesn't this fit a repeated pattern in this forum? I have a problem, I think I know what the solution is, I ask how to implement my imagined solution, when that is not the real issue at all.

Not just you, that's all of our patterns :).

-rob.

[noisneil]

Another slower-downer-er is the conditional real-time status check. I try to remember to turn it off if not needed (which is most of the time).

[peternlewis]

Yes, several different things can slow the Keyboard Maestro editor down.

BTW, the editor does not save every keystroke - saves are held off until the typing finishes (you can see this visually with the small “dirty” symbol at the bottom right corner of the editor window).

Things that can slow the editor down:

• Very large macro files
• Large macros
• Multiple editor windows
• Evaluating conditions involving interprocess communication or images

Remember that anyone considering something like “how can I archive some of my macros” is already likely in the top few percentiles of Keyboard Maestro users. Heck, I expect the forum is almost entirely populated with people in the top quartile of Keyboard Maestro users.

[Airy]

@griffman

I'm not saying thinning the collection isn't an important task at some point.

My thinning process is to completely wipe out all my macros every time I get a new Mac. To be honest, I do occasionally back them up, but I never reload them. In a way, I eagerly look forward to losing all my macros.

[griffman]

Glad that works for you, but there’s no way I’d want to rewrite some of mine from scratch - MacroBackerUpper alone is probably 500+ hours of work. Also, because many are distributed, I have to worry about updates not breaking stuff for others.

-rob.

[August]

@griffman

Do you use multiple editor windows?

Only rarely, and I experienced that same issue.

@peternlewis

Things that can slow the editor down:

• Very large macro files
• Large macros
• Multiple editor windows
• Evaluating conditions involving interprocess communication or images

Peter @peternlewis, could you give us some sense of scale for what are "very large macro files" and "large macros"? Of course it will be hardware dependent and depend on how many other apps are open (Chrome being a notorious hog when the number of tabs gets over 100 or so). E.g., is Rob's 11MB Macro file reasonable or marginal?

[peternlewis]

@August

Peter @peternlewis, could you give us some sense of scale for what are "very large macro files" and "large macros"? Of course it will be hardware dependent and depend on how many other apps are open (Chrome being a notorious hog when the number of tabs gets over 100 or so). E.g., is Rob's 11MB Macro file reasonable or marginal?

It is very hardware dependent. Having an SSD will make a big difference obviously. Mine is 5MB and I don't think it causes any issue, but I have a fast Mac.

To test it, just create a small macro with a simple action (say Set Variable to Text action), and then type, pause, type, pause, type, pause and see if you notice any lag.

If not, and if you do at other times, then maybe it is one of the other issues.

[_jims]

@peternlewis

BTW, the editor does not save every keystroke - saves are held off until the typing finishes (you can see this visually with the small “dirty” symbol at the bottom right corner of the editor window.

@August, if you are not a touch typist you might consider composing your comments in TextEdit then pasting them in your KM comments. I do this, particularly for my larger header comments. Helvetica Neue 13pt works very well.

[August]

Hi Jim,

@_jims

if you are not a touch typist

I'm not sure what difference you're getting at here. If I was a slow typist, I wouldn't need TE because the KBM Editor would keep up with me. It's when I'm typing too fast for the Editor that I'm having problems.

With more banging on my lag problem, it goes away if Chrome is not open in 23 windows with 98 tabs.

[griffman]

@August

With more banging on my lag problem, it goes away if Chrome is not open in 23 windows with 98 tabs.

It'll go away completely if you delete Chrome :).

-rob.