@gglick Ah, thank you! That’s cleared up my confusion completely, though left me a little disappointed.
On the upside, we can now all play a game and upload macros onto the forum for other people to test, populated with absurd variable names like LOCAL__jigglebottom and INSTANCE__cam4videos.
When I import a macro it should probably tell me what global variables are being used in the macro. That would help me protect my programs from breaking due to global variable conflict. I wouldn't expect this to work 100% accurately because variable names can be obscured by some programming.
If this isn't a task Peter would do, maybe a macro can be written by the community. I see it working this way: For each macro group, show as many global variables from within that group that is used in there.
I disabled all my macro groups, all 200 groups, (doing that manually took ten minutes because every time you click on the button to disable a macro group the KM Editor becomes unresponsive for about three seconds as it... recompiles?) and overnight there was no memory leak. That's great. That suggests the problem is one of my macros. Now I guess the quickest way to figure out which one is to do a binary search by enabling 100 macro groups, then 50, then 25, etc. until I find a specific group that causes a leak. That will get me close to finding the cause. However if I go through that much work I will post the results in a new thread. This is not the correct thread for that work.
Hey @Sleepy,
Three seconds is a very long time for this...
Run this from the Script Editor.app:
tell application "Keyboard Maestro"
set macroGroupListEnabled to name of macro groups whose enabled is true
end tell
You'll get a list in the result panel that looks similar to this (but longer of course):
{"[KM]", "[TEST]", "Activity Monitor"}
Create a new script using the following code and your newly constructed enabled list.
set macroGroupListEnabled to {"[KM]", "[TEST]", "Activity Monitor"} --<<< Paste the result of the above script here.
tell application "Keyboard Maestro"
repeat with theMacroGroup in macroGroupListEnabled
set macro group theMacroGroup's enabled to true -- or false
end repeat
end tell
Now by changing the boolean flag in the above script you can enable/disable your groups just a little bit faster.
Disable the Keyboard Maestro Engine.
And then run your script from the Script Editor.app.
-Chris
Wow. That opens my eyes a lot. The three second delay happens all over the editor, not just there. Can you help with the more general problem? I just tested if disabling the engine removes that delay. Apparently not. My main problem with the editor are these three second delays that happen constantly as I'm editing actions. Roughly 50% of the time I spend in the Editor is just waiting for the hourglass to disappear when I look at the screen, and then waiting for my typing on screen to catch up with my fingers. It seems that the delay is only created if I pause my typing for about 1 second. My guess is that the editor does not call the engine to recompile as long as I'm typing without a pause. But that doesn't add up since shutting down the engine doesn't seem to reduce the delays. I really don't know what's causing these delays. If you can fix this problem I will be indebted to you forever. I can use KM all day, but it's time for me to get some food.
Hey @Sleepy,
It's hard to say for certain, but you do have a lot of groups and macros.
On my system I have:
Macro Groups: 152
Total Macros: 1874
Enabled Groups: 105
Enabled Macros: 405
I haven't verified these figures, but they should be correct.
Here's how I counted them (run in the Script Editor):
# Count Enabled Macro Groups and Enabled Macros
tell application "Keyboard Maestro"
set mGroups to (get macro groups whose enabled is true)
set enabledMacroCount to 0
repeat with i in mGroups
set enabledMacroCount to enabledMacroCount + (count of (get macros of i whose enabled is true))
end repeat
set AppleScript's text item delimiters to linefeed
return {"Enabled Groups: " & (count of mGroups), "Enabled Macros: " & enabledMacroCount} as text
end tell
-Chris
Well, then maybe I need to export all the macros I don't use often in order to get the Editor to be more responsive. I'll check into how easy that is to do with many hundreds of macros and groups. The truth is most of my macros I use for a few months then stop using them, I keep them around only to help me research how I solved a problem before. But if they are slowing me down, I can probably just copy them into some Apple Notes folders for reference purposes. I notice that if I click on the EDIT button in the KM editor the macros are more accommodating as lines of text for saving into Notes. It would be cool if the editor let me import macros that were exported this way. It looks like all the information is there.
I suppose a quicker test would be to use a different Mac. The problem is my older Macs have even MORE macros because every time I buy a new Mac I just rewrite all my macros out of my head and that is a natural kind of cleanup. I love rewriting my macros because I usually make them better each time around.
Only Local or Instance variables that are in an existing macro are included in the list. So they should not “linger around permanently … even if the macro that contained them is long gone”
This would only work until the macro was deleted.
I have no plans to implement this.
I have no idea what would cause that. The only thing the editor does is save the macros. That can take a long time if your macro file is extremely large (which generally happens only if you have a lot of macros with very large amounts of data in them, such as inline images). Check your Keyboard Maestro Macros.plist file and see how large it is.
I tried to duplicate this memory leak. If I toggle a macro group off and on multiple times, I can see the apparent memory use in Activity Monitor grow. However if I actually look at the heap I cannot find anything in there that is growing - at least not anything I have any control over. So currently I don't have any suggestions as far as the memory usage goes.
Note that an increasing number in Activity Monitor does not necessarily mean very much. If the memory is lost to the application it will eventually migrate on to disk and become irrelevant to actual memory use.
I stand corrected! Sorry if I inadvertently cast aspersions where they weren't warranted; as I said earlier, I understand why KM populates the Insert Variable list with local and instance variables and appreciate that it does so. I did think that local and instance variables stuck around even after any macros containing them were deleted, especially since @CJK said they do for him as well, but I admit I didn't test this, and I'm happy to take your word for it.
If they do, that is a bug. They did not in my testing.
It took me 5 minutes of searching this site to find where the Macros.plist file was. Then it took me 5 minutes to figure out why my ~/Library folder didn't seem to exist on my Mac. It's probably "hidden" by default in the macOS GUI, but I found it. You asked me for the size and it's 72 MB. I have no idea if that's big or not. I tried to open the file to see it but it said I needed additional Xcode components and I wasn't sure I should click on that so I declined. Should I accept and view the file?
I'm sure you are right that virtual memory does not necessarily mean that macOS will get bogged down, but I can tell you that around 5 GB my computer grinds down to a frustrating level and around 8 GB lots of important features of macOS stop working properly. Like the "Force Quit Applications" window which takes minutes to pop up. So this increase in memory is a functional problem. I even had one time where I couldn't do anything but press the power off button, and I waited a long time.
If I take my car to the dealer to fix a squeak, and there's no squeak, they say they can't fix it. At least here you've heard the squeaking, so I don't feel too guilty. And you've got a copy of the car in your shop so I'm perfectly content.
For reference, my Keyboard Maestro Macros.plist is 1.1MB. I have 159 macros. Scaling proportionally, I would estimate you to have 11,450 macros.
For further reference, I have 873 and my macro plist file is 2.1MB.
I do have a lot but I don't think I have that much. SO something may be wrong in that file. Should I try to read its contents? How would I reset it?
Yes, that is large. Every change in the editor is made by writing the entire file. So you are basically saving a 72Meg file for every change.
Mine is about 1.5MB and I have about 1500 macros.
Probably you have one or more macros with a large image (or maybe large styled text) in it. Note that custom images pasted in to icons could also count here, although any icons generated by Keyboard Maestro or from the icon chooser will not count for much.
Realistically, the design for all this was appropriate for a few hundred macros, and it works reasonable with a few thousand, but if you start getting way past that or start having large amounts of data stored in macros, then it is going to bog down. Maybe one day I will rewrite it to use a database of some sort, but that will have its own problems (like database corruption for example, which is rarely a problem with Keyboard Maestro since each write is the entire data).
There is no easy way to determine which macro is large, probably the easiest way is export your macro groups (perhaps in sets) and see which ones produce a large export file - the export file will be comparable in size to the size taking up in the Macros plist file.
I downloaded the Xcode components required to view the plist file. That took 15 minutes. I've opened the file. It's some sort of tree data structure. But I don't know what this structure means. I see a folder at the top called MacroGroups. I'm starting to decode what I see, but now I notice Peter's new post so I'll stop writing this post and study his interesting response.
Yes, everything you said makes sense. I'm not sure what the 72MB is caused by, but I do use FindImage a lot so I'm guessing I have an excess of image data in there. I think for me the quickest option is to find some older folders that I'm not using and see if I can export and delete entire folders. I didn't know you can do that in "sets". But I need to figure this out.
Currently, Find Image pretty much has to have the image in the action, although the images tend to be relatively small, they can add up.
In Keyboard Maestro 9, you can have the image stored elsewhere, in a file or a Named Clipboard, either of which would remove it from the macros file per se.
Yes most of my Find Image images are small, like the ones I use in my OCR utilities to convert bitmapped letters in screen images back to text, but it never occurred to me that size matters, so it's likely I have some large ones that are using a fair chunk of my 72 MB. I'm trying to do searches (for "screen"?) right now in the Editor to see if I can find the main culprits. But I live in a time zone where it's about time to go to bed so I'll have to try this another time.