How to Generate a Memory Leak in the Keyboard Maestro Engine

Images are stored as compressed tiffs.

If it is growing that much that quickly, there must be stuff going on a lot in the hour. What is happening over that hour? Presumably you're not just sitting there toggling the enable on a macro group all hour.

If you can narrow down the behaviour that causes the leak, then I can have a chance to fix it.

You are right I'm not just clicking on the button to enable/disable groups. Instead I am writing macros that take plain ascii text out of old public domain books and paste them into Keynote for easier viewing. It's just basic text processing. With a little bit of Application switching. There are no images involved. But as far as I can recall I've had this problem for years, every day, no matter what kind of macros I'm working on. I don't recall if the memory is leaking when I'm not using the computer, I should take note of that. I'll make note what the memory level is at when I'm done for the night and check in the morning.

I'm not really sure what to suggest. Your experience seems quite different to other folks, so that is why I'd like to know what is different about your Mac or your macros or your behaviour that results in such a significant leak.

I have two big old Mac computers sitting next to me and I could turn them on and see if they also have the problem, even though they aren't sharing the same collection of macros that I have here on this Mac. Which might make it a good test. Maybe tomorrow, tonight's almost done.

So, the problem does persist across OS updates? And over different KM versions?

Since I don't really pay attention to specific release dates for OSs or for KM, I should be cautious saying yes here. But I think if there's a way to search these forums for old posts from old users like me, you will see I specifically raised this issue years ago. So if we can find those posts we can put a minimum duration for this problem.

Overnight, with all macros disabled, my KM Engine levitated at its new baseline of 44 MB ram.

Your oldest post under your current name is from 2017. But I don’t know if you have posted before with a different name.

Here are all your topics, here all your replies.

Quickly skimming through your topic titles I couldn’t find anything related. So I guess it’s hidden in some reply.

OK, found a post from April 2018, where you briefly mentioned a memory leak:

It’s a VERY efficient action/condition. I would call it magical. I love it. I use it so much it could be the reason I am experiencing memory leaks with the KM engine.

This was clearly before Mojave (10.14). So, if it was the same problem, then it would have persisted through the Mojave update.

So, my guess would be that it is related to some older macro(s) from 2018, maybe in interference with another application that you are using frequently for the last year.

Your mention from 2018 was in the context of the Find Image action, which is one of the more “heavy-weight” KM actions.

I found two more posts talking about memory leaks, and even PeterNLewis chimed in in them:

If all my macros are disabled, and I don't use the editor, the Engine stays low around 44 MB. But if I use the editor for an hour, like I was doing today helping people on the forums, the memory climbs, and right now it's over 1 GB. That's not enough to halt my system, but after another hour or two it will be at 3GB and I'll notice my system dragging down so I will restart the engine by then.

The funny thing is, there are only two macros active in my KM right now. I have every group disabled, except for the one macro I'm using to test things to help people on this forum, and one macro that maps F12 to "Cancel all macros" which I consider essential in case I end up with a macro in an infinite loop while it's grabbing my mouse and I otherwise lose control of my computer.

That discussion doesn't have anything to do with a Keyboard Maestro memory leak. At all. I see the link to this discussion but it seems entirely our of place.

Really? You don't consider this to be discussing a leak? ... "More commonly, perhaps once or twice per week I get into the situation where the KME is “not responding” and I notice it has grabbed 4.5-6 GB of RAM. So I have to restart the engine regularly."

The discussion I quoted, Sleepy: ( Slider support to display and modify variables Questions & Suggestions). Not this one.

I see this relevant quote on that page,... "the KM engine hangs up and I have to Force Quit the KM Engine (with up to 8GB of memory assigned to it)." That sounds like it's also about a memory leak to me.

Sorry, I saw both links in your response so I thought you were referring to both links.

@Sleepy, it seems you have really copied slightly wrong links.

But in the vicinity of the posts you have linked to I found some interesting statements by you and others:

[You, March 2018] perhaps once or twice per week I get into the situation where the KME is “not responding” and I notice it has grabbed 4.5-6 GB of RAM

and @peternlewis replying:

Most of the serious memory leaks in the past have been related to image searching

This was indeed around the same time as your memory issues with the Find Image action.


[You, March 2018] Javascript is ugly and when I make a little typo in my Javascript the KM engine hangs up and I have to Force Quit the KM Engine (with up to 8GB of memory assigned to it).

Is this still the case? This again was around the same time, but it seems to be completely non-related to any image actions.(?)


BTW, @JMichaelTX gave you a good tip here, concerning variables:

That makes me think you must have some KM Variables that are storing very large amounts of data, maybe even binary data. I suggest you review your KM Variables, and Named Clipboards, in the KM Editor Preferences, and delete everything that you don’t need. If you need to store large amounts of data then do so in files, NOT KM Variables or Named Clipboards.

Have you thoroughly checked your variables and clipboards? I’m asking because in your reply you only said that you have 450 variables and that you think that your largest variable is one with 500 lines of text.

In case you still have 450 variables, I really would thin them out. Maybe you have some variables that are consuming lots of memory because of some accidental loop in one of the macros. But with 450 variables it is not easy to keep the overview.


And in the same post (from March 2018) you said something very interesting:

this problem has been around for years, even on previous hardware before I got his iMac

So, this would mean that the issue has not only persisted through the Mojave update, but is/was also present on different machines(!)

Weird, or maybe not. Assuming you are migrating your Macro library between machines and OS (and KM) updates, this really makes me think that the problem lies somewhere in your macro collection…

Some of your points are good, and I will try to answer them.

Firstly, I opened the two links provided my mrpasini and scrolled down until I saw the word "memory" or "ram" and that's why I say that those quotes came from those pages. If people here want to say that I provided irrelevant links or the wrong links I'll just let my quotes about memory and ram from those pages speak for themselves. I won't argue this. The quotes I gave are from those pages.

Regarding the Javascript issue. I don't recall ever using Javascript since that time. I probably gave up on it since, as I wrote "when I make a typo in it the KM Engine hangs up and I have to Force Quit it." But I continue to assert that every hour since spring 2018, the KM Engine has been growing at a rate of 1 GB ram per hour, roughly. After about 3 hours my system feels sluggish and I check the Activity Monitor and I see the Engine is up to 3GB again. This has nothing to do with Javascript, as I stopped using that last year, and it has nothing to do with the Find Image action, as I showed this week that I get similar results when I disable all my macro groups and just click on buttons in an editor with no groups enabled. I reported that I deleted about two thirds of my 1800 macros and I'm still getting similar results, even though the baseline RAM has dropped from ~150 MB to ~50 MB.

One thing I didn't do was delete my variables. That's a good point. The Preferences page shows about 17 variables per screen, and I count about 48 screens, so that's 800 variables. So I went back to the post by JMichaelTX. He gave me a script to check my variables with. I responded that "For that to work I would have to install the BridgePlus Script Library. I will need to consider who wrote it, and what it does before I install it. That would be a lot of work, so I just decided to solve the problem using KM itself." So it appeared I solved the problem using a different method. But I didn't follow through by deleting my variables. Not sure why. Are you suggesting that if I delete some variables the leak will stop? How low do I have to cut my variables to be considered reasonable? What number of variables is too many? In any case I'll go delete some (although they may get recreated when the macros run.) And I've started using more local variables in my macros over the past few months.

I don't recall clipboards being discussed before. I'm not sure if I've ever used them, other than perhaps by accident or while experimenting. I think I have only a couple of them.

I don't use KM every day, but the moment I do, the Engine starts grabbing memory at a steady pace. I'd estimate I have been restarting the engine every couple of hours for the last couple of years. But I still love the product! I can live with this minor issue.

I don't think I transferred any macros electronically from my old to new computer. I enjoy rewriting macros and I think I probably just rewrote them by hand. That does't mean the issue isn't in my macros. But I did point out that even when all my macros are disabled I can cause the memory leak to go to 3GB in a very short time just by clicking on buttons in the editor.

Firstly, I opened the two links provided my mrpasini and scrolled down until I saw the word "memory" or "ram" and that's why I say that those quotes came from those pages. If people here want to say that I provided irrelevant links or the wrong links I'll just let my quotes about memory and ram from those pages speak for themselves. I won't argue this. The quotes I gave are from those pages.

???

Well, just click your links here and you’ll see what I meant.

But that’s OK, with the help of the forum’s in-page search I found the right posts. So everything fine.

The Preferences page shows about 17 variables per screen, and I count about 48 screens, so that's 800 variables.

Not sure what you mean with screen. To quickly count the variables, just select the whole variable list (⌘A), and copy/paste into a text editor that shows you the number of lines.

Are you suggesting that if I delete some variables the leak will stop?

No. As mentioned, which such an amount of variables it might just be a bit difficult to keep the overview.

although they may get recreated when the macros run

Exactly. Normally you can delete almost all variables, maybe except for some that hold some complicated macro settings. And if you’ve started to use more local variables, then you will have seen that global (=persistent) variables are pretty rarely necessary.

I don't recall clipboards being discussed before. I'm not sure if I've ever used them, other than perhaps by accident or while experimenting. I think I have only a couple of them.

They are also in the Preferences, the tab just to the left of the variables. Basically clipboards are also variables, but they can be larger, because they can hold other stuff than only text.

Another thing you could do, is checking the size of your variables database (~/Library/Application Support/Keyboard Maestro/Keyboard Maestro Variables.sqlite).

I can live with this minor issue.

I wouldn’t call this a minor issue.

I don't think I transferred any macros electronically from my old to new computer.

Well, this makes the story really weird. I don’t see any pattern then, that could explain why you have this issue over several years. Because if you had a very buggy macro, chances are rather low that you will copy the exact bugs when re-writing the macro on your new Mac, no? (But it can’t be excluded, of course.)

But I did point out that even when all my macros are disabled I can cause the memory leak to go to 3GB in a very short time just by clicking on buttons in the editor.

Yes. Here and there you said “with all macros disabled”, then you said (in the same post) “I have every group disabled”.

You know that also a macro in a disabled group (and even a disabled macro) can perfectly be executed, for example by another macro.

So, disabling a group (or a macro) does not automatically prevent a buggy macro from causing havoc.


But, somehow, I don’t think, this is it.

You have the memory issue with image actions, you had the issue while running JavaScript, you have the issue when only enabling/disabling groups. The issue seems to be hardware-independent and OS-independent, and it goes away when doing nothing with KM.

Well, I think, I can’t help you much here. But I’m very curious to learn what you’ll find out.

Maybe it helps, next time when the memory footprint goes up to several GB, to make a sample (via Activity Monitor) and send it to @peternlewis . (?)

I guess, the Engine log files, you have already checked?


PS:

Just in case you have any weird anti-virus software running, then I would disable it and see if it changes something.

Just a few quick answers. Some of your questions may be too deep for quick answers.

Peter did ask me about Engine logs before, and if I recall I did tell him what was in there.

No, I don't have anti-virus software.

No, I didn't know disabling a macro group didn't disable its macros. Obviously there are some elementary things about KM that I just don't grasp yet. So I may have misspoke because I didn't know the difference. I thought that the purpose of disabling a group was so that you didn't have to go manually through a list of 100 macros and disable them manually and separately.

You think that the chances are low I wold retype a bug into a new Mac with KM. I see your logic, but you seem to be excluding the possibility that the bug is in KM itself. I'm careful not to blame KM in any way, because that would seem accusatorial, but I think it's a real possibility.

Thanks for the tips. I didn't know about the Variable database. I see it now, mine is 4 MB. I trust that's reasonably small.

Since I'm poking around in that folder, I notice there is a folder in there called Revisions Version 8. That folder is over 11 GB. Could that be a factor? Looks like a backup folder, I can't see why that would be a risk here.

Your idea of copying and pasting the list of variable into a TextEdit program doesn't seem to work correctly because not all variables are separated onto their own lines. I see 400 lines, but a few hundred are on the same lines as their predecessors.

Peter did ask me about Engine logs before, and if I recall I did tell him what was in there.

And I assume he couldn’t find anything suspicious.

No, I didn't know disabling a macro group didn't disable its macros. Obviously there are some elementary things about KM that I just don't grasp yet. So I may have misspoke because I didn't know the difference. I thought that the purpose of disabling a group was so that you didn't have to go manually through a list of 100 macros and disable them manually and separately.

Actually I’m not quite sure if there is a difference between a disabled macro and a macro whose group is disabled. I thought there is one, but can’t find any related information at the moment.

In any case, the point is that disabling a macro or disabling its group does not prevent the macro from being executed. You are just disabling the trigger of the macro, but the macro can still be run, for example via an Execute Macro action by another macro.

I see your logic, but you seem to be excluding the possibility that the bug is in KM itself. I'm careful not to blame KM in any way, because that would seem accusatorial, but I think it's a real possibility.

No, I’m not excluding that. But 1) the very remarkable circumstances/history of your issue, 2) the absence of any coherent pattern and 3) the factor that you seem to be the only one on this forum suffering from such a memory leak in the past years, makes it unlikely that KM is the (main) culprit.

There must be some very important factor involved, something very specific to your software/system setup. And if you are able to find that out, I’m pretty sure that Peter will do his best to make KM more resistant to that factor.

If I were you I would start the Mac in safe mode, and see if I can reproduce the issue there. (Booting in safe mode takes a bit longer than a normal boot, and you should bring in some time to test different scenarios/actions.)

If the issue is caused by interdependencies with some other process that is running in the background, then chances are good that you can catch it this way.

Since I'm poking around in that folder, I notice there is a folder in there called Revisions Version 8. That folder is over 11 GB. Could that be a factor? Looks like a backup folder, I can't see why that would be a risk here.

Yes, these are old versions you can restore in case of unfortunate things.
Since you said that you often work with images in your macros, I think, this can explain the 11GB.

Your idea of copying and pasting the list of variable into a TextEdit program doesn't seem to work correctly because not all variables are separated onto their own lines. I see 400 lines, but a few hundred are on the same lines as their predecessors.

This sounds odd to me. I just double-checked on my system, and definitely each variable name is terminated by a line break.

With your variable names that are concatenated to one line, is there any delimiter in-between the variable names? (You might have to enable “Show Invisibles” in your text editor, or save the file and upload it here (or via PM).)

Into which text editor did you paste the copied list?

Thanks for your patience with me and your answers. You are indeed trying to be helpful.

I used TextEdit. I aw no delimiter between variable names that were compressed onto one line. I should note that the first ~120 or so variables were correct, the, the next ~120 were compressed onto a single line. This repeated about three times until the end, at which point all ~800 variables were displayed. Since TextEdit is pretty reliable, I'm guessing that the CMD-C in KM Editor failed to insert the CR's.

You said "you seem to be the only one on this forum suffering from such a memory leak". Well, there were some people on this forum that pointed out they could get substantial memory leaks by following my instruction of clicking Enable/Disable. I think one user even said his engine reached 8 GB at one point in the past. That shows I'm not the only one badly affected.

Overnight my KM Engine memory hit 700 MB. The system doesn't feel sluggish yet, but it's time to restart the engine.

I really like your safe mode idea. Great idea. Great suggestion. I love it.