How to Generate a Memory Leak in the Keyboard Maestro Engine

Questions & Suggestions
91

Sorry, I was away for a couple of days.

Did you find out the source of that corruption? (Most likely this is your problem.)

Another thing that can help – not only in this case –, is running a little diagnosis.

Download the free version of EtreCheck and run it (choose “No problem – just checking”). When finished review the results; check the warnings that are displayed and look out for anything unusual.

If you want you can upload the report here, or in a PM to me.

There are two ways to do that:

  • Complete report: Go to File > Save and upload the saved file.
  • If you think the report may contain sensitive information, then just copy the text:
    1. Click the Share Report button in the toolbar and select Copy Report.
    2. Paste it into TextEdit.
    3. Remove any sensitive information from the text. Place a note where you have removed something, so I know that the section is not complete.
    4. Save the text file and upload it.

EtreCheck does not provide very in-deep diagnostics, but it gives a good overall-view of what is going on on the user’s box. And this is very important. (Heck, I’ve seen setups with three different anti-virus programs running in the background, and the guy wasn’t aware.)

PS:

I just noticed that I asked you a couple of times if your “disable/enable groups” method still triggers the memory spike in Normal mode now (after having been in Save mode).

Any info on that? (Sorry, if I missed the passage in your texts.)

92

Thanks for your concern. Part of the reason it's hard to make progress is that it takes exactly 20 minutes to boot to Safe Mode. Another part is that under the advice of Peter the testing in Safe Mode is done under a new macOS account, which is a clean account. It was done that way on purpose so I wouldn't have 800 macros in my KM.

The first time I did this, for my new account in Safe Mode, the Engine took 25 MB. Then when I tried that same account under Normal Mode it was quite a bit more then when I switched back to Safe Mode I think I reported 140 MB for the Engine which didn't make sense because it was a fresh copy of KM on a fresh MacOS account with no macros at all.

However today I rebooted into Safe Mode on my new account, which is still clean and fresh like a Downy towel, and I'm at 21 MB which is good and healthy.

At one point I said I may do a binary search of enabling/disabling all my macro groups in my normal macOS account to find the corrupt macro, if any, but I haven't done that yet. I think that's what you want me to do.

I guess I can run EtreCheck if you recommend it, which you are doing (I'm only 90% paranoid.) I presume you want me to do that from my normal account, not my clean account. I'll tell you what. I'll download it onto my clean account now, run it, and see if I'm willing to run that on my normal account. I think you're primarily interested in the data from my normal account, but it sure couldn't hurt to have a clean baseline for comparison. So I'll start with that now.

93

I generated some EtreCheck results and wanted to take you up on your offer to PM you the results. But when I tried to do that it appeared to post my Mac's results right here in the public forum. I think that's a bit too public for me. So let me do some research to see if I can figure out how to PM you.

94

Back on my regular macOS account where all this memory leak started, I'm still getting a leak in the Engine. The amount is exactly 24 MB per minute of clicking on macro group checkboxes. After about 30 minutes I was around 600 MB. It was always going up, never down at all.

After the first five minutes I realized manual clicking was ridiculous and so I made a macro on my Rival 500 mouse to repeatedly click wherever my mouse pointer was.

Okay I've gone through another major cleanup, exporting all but 257 of my macros. (I think I was around 800 earlier this evening.) Now I'm down to 9 MB per minute lost.

Think about that. I've removed exactly two thirds of my macros, and my accumulating memory loss has dropped by exactly two thirds. This suggests it's not one of my macros per se, but exactly linearly proportional to the number of macros.

Why don't we just have me start afresh? I could delete (ie, export) all my macros, delete KM, re-install KM, and re-install macros as needed? Does anyone object to that?

95

You did it right, I think. I see a deleted post in my private forum mailbox, and before I received the text as mail. Anyway, you have to click my avatar and select “Message” there.

But better upload the zipped report, instead of copy/pasting the text. (Just drag the zip file into the editor window of the private post.)

And please run EtreCheck with administer privileges, otherwise many infos are missing.

96

Because we noticed that your issue persists across different hardware. So, you did already a fresh KM install in the past. In addition, you said that you use to re-write the macros on new machines.

But, of course, try it again. If you export/import the macros, it’s a thing of 5 minutes.

But don’t forget to wipe KM completely, that is, have a thorough look through your ~/Library, especially Application Support and delete everything related there. Also the preferences plist in ~/Library/Preferences. (Back up the files before to some other place.)

97

Once again, sorry for the delay. And, in addition, it seems that I missed some points in your posts when I read them the last time.

So, one thing after the other:

Part of the reason it's hard to make progress is that it takes exactly 20 minutes to boot to Safe Mode.

As mentioned by me and also by @peternlewis, safe mode is not a thing to live in. You’ve already done some tests in safe mode, and the confirmed conclusions were:

[@Tom:] So, to make a short intermediate resume:

By just “using” KM Editor in safe mode for a couple of hours the engine stays at around 500MB, with some minor “spikes” up to 830MB. You are unable to bring it up to several GB, as you can do it in normal mode.

Is this correct?

I assume you have also tried the “repeatedly enable/disable groups” test in safe mode?

You confirmed this in your follow-up:

Everything you wrote is correct.

I know, at that time you haven’t spent much time in safe mode, I think, 20 minutes you said. But I skimmed thru your later posts and I couldn’t find any mention that you succeeded to reproduce a memory leak of 1GB or more with subsequent safe mode tests. (Correct me, if I missed something.)

So, we have already a pretty heavy clue here: The issue doesn’t seem to be a KM-“standalone” issue. (Otherwise it – likely – would have appeared in safe mode, too.)

Your results in a vanilla user account seem to be similar, with or without safe mode.

So, the pretty logical conclusion was: there is one or more third-party players in the game.
[this part is in bold, because it is a very crucial assumption, based on your results, as I interpret them.
If this is wrong, you have to tell me.]

That’s why I asked you to run the EtreCheck diagnostics. To get a rough overview of what is going on on your computer.

[you:] I presume you want me to do that from my normal account, not my clean account.

Yes, because with the problem (> 1GB issue) not being present in a vanilla account and/or in safe mode, it wouldn’t make sense to run the diagnostics there.

And, if you are still willing/motivated to track down the problem, it would be helpful to run the diagnostics as admin. Otherwise, lots of infos will be missing.

(As said, in case of privacy concerns, upload the zipped result as a PM to me and/or redact the sensitive parts before; leaving a note in the text where you have removed something.)

– Tom

98

I am very appreciative that you are trying to help. Sometimes it's best not to do too many things at once because then you don't know what fixed it. In the last few days I've removed ALL my global variables, and clipboards, and most of my macros. I don't think any of my remaining macros are triggered by anything I'm doing (except I'm always working on one new macro.) AND YET the problem is still HUGE. Within a minute of my last login my KM Engine memory flies up (as I start to run a macro I wrote yesterday) from a small starting value of about 40 MB up to over 1 GB. Then it fluctuates wildly and maybe settles around 200 MB but refuses to respond to anything including clicking on the engine icon up in the system try in the upper right corner of the OS. The Engine is dead. I wait a few minutes and there's no response so I force kill it. I probably had to kill the Engine about 5 times today. I tend to notice it when it reaches about 1 GB and becomes sluggish.

When you say there's a third party player it seems that you think something else running on my system is somehow causing the Engine to leak. It's possible.

Today I spent most of my day writing a KM macro that lets me edit/filter my global variables, and even my text clipboards and to a degree, some text files. I haven't fully debugged it, but what I've tested works great, and I was going to upload it when it's all done. I don't see how it would help to upload this macro right now. I can't see how a basic macro that processes small text files with filters could possible cause 1 GB Engine memory leaks.

So I think what I'm going to do now is reinstall KM. You gave me some tips above to make it a clean install. After that I'm thinking of using my clean new macOS user account to do all my KM development. If it still doesn't work then, we can consider EtreCheck.

99

Yes, to find this out was the main point of doing the safe-mode experiment. And according to your reports it seems that this is indeed the case.

If this is true, then it is probably not very efficient if you continue to search for the cause inside KM (variables etc.).

100

I have found what might be an important clue. I decided to open the debugger and see what actions were running when the leak was occurring. Not sure why I didn't think of this before. It turns out that there's only one type of action that is running when the Engine hangs up for minutes at a time (and the memory leak tends to occur during these freezes). That is the Search and Replace Variable action. I use these actions from time to time in various macros. Usually they work 100% fine. But sometimes they freeze the Engine for 1-10 minutes. Here's the one it's frozen on right now for over 5 minutes:

image
image.png978×458 30.8 KB

In English, that action replaces "a space followed by a vertical bar" with "a space followed by a lowercase l". Is there something about that particular action that is the direct cause of my Engine hangups and memory leaks? It's not even a Regex search!!!!

The reason that the name of the action starts with "a1" is so that I can see in the debugger which Search and Replace action is hanging the Engine.

Okay while I typed that the Engine has completed that action and has moved to the next action which is:

image
image.png978×458 33 KB

This image replaces "space letter" with "letter-l letter". And it looks like the Engine is going to hang on that one for five minutes also. This one IS a Regex search.

If it was something unrelated to KM that was causing this leak, wouldn't you think that I would occasionally see other actions besides that one where the Engine freezes on? I don't use Search and Replace very much but it's the only action that the Engine freezes on.

I might wait for your thoughts here before I reinstall KM. Because this might be a big clue.

101

After 40 minutes that last Search and Replace action finally completed. And the memory for the Engine, while reaching large values during those 40 minutes, is now back down to a more reasonable 87 MB.

102

I have spent hours trying to reduce my macro to the smallest macro that causes the memory leak, or the Engine hangups. And I made a wonderful discovery. In my code fragment (above) the variable name LocalTestOCR exists. The problem was in my code the correct variable name was LocalTextOCR. When I use the correct variable name, KM Engine is stable. When I make a typo in the variable name, to a non-existent local variable, the KM Engine crashes within a couple of minutes. I'll have to do some more testing but the Engine seems to be stable now that I'm using the correct variable name, at least for 5 minutes. But I know Peter would like to see the smallest code fragment that causes KM to crash, and I'll try to create that now.

103

Okay here's the shortest program I could write that causes the Engine to crash within 20 loops. I count the loops by listening to the sound of the glass sound effect. By the time it reaches 20, the Engine is nearly dead, although the amount of memory it grabs is fairly small.

The action has to be a Regex search, although if you add a non-Regex search to the loop that will slow/hang too, it just won't contribute itself to the memory loss. You need the vertical bar to make it crash.

image
image.png1458×880 74.8 KB

It's entirely possible that many of my thousand macros use these Regex actions. I really never considered that as a problem before today. I supposed I could use awk and sed in place of the Regex action in KM, I just didn't know it was causing me any issues. I also use lots of "String contains" conditions in KM and I'm not sure if they are using Regex routines "down under."

104

I don't think I've ever had the search action or string condition fail or slow KM down myself. What are you using for the contents of the LocalZZ variable? If you share that too, @Tom and I and anyone else who's interested can do the same test themself and report our results.

105

The contents of any local variable are empty when you start running any macro. That's important here. When I used local variables that already existed, I didn't see the crashes, as I stated in a previous post.

106

So, you didn't initialize the variable before this search? Not to be like the doctor in that old joke who tells the patient it hurts when he does this to stop doing that, but how often do you need to search an empty or non-existent variable? Even if searching them does cause a crash, if there's no practical reason why you would ever need to do so aside from a test like this, I kind of fail to see the point. Am I just missing something?

107

I appreciate your humour. It's a good joke, along with the line "if it ain't broke, we cannot fix it."

I often use variables, even local variables, to accumulate data that I collect. So it is quite common for a macro to use a local text variable and assume it's empty at the start. Should I not be taking advantage of this language feature, because doing so will break the Engine?

I suspect that you, or someone, will be testing my code, to get a second opinion. Maybe you won't get the same results. Since we don't want average people downloading code that breaks the engine, I decided it was best to just include a screenshot, not upload the code itself.

108

Philosophical questions about how the engine ought to be used aside: would it make sense to test if the variable is empty before you try and search within it for a string? It's an extra action but it might prevent the issues you seem to have been having.

109

I fully understand what you are saying. That might be the best solution. On the other hand, this might be an actual bug in the Engine that can and should be fixed, in which case I don't have to modify hundreds of macros.

Do you think KM should allow us to use empty local variables? Because it's not just at the start of a macro where I would have to test this, it's every time I use the variable inside the macro. Every single time. Do we really want to have to write extra code to avoid using an empty variable in a common KM action?

I have roughly 1000 macros (most of which are archived at the moment) and if I have to modify all of them to check if a local text variable is empty before I use it in an action, that would be a terrible problem for me.

110

I'm a little surprised nobody has tested my Engine crash macro yet.