The application is frozen, where are the variable files located?

[sursur]

KM is frozen, and even if I close and open it, engine and/or editor, it stays frozen. I think it’s because I made a regex that was too complex set to search a variable, and the variable input was too large, causing the nonstop freeze (since when I open it back up each time I think it’s still trying to do that just I guess to show what the possible output would be on the action box on the app), so I need to clear that variable to stop it but I can’t from the app itself since it’s frozen.

There must be a file with the variables where I could manually go in and delete the necessary one so the app will unfreeze.

[JMichaelTX]

@sursur

KM is frozen, and even if I close and open it, engine and/or editor, it stays frozen. I think it’s because I made a regex that was too complex set to search a variable

I doubt that a large RegEx is causing KM to freeze.
Stop the KM Engine using Activity Monitor, and then:

Launch the Keyboard Maestro editor app without starting its Engine
Hold down ⌘⌃⌥⇧ (that's Cmd-Control-Option-Shift) and click on the Keyboard Maestro app in the Dock, or double-click in the Applications folder.
This will quit the Keyboard Maestro Engine and start the Keyboard Maestro app without starting its Engine.

You can then change the macro and/or KM Variables using the KM Editor.

[peternlewis]

@JMichaelTX

I doubt that a large RegEx is causing KM to freeze.

It is certainly possible, you can create pathological regex that have exponential time requirements.

Probably the solution here is to forcibly turn off edit mode so Keyboard Maestro editor probably wont try to evaluate the regex when you launch it. Run the Terminal command:

defaults write com.stairways.keyboardmaestro.editor MacroWindowEditState -bool NO

Then launch the Keyboard Maestro editor. Hopefully it will launch without freezing. Then you can delete the macro in question. If the macro is complex and you just want to delete a single action, that would be more challenging, but you can probably export the macro, delete the macro, hand edit the XML of the export to remove the pathological regex, and then re-import the macro.

[peternlewis]

And if the above it not the issue, then you’ll want to get a sample of the Keyboard Maestro editor when it is frozen using the Activity Monitor and send that to support and I can take a look.

[JMichaelTX]

@peternlewis

It is certainly possible, you can create pathological regex that have exponential time requirements.

No doubt it is "possible", just as infinite loops are possible, just not very likely.
But don't most modern RegEx engines trap for this, timing out after an excessive amount of time?

BTW, to all, one of the reasons I really like to use Local Variables is that there is nothing to evaluate in the KM Editor.

[sursur]

A late thanks, but your suggestions did help me to fix the freeze.

@JMichaelTX

I doubt that a large RegEx is causing KM to freeze.

I beg to differ! Heh. Actually, I'm sure you know much more about this stuff than I do, but something is causing KM to often go wonky on me.

It happens often enough now that I'm just used to it. I'm already well versed in forcing quit KM and the engine and restart, or at worst restart the computer. What I've found is that it's not exactly a large regex but a large variable. It inevitably won't work and I've learned that I have to keep my variables as small as possible.

For instance, say I create variable A which is very, very large and I need to search it for a lot of things. I learned long ago not to load too many searches into one variable search as that clogs things up too fast (e.g. in one variable search adding in many parentheses so that it captures multiple things in that one search). KM works much better with every variable search in finding only one thing and so I always have a separate variable search box for each one.

But even then, I have to make sure not to have too many on the same macro page, even with a not too big variable to search, or else it will often freeze up too. So I'll put say 5 or so on one macro, then execute a new macro with 5 more and copying the other variable to a new variable for those new five to search, and so on.

BUT even then, if the variable I'm searching is too large all bets are off, and even one tiny little variable search of it will completely freeze KM. It can be a bummer because this type of stuff is the majority of my KM usage, which is why it happens to me so much and why I've learned the maxes that it can go for me.

I've also figured out (better late than never) that shutting down the preferences box helps since KM seems to put more of its power into that when it's open (I generally like to keep it open to view my variables), and also that shutting down the editor while leaving the engine going also helps since KM seems to put more power into that too when it's open and it clogs things up more (I also generally like to keep the editor open while my macros are running for quick changes and such).

But still, even taking all these precautions, I have to be careful to keep variables small. What is the worst is when I need info from the bottom of a large string or text. The best I've come up with is the copy to the clipboard, then only get a substring for the variable, but for some reason KM often won't do it if the substring is too far away from the beginning, no matter how long of a pause I give it. It's led to, on large strings/text, having to use the mouse scroll/click/drag features to get the the right parts, but those take longer and require much more precision and care as it's easier for those to go wrong in general.

I love Keyboard Maestro and it's awesome, but I just felt compelled to reply about this freezing issue since it's the biggest thorn in my side with using macros, and if any of you do have any suggestions on how I can improve my workflow to minimise or eliminate that I'd love to hear it! My usage has been pretty specific so there's lots about KM I don't know yet.

[JMichaelTX]

@sursur

I doubt that a large RegEx is causing KM to freeze.

I beg to differ! Heh. Actually, I'm sure you know much more about this stuff than I do, but something is causing KM to often go wonky on me.

That is fine. That was just an opinion based on my experience in general, not on any specific testing. And let me clarity what I mean by "doubt". I mean it is unlikely that a well-formed RegEx would cause KM to freeze.

However, a poorly-formed RegEx could run excessively if it has too many feedback loops. So this might appear as a long-running RegEx Action, which would look like KM has frozen. I suggest that you test all of your RegEx at regex101.com

@sursur

What I've found is that it's not exactly a large regex but a large variable. It inevitably won't work and I've learned that I have to keep my variables as small as possible.

This makes sense. You should also delete all KM variables (or at least the large ones) at the end of your macro, if you don't need them for further macro executions.

But let me ask you, what is your definition of "large"? How many characters in your largest Variable? If possible, you might try storing the very large Variables in files, and then deleting the Variable. Or, better yet, just save the Clipboard to file as text.

KM makes it easy to:

• Write to file
• Read from file
• RegEx Search of a file

@sursur

BUT even then, if the variable I'm searching is too large all bets are off, and even one tiny little variable search of it will completely freeze KM.

Please isolate this in a test macro, with just the Set Variable and the RegEx Search, and test. If this still causes KM to freeze, please post/upload your test macro.

@sursur

For instance, say I create variable A which is very, very large and I need to search it for a lot of things. I learned long ago not to load too many searches into one variable search as that clogs things up too fast (e.g. in one variable search adding in many parentheses so that it captures multiple things in that one search). KM works much better with every variable search in finding only one thing and so I always have a separate variable search box for each one.

I find this very surprising. Could you please post a test macro that illustrates this?

In general, I would expect repeating the same RegEx Search with the same RegEx, but with just a different Capture Group, to take much longer. I don't know how the RegEx engine works internally, but my guess is that it first finds the complete match, then parses each Capture Group in that match. This should be much faster than repeating the same search multiple times. You can easily test this for yourself. Create two test macros that use the same very large KM Variable (or file). One macro should have just one Search with multiple Capture Groups. The other macro should have multiple Searches, each with just one Capture Group. In case you don't know how to time a macro, I'll post later an example of how to do this.

@sursur

What is the worst is when I need info from the bottom of a large string or text. The best I've come up with is the copy to the clipboard, then only get a substring for the variable, but for some reason KM often won't do it if the substring is too far away from the beginning, no matter how long of a pause I give it.

Actually this should be easy. You can use the Substring of Variable or Clipboard action like this:

Here I'm extracting the LAST 200 characters from the Clipboard, and storing the results in the KM Variable SourceToBeSearched. You could then use that Variable for a RegEx Search.

---

MACRO:   How to Time a Macro

#### DOWNLOAD:
<a class="attachment" href="/uploads/default/original/3X/b/4/b482be4adcce7f090af114793fc45d32d79cf5da.kmmacros">How to Time a Macro.kmmacros</a> (3.2 KB)
**Note: This Macro was uploaded in a DISABLED state. You must enable before it can be triggered.**

---



![image|496x514](upload://9L1y0hMkxRgb5Fi2q8dt09kTpnS.jpg)

---

Questions?

[JMichaelTX]

@sursur

A late thanks, but your suggestions did help me to fix the freeze.

@JMichaelTX

I doubt that a large RegEx is causing KM to freeze.

I beg to differ! Heh. Actually, I'm sure you know much more about this stuff than I do, but something is causing KM to often go wonky on me.

I'm responding to this again after I read the post by Peter:

@peternlewis

It is certainly possible, you can create pathological regex that have exponential time requirements.

First, @sursur, let me suggest that you read this blog, and take it to heart that it is certainly possible to contrive a "pathological regex".

However, it is very interesting to me that the "pathological regex" that was posted:
(x+x+)+y

does NOT actually cause a problem in either RegEx101.com or Keyboard Maestro 8.2.1 (8.2.1) on macOS 10.12.6:

RegEx101.com

---

KM 8.2.1

---

@sursur

What I've found is that it's not exactly a large regex but a large variable.

If I understand your workflow correctly, you are creating these very large KM Variables from the Clipboard, which also would be very large. I have observed that a very large Clipboard can cause KM to slow down.

So, I'd suggest that you delete ALL KM Variables and ALL Clipboard History entires that you don't need. I have tools that will do that for you:

1. MACRO: Get List of KM Variables and Size @KM @Tool

2. MACRO: Clean-up DELETE KM Variables @KM8

3. Clear Delete Clipboard History @KM @Tool

This does not include macro file, it is just an image, since it is so simple.

Warning!
This macro Makes Major UnRecoverable Changes to Your KM App

• It DELETES Your Entire KM Clipboard History
• You may want to backup your KM Clipboard History Plist at
  ~/Library/Application Support/Keyboard Maestro/Keyboard Maestro Clipboard History.plist

---

Questions?

[peternlewis]

@sursur

Actually, I'm sure you know much more about this stuff than I do, but something is causing KM to often go wonky on me.

It happens often enough now that I'm just used to it. I'm already well versed in forcing quit KM and the engine and restart, or at worst restart the computer.

This is most unusual, and something that in all likelihood could be solved.

There are basically no known cases where Keyboard Maestro or Keyboard Maestro Engine need force quitting.

As I discussed previously, a pathological regex can lock up Keyboard Maestro or Keyboard Maestro Engine, but that is a very unusual case and easily resolved.

So if you are having cases of Keyboard Maestro or Keyboard Maestro Engine regularly freezing, then isolating those causes and resolving them would be a good idea.

In terms of variables, Keyboard Maestro can handle large variables without any great problem (large as in say a MB). If you want to store larger data than that, then you should consider using a different method. Similarly, it can store large clipboard data, but again, very large clipboard data (dozens of MBs for example) would not necessarily be ideal.

@sursur

BUT even then, if the variable I'm searching is too large all bets are off, and even one tiny little variable search of it will completely freeze KM.

this is simply not true in general, so you should come up with a case that causes this and post it. Either we can assist with resolving the issue, or we can determine that the issue is specific to your Mac and perhaps there are other things going on. Either way is better than just force re-launching Keyboard Maestro.

@sursur

I've also figured out (better late than never) that shutting down the preferences box helps since KM seems to put more of its power into that when it's open (I generally like to keep it open to view my variables), and also that shutting down the editor while leaving the engine going also helps since KM seems to put more power into that too when it's open and it clogs things up more (I also generally like to keep the editor open while my macros are running for quick changes and such).

Except for updating action condition/results live, neither of these should have appreciable affects on your system performance.

@sursur

I love Keyboard Maestro and it's awesome, but I just felt compelled to reply about this freezing issue since it's the biggest thorn in my side with using macros, and if any of you do have any suggestions on how I can improve my workflow to minimise or eliminate that I'd love to hear it!

If everyone’s experience was like yours, I doubt there would be many people using Keyboard Maestro. So something else is going on, either in how you specifically are using Keyboard Maestro, or on your Mac. Figuring out what is going on should improve things dramatically.

[sursur]

Thanks for the replies, and I'll respond to more when I have more time and a chance to try some of these things out; I'd definitely love to get this fixed if there is a way!

@JMichaelTX

But let me ask you, what is your definition of "large"? How many characters in your largest Variable? If possible, you might try storing the very large Variables in files, and then deleting the Variable. Or, better yet, just save the Clipboard to file as text.

Well, my definition of large depends and changes and I haven't kept track so I'm not sure what the largest I've tried is. However, just the other day there was one I gave up on because it wouldn't work and caused the freeze situation and more, and I know just what its characters were because I had to try to get character substrings as a solution. It had just over 200,000 characters. I was searching for four very simple variables in it all near the end. Doing that by searching those characters in a clipboard or in a variable if I transfer it (I always transfer to a variable before searching, but when testing I'll try in the clipboard too) always caused KM to freeze up. By 'freeze up' I mean trying to do the search (the spinning rainbow wheel when on KM) but never finishing and it causing KM to be able to do absolutely nothing else until I force quit it all and restart; I can't even clear the variable so it will stop trying to search the large amount of characters. In fact, when I restart I have to quickly delete that large variable/clipboard before KM tries to search it just to show the possible find on the editor macro variable search box.

So anyway, I next tried to get a substring of just the end of the all those characters. I did 'Get Substring' 'from to' and basically did '195000' to '199999'. When I manually would click 'try' on the action box, it would work, and fast. But in the actual macro run, it would rarely work no matter what I tried, even giving a large pause between copying the characters to the clipboard and getting the substring. The clipboard would have the characters, but the variable would not be found, it 'failed' I suppose.

Here is the pertinent basic starter (before trying this and that) workflow I did on that macro section:

-type keystroke command-a
-pause .3
-type keystroke command-c (also tried with 'copy' action instead)
-pause .3
-get substring of system clipboard from 195000 to 199999 and save to variable 'substring'
-pause 1
-search variable 'substring'

I tried it with much longer pauses since I have found before on semi-long variables KM may actually work after 3-5 seconds instead of freezing, but on this one not at all, even with over a minute of waiting on each pause. Unfortunately, I couldn't do that scroll-click-drag actions to get the pertinent section because it was always similarly large but varied enough each time and with areas that changed exactly where they were at each time that I had to be careful not to click and so it wasn't feasible to acquire the characters in that way.

I have not played around with saving to files on KM instead of variables so that's an idea I will work with. I hadn't previously because when manually saving files on my Mac (not in KM) it usually takes some seconds to create files or save to any files not currently on the open and front application, which those seconds would add up and hamper my workflows, and so it hasn't even entered my mind to try working with them alongside clipboards and variables. That was my initial reasoning but you make me think I should try them.

[JMichaelTX]

@sursur

I had to try to get character substrings as a solution. It had just over 200,000 characters. I was searching for four very simple variables in it all near the end. Doing that by searching those characters in a clipboard or in a variable if I transfer it (I always transfer to a variable before searching, but when testing I'll try in the clipboard too) always caused KM to freeze up.

Please post the RegEx you used in this macro, and all macros where KM is "freezing".
It would also be helpful if you could post the 200,000 string as a text file in a zip file, so we can test.

@sursur

-type keystroke command-a
-pause .3
-type keystroke command-c (also tried with 'copy' action instead)
-pause .3
-get substring of system clipboard from 195000 to 199999 and save to variable 'substring'
-pause 1
-search variable 'substring'

It would be better to upload the actual Macro.
IAC, it is better to use the KM Copy Action than Type Keystroke C, since it will wait until the copy is complete (very important for large files).

Unless you need the KM Variables later (after the macro finishes), then it would be best to use Local Variables.

@sursur

it usually takes some seconds to create files or save to any files not currently on the open and front application, which those seconds would add up and hamper my workflows,

I guess it might depend on the power of your Mac, but on my iMac-27 (late 2015) reading and writing files in KM is very fast.

[peternlewis]

There is no use in a Pause after the Command-A and before the Command-C, the system queues the keyboard events.

A pause after the Command-C is needed, since, again, the system queues the keyboard events, and so the target application can perform the copy in its own time. Which is why, as @JMichaelTX suggested, you should use the Copy action, which sends the Command-C, and then waits for the clipboard to change.

I performed the attached macro on a BBEdit text document that was a little over 200,000 characters long. The document had the text of your message appended a bunch of times at the end, and the result I got was:

Waiting for Copy: 0.506263
After Copy: 0.040054
Time Taken: 0.546317
Result: Here is the pertinent basic starter (before trying this …

So the initial Copy took BBEdit 0.5 seconds to perform, and then the extraction and search too 0.04 seconds.

Obviously using a different target application will result in a different amount of time copying the text, and depending on how it is copied or how large it is, might result in a different amount of time spent in Keyboard Maestro. But at least in this case, there is nothing Keyboard Maestro could do to speed up the process significantly since the bulk of the time is spent in BBEdit copying to the clipboard.

Display Text.kmactions (1.6 KB)

[JMichaelTX]

@sursur

Well, my definition of large depends and changes and I haven't kept track so I'm not sure what the largest I've tried is. However, just the other day there was one I gave up on because it wouldn't work and caused the freeze situation and more, and I know just what its characters were because I had to try to get character substrings as a solution. It had just over 200,000 characters. I was searching for four very simple variables in it all near the end.

Well, we'd still like to see the actual macro and RegEx you are using, but in the meantime, I've run a little test, using a file that is 201,009 characters, with a simple RegEx looking for text at the bottom.

RegEx:
(After you use.+)\n(.+TEXT TO FIND.+)

Conclusions

Running Keyboard Maestro 8.2.1 (8.2.1) on macOS 10.12.6

1. KM can handle very large (200K) character files easily
2. KM can run a well-formed RegEx on this size very quickly (~0.01 sec)
3. The issues @sursur is seeing must be due to something unique on his Mac. It is clearly not affecting all KM users.

---

My Simple Test

---

Results

It was even fast showing the results in the KM Editor before I ran the macro.

---

MACRO:   TEST RegEx with Very Large (200K+) Text File @RegEx @Test

#### DOWNLOAD:
<a class="attachment" href="/uploads/default/original/3X/1/9/195a65dfc1c5b01c63ac8448f86ae3d0b16714f5.kmmacros">TEST RegEx with Very Large (200K+) Text File @RegEx @Test.kmmacros</a> (4.2 KB)
**Note: This Macro was uploaded in a DISABLED state. You must enable before it can be triggered.**

---



![image|631x860](upload://qkmveyBYt0jpBNtvqpPJHjSlTXG.jpg)

---

Just so you can see what I was searching for, here's the bottom of the file:

```txt
opens a Differences window which contains both documents. 

The Differences window lists all the differences between the left-hand and right-hand documents. To see the differences in context, click a line in the Differences window; BBEdit scrolls and selects that spot in both files. 
The range of lines within each file which belong to the selected difference are highlighted with the current color scheme’s “Differences” color, while all other differences within file are drawn with a light grey background. 
Reviewing and Applying Differences 
To view and apply individual differences within a line or region (i.e. sub-line differences), just click on the triangle to the left of that difference to expand the list and select the appropriate character difference. 
Use the Apply to Left and Apply to Right buttons in the Differences window to transfer the differing text from the new file to the old file, or vice versa. After you use one of these buttons CHANGED, 
[TEXT TO FIND AT BOTTOM]
```

BTW, for anyone interested in easily generating a large text file, just export a PDF to text.  I exported the 10.1MB BBEdit Users Manual, and it results in ~700K characters.

---

@sursur, if you will post the RegEx's that you are having problems with, I will try to analyze for problems, and will try to model in a test with my data.

[JMichaelTX]

@JMichaelTX

MACRO: TEST RegEx with Very Large (200K+) Text File @RegEx @Test

@sursur, OK, I've run another test, this time based on doing a RegEx Search of the Clipboard. It was still fast, but did take ~1 sec.

---

Results

---

MACRO:   TEST RegEx with Very Large (200K+) Copy to Clipboard @RegEx @Test

Same test as above, except  for:

1. Copy to Clipboard
2. RegEx Search of Clipboard

#### DOWNLOAD:
<a class="attachment" href="/uploads/default/original/3X/8/2/825f245d54410c9e20f020fd9c3ec7e74172da41.kmmacros">TEST RegEx with Very Large (200K+) Copy to Clipboard @RegEx @Test.kmmacros</a> (5.8 KB)
**Note: This Macro was uploaded in a DISABLED state. You must enable before it can be triggered.**

---

### How To Use

1. Set focus to text area of document of chosen app (I used BBEdit)
   * Make sure you have clicked in the text area of the  document.
2. Trigger macro

---

![image|526x1385](upload://nsEpJTpytxQsE2bd4jzecP0V9vW.jpg)

[JMichaelTX]

@peternlewis

There is no use in a Pause after the Command-A and before the Command-C, the system queues the keyboard events.

Peter, FWIW, I found that a pause was necessary between Edit > Select All and Copy, else the BBEdit Copy menu would not be enabled, and the Copy would fail. So I used:

[peternlewis]

@JMichaelTX

Peter, FWIW, I found that a pause was necessary between Edit > Select All and Copy , else the BBEdit Copy menu would not be enabled, and the Copy would fail. So I used:

Yes, those are not keyboard events.

If you do Command-A, Command-C, or Command-A, Copy, there should not be any need for pauses because the keys are processed by the target application in its own time.

That said, nothing stops the application from processing the key and then deferring the action and processing the next key and messing up. It's just unusual.

[JMichaelTX]

@peternlewis

If you do Command-A, Command-C, or Command-A, Copy, there should not be any need for pauses because the keys are processed by the target application in its own time.

Thanks for sharing this, Peter.

Now when I use:

The time is reduced from ~1 sec to 0.31 sec: