Tight loops cause Execute Shell Script actions to fail

If I run any Execute Shell Script action in an infinite loop, that macro always fails/crashes, on average within 30-60 seconds. Never longer than a minute (your results may vary). Like this trivial one:

image

There are no other macros running at ANY time, which I am certain of because I have the "Pause New Macros" flag open in the debugger, (as you can see below) and no other macros appear.

If you insert any other actions inside that loop, the failure rate drops a lot. It's hard to get it to fail if you insert any statements that take time to execute, like playing a beep sound.

I thought I could fix that by modifying the flag on the action so that the action does not cause the macro to fail, but then something even worse happens!! The macro just "pauses" on the action in question. Here's a screenshot of the frozen macro:

It just sits there in the interpreter in the running state (the blue Play button indicates it is running, but it isn't running.) Clicking on the Play button doesn't wake it up. Clicking on the Pause button followed by the Play button also does not wake it up.

Oddly enough, there is one crazy thing I can do to wake it up, and this ALWAYS works. If I open a macro in the GUI and click the "Try" menu item for any Execute Shell Script action (only that action will work, nothing else works, although any other actions other than Execute Shell Script actions will still run, so the KM Engine isn't dead), like the same empty action below, then the one that is hung instantly wakes up, and the macro continues (although it's likely to freeze again within 10 seconds.)

So one thing I have experimented with to ensure that my Shell actions never get hung is to put the above empty shell command into a macro that triggers periodically every 1 second. That will wake up the hung shell and result in one more iteration of the loop (sometimes more than one), after which the original macro hangs itself again. So that's not a proper fix.

So to summarize:

  1. Any Execute Shell Script action fails about 1% of the time when placed in a tight loop.
  2. By using the flag on the action, you can convert the failure into a freeze.
  3. By having an external macro trigger every 1 second on an empty Execute Shell Script, you can get the frozen actions to unfreeze. Sometimes unfreezing lasts for only one iteration of the loop, and sometimes it lasts for hundreds of iterations.

Now for my opinion. Well, this paragraph used to have my opinion but I delete it because I really don't know what's going on here. I'm not sure if macOS is failing to respond to the KM Engine, or if the KM Engine is getting into a bad mood and needs prodding to wake up.

I need to be able to use Execute Shell script actions in a loop, (I've been trying to use KM to create a new interpreted language) and for weeks I've been experiencing this problem without fully understanding it. Now with this diagnosis I think I understand the problem better and I think we need the experts here to test this themselves.

When you see “fails/crashes”, what exactly do you mean?

What “freezes”?

To be clear, crash and freeze have specific meanings:

Crash: the Keyboard Maestro or Keyboard Maestro Engine application quits, leaving behind a diagnostic crash report in the ~/Library/Logs/DiagnosticReports folder.

Freeze: the Keyboard Maestro or Keyboard Maestro Engine stops working, but remains visible. It remains this way, doing nothing, for an extended period of time.

I would fully expect this macro to fail eventually. The reason is that executing a script is a very expensive operation, and uses a bunch of different resources, all of which are finite. Things like process ID numbers. This system only has a finite set of these available, and while they are recovered eventually, they will run out if you use them faster than they are recovered. After that the system will not be able to execute new processes for some period of time while it recovers - and anything could trigger it flushing them out and recovering, which probably explains the various odd recovery behaviours you have seen.

While none of these should crash or freeze Keyboard Maestro per se, the fact that a previously trivial action cannot now complete at all may result in a crash or freeze (for example, the application might try to execute its own script to do something it expects to trivially resolve, but that fails unexpectedly).

There are only two possibilities. Really only one.

The first (which wont likely work) is to figure out what resources are being used up and try to configure the system to have more of them. Good luck with that. I would have no idea how to do that, and it may not even be possible.

The second, is the same answer the Doctor will give you if you go to him and tell him that your arm hurts if you twist it hard behind your back - stop doing that.

There really is no other solution - don't engineer macros that continuously execute scripts at high speed indefinitely.

1 Like

When I said fails/crashes, I meant "fails" as in triggers the KM fail flag. I shouldn't have said crash. It's a regular fail, which can be tested with the condition that tests for failure. I should have only said "fail." I meant "fail" in the normal KM sense of the word.

When I say "freeze" what I meant was that the KM engine fails to move to the next statement. It fails to move to the next statement and you can't make it move to the next statement by pressing the blue Play button. Or by pressing the Pause/Play button. It simply will not continue execution. As I said above "any other actions other than Execute Shell Script actions will still run, so the KM Engine isn't dead" which means the KM engine wasn't frozen, it simply refuses to execute any Shell Script action in any macro until I prod it a few times manually.

This macro is the simplest case I could come up with that fails. I don't have any intention to run this macro. I have much more complex macros that do many things between shell actions. I had no idea that there was a maximum number of times per second that you could call shell actions. I thought when one terminated, that all resources would be cleared. If there's some limit, would it be safe to say that I can use the Shell Script action at least ten times per second, or is that above the safe limit? I suppose I could convert some of my Shell actions to KM actions, would that be easier on system resources?

I didn't make this macro to find some way to make KM fail. I did it to find the simplest routine that's guaranteed to fail (because I was having problems!) So the advice to "stop doing this" doesn't really apply. I was being much more judicious in my use of Shell Scripts. But I must have exceeded some threshold. Would the logs be able to tell me how often I was executing the Shell Script action?

Probably. It's hard to know what the low level stuff does.

It depends on the actions - actions that execute AppleScript will execute a script to accomplish that for example.

No. You could add a Log action for each executed script, and then you could see them in the log file, and get some idea of how many per second are running.

It is a good macro to produce the issue. I will look in to it further in due course and see if I can find any way to prod the system in to releasing whatever it is holding, or wait until it has or some such.

I left your macro with a 0.1 second Pause in the loop going and it seems to continue fine, but maybe it will just take much longer to fail.