There is no reliable way to do this.
There is no functionality in launchd to do this.
The right solution would be to figure out why it is happening. I have to say that I can't remember the last time my Keyboard Maestro Engine was in an unresponsive state, so it could be that something you are asking it to do is misbehaving, which is what is causing the Engine to hang. It might be worth investigating and setting some shorter timeouts to macros that might be the culprit.
In the meantime, a method to identify when the Keyboard Maestro Engine is "stuck" could possibly be developed, but the “how” is somewhat dependent on what happens when it is “stuck” and you try to run a macro -- does it just “hang” forever? Or does it timeout? Or something else?
One thought (which is entirely untested and theoretical) is that you could create a very simple macro that does something trivially easy such as output the letter "A", and then run a shell script every minute that runs that macro.
If the macro fails to output the letter “A” then send a:
pkill 'Keyboard Maestro Engine'
which will cause the engine to exit, and then launchd will restart it automatically.¹
The problem, of course, is that if you are in the midst of running a complex macro when the engine quits, the macro will obviously quit too, which could be annoying, at the very least.
But the short version is that there is no built-in / simple way to do this, which surprises me (Activity Monitor clearly knows when there is an “application not responding” as does macOS in general, but there is no way that I know of for a user to get a "yes / no" response to the question “Is this process not responding?” which seems like something that could / should exist, but does not (again, as far as I am aware, and I have looked for it in the past).
¹ a still-better option would be to run sample on the Keyboard Maestro Engine PID when it appears to be hung, and send that to Peter to see if it offers any clues as to what is going wrong with it.
