Macro Limit Reached. Help Please

[Byrein]

So last week I had asked how to track error logs as my macros were randomly stopping and I didn't think my actions triggered them to stop. Peter suggested the Engine.log in another forum (Go to Help, open Logs Folder, Engine.Log) - this was really useful. Kind of like a Debugger but on Steroids.

Anyways I got this message..

"2021-04-15 10:46:44 More than 50 simultaneous macros - aborting everything"

I didnt have 50 macros running at one time.. So I am not sure how it identified that.

Does it measure amount of macros based on CPU consumption?

The Macros basically work like this.

One "Execute Macro" checks criteria for multiple users that use a CRM for Leads. If the CRM needs stuff done to it it will execute the macros needed (6 at most can run per person, I have 4 people now, the most I have ran was like 2 per person at a time, this failed with 6 macros running)

I am definitely using KM more than the average bear, its just so useful and simple and I don't have to learn Javascript or Python to Automate my stuff so any suggestions are appreciated. If I have to get another machine and divide the macros I can do that too.

[tiffle]

The KM engine knows how many instances of macros are running. I presume that if that number is 50 or more then Error!

I noticed you asking about semaphores in another thread. You know that a macro that’s waiting on a semaphore is still running? So it counts towards the 50 limit.

I’m just wondering if you are somehow forgetting to unlock semaphores so there are macros queued up behind the lock waiting to continue - eventually reaching 50? Is that possible?

[Byrein]

The semaphore unlocks when the macro ends, right? And Yes, the Macros run everytime if the criteria meets, The execute macro runs every 10 minutes or so since data is constantly changing.

But I put an action time out on the Semaphore lock for 1 second, so it aborts right away. Maybe I can make it 1 hundreths since KM is so fast it will execute them right away.

I read another forum ("More than 50 simultaneous macros - aborting everything") and it kind of reads like if I have an "execute Macro" action in the macro even if it doesnt execute it, it will count that as a macro? My debugger only showed 6 macros at one time.

[Byrein]

Shorter Action Timeout on Sempahore Lock Did not Work The CANCEL Section in the KM widget (top right of display) only showed 6 macros running when it failed again with the same error.

[tiffle]

@Byrein

reads like if I have an "execute Macro" action in the macro

What - your “Execute Macro” is executing itself?

[Byrein]

No, i use the periodically trigger - but I have a macro that updates and schedules SMS's to leads - it takes about 1 second to send an SMS to a lead - so I have it set up to where if the number of leads is greater than like 3000 it will execute 2 other Macros that accelerate the SMS's.

Basically I was saying if I have an action in a macro that "executes a different macro" within an IF statement, and the macro does not have to execute, does KM count the "execute macro" action as an instance?

If I have a macro that had 10 execute macro actions in it, but only 2 actually executed would it count it as 11 macros or 3 (carrying 1 for the orignal)

[tiffle]

I’m no expert at macro counting, so you need to go back to Peter.

I’m just making guesses here, in the absence of anything to actually examine, hoping maybe something will hit a target.

Really the best way to get to the bottom of this is to let us see your macro system.

I assume you’re familiar with this:

Tip: How Do I Get The Best Answer in the Shortest Time?

Tip: How Do I Get The Best Answer in the Shortest Time? TLDR: The better, the more complete, info you post about your problem, the quicker we can get you a good answer. Show us what you want, in addition to short descriptions. Give a real-world sample of your input data, Show your desired output. Post/upload your macro or script if you are asking questions about it, even if it does not currently work. Don't just tell us "it doesn't work." Tell us specifically what it is doing wrong, how it …

[Byrein]

On it, it uses a lot of API keys etc so Ill just make another macro with a super similar concept.

[Byrein]

00) AX KM Example Macros.kmmacros (2.0 MB)

The comments explain what is happening

[tiffle]

Cheers for that, I’ll have a look tomorrow unless someone else beats me to it!

[Byrein]

You're the man! Its not a day one KM question, and I may be on the right path but its just doing a lot.

[tiffle]

Hi @Byrein - good morning!

I'm not sure I'm the man after looking at your macro system. You're definitely not exaggerating when you say "it's just doing a lot"!

Anyway - I've been looking at it for about 5 minutes and I've started to simplify it in the hope I can force it to error here. While looking at it I noticed this block of actions in the 01) SMS Macro:

This may not have anything to do with your problem, but it is odd to my way of thinking.

Let me explain...

If the action at point (1) gets executed - it may or may not depending on the enclosing For Each and IF actions, it results in the local count variable being incremented.

Then, immediately following in point (2), the same variable local count gets decremented.

So, should the variable be incremented it will always be decremented - i.e. its value won't be changed.

That does not sound right to me but it might actually be absolutely what you intended. I don't know but I thought I'd point it out. (No need to explain the logic to me, it just felt odd.)

Back to the real issue, given your macro system is so complicated it is going to take me some time to (a) simplify it and (b) come back to you with a result. Can't say how long it will take as things are busy and the weekend is looming...

Meanwhile, I'll try to unscramble my brain!

[Byrein]

Intentional - does look odd though.

All that is doing is assigning how many texts need to be scheduled - the headings in my google sheet start with SMS 1,2,3 etc... and an additional heading says SMS seq (isn’t actually a text, so it gets counted everytime) therefore if there’s 3 findings it’s really only 2.

Anyways I’m not sure if I should ask Peter or what not - I don’t get how I’m getting that error if I’m not running 50 macros

[tiffle]

@Byrein

I don’t get how I’m getting that error if I’m not running 50 macros

But you are getting that error therefore you must be running over the limit.

I haven’t finished studying your magnum opus - or should it be macro opus - but I’m getting a feeling that your use of the semaphores is at the root of the problem. Ever heard of a deadly embrace? If not, look it up and you’ll understand why I have this sneaky suspicion about your system.

Meanwhile I need to get onto something else, but it may be worth flagging this up to Peter...

Just had a thought - it would be useful to know how many items are queued up behind a semaphore. It would be worth asking Peter if it’s possible to find this out somehow. I’m going to ask this question in the forum here and direct it at Peter just for my own peace of mind.

[Byrein]

Thank you for your help and I looked it up. I get where you're coming from. I altered the macros to check if the list of all running macro UUIDS contains the UUID of the macro it wants to execute. Basically if the macros running ID was or is "xyz" and the list of all running macros was "xyz,abc,qwe" then that means the macro is currently running so dont execute it. The Semaphore Lock is basically not needed now, but extra security. I am using the %ExecutingInstances% Token By the way.

So Peter, if you would be so kind to look at this macro group it would really help me. I Please let me know if you have any questions, I am so close to obtaining a system that I have been working on for almost 10 months now! Lots of trial and error. Here is the new group for the example. 00) AX KM Example Macros.kmmacros (2.1 MB)

@peternlewis

[peternlewis]

I'm sorry, that set of so many macros and how they operate is beyond what I can reasonably support.

I can only offer specific help on using Keyboard Maestro, not designing or debugging entire macros.

[Byrein]

Okay, thank you I understand. Can you shed some light on why KM says 50 macros running when there isn't close to that many? I just need to know how to identify the limit so I can prevent that from happening.

[peternlewis]

@Byrein

Okay, thank you I understand. Can you shed some light on why KM says 50 macros running when there isn't close to that many?

You are triggering multiple macros, or multiple levels of macros, either through triggers (which would show up in the Engine.log file), through Execute a Macro actions, through heavily nested actions (eg Groups, While, Repeat, etc), or by AppleScript or other means.

[tiffle]

@Byrein

I altered the macros to check if the list of all running macro UUIDS contains the UUID of the macro it wants to execute. Basically if the macros running ID was or is "xyz" and the list of all running macros was "xyz,abc,qwe" then that means the macro is currently running so dont execute it. The Semaphore Lock is basically not needed now, but extra security. I am using the %ExecutingInstances% Token By the way.

Hi - just out of curiosity (because I don't really have the time today to do any serious "digging") I looked at your new macros. I can see a number of places where you set your variables using %Executing Instances% but I haven't been able to find where you make a test and prevent a macro running based on that. Maybe I haven't looked hard enough

In any case I suspect what you've done will make no difference because all this setting of variables occurs after the use of the semaphores. In other words, if there is a problem with the semaphores it will still be preempting whatever other measures you've currently put in place.

And on that note - have your changes made a difference - do you still get the error?

Cheers for now.

[Byrein]

It has made a difference, each SEMAPHORE lock is different even if on the same macro - yes you read that correctly.. So basically the trigger value is set for each user and that is what is in the semaphore lock. So the same user cant have the same macro running for them, but another user can execute if it needs to, however the Semaphore locks are clogging up KM, even with a timeout so I had to adopt the executing instances check.

So to answer your concern about the %Executing Instance% (singular) and %Executing Instances% (plural) - Each time a macro runs, the variable that is set to an %Executing Instance% (singular) token is unique to that user as well. Every user is counted and assigned a number, 1 through how many users there are. So when the macro executes, let's say the SMS macro, the UUID of the macro is set to a dynamic variable for that user (in the SMS macro the dynamic variable is ADMIN__GROUP %Variable%local group% SMS, which is set to the %ExecutingInstance% (singular) token.)

So if the execute macro runs again.. then I take the group number for that user and filter the executing instance variable to get the value of the running ID for it. If the token that has all UUIDS of all RUNNING macros %Executing Instances% (plural) contains that ID I am checking then the macro won't execute.

With that being said, the semaphore lock really isn't even needed at that point and it isn't hurting anything.

You can see in the execute macro the IF to launch macros must meet 3 things to trigger. 1 the switch has to be on (that is set in each users google sheet) 2 the results I am testing contain leads (local initiate is > 0) and 3 the variable local csvids which is set to %ExecutingInstances% does not contain the dynamic variable for that user in the macro it is looking to launch.

That may just be the only way I can do this. I'm using one group that contains macros with all dynamic variables and using one macro in that group to see if that user's macros need to run.

I'm glad an advanced user looked at this with me, just to double check there wasnt a better way. So if you do see something, please let me know. But please don't look into too much, ill figure it out in the next few days im sure.