Checking the Engine.log file I see that I have this macro that always has one of these messages after the macro starts running:
2024-06-30 23:52:51 Variable “KMVAR_Local__files” excluded from environment to ensure the environment is not too large
2024-06-30 23:58:46 Variable “KMVAR_ObsidianVaultFolders” excluded from environment to ensure the environment is not too large
This is a macro that checks a few folders and makes a list of all files that were modified between 2 dates: the current date and the date of the last time I ran that macro (this is stored in a global variable).
What does it mean? Does it mean that any file using these variables aren't copied to the location? How do I work around this issue?
I saw this suggestion, but that doesn't seem to be the issue, because one of the variables is Local and I get that error message anyway:
Then there's this:
But this starts getting too complex for me. If anyone can help me understand this, that would be great. I don't understand the whole limit thing, the 100k... I'm pretty sure that's not related to the file size, otherwise we wouldn't be able to do almost anything that involved files that are bigger than 100k (assuming this is related to 100kb?).
Any info and tips are welcome. I've been dealing with this for so long and I would like to finally fix this macro and trust that the files are being copied.
I was checking the message:
2024-06-30 23:58:46 Variable “KMVAR_ObsidianVaultFolders” excluded from environment to ensure the environment is not too large
Looking at the date it was ran yesterday. Thing is, when I try to find a macro that has that variable, it shows me no results:
I can see that the variable exists in the variables pane though, so I wonder how is the Engine getting this message, if the variable is not being used by any macro...?
I'm guessing that macro has a "Execute a Shell Script" action in it?
If you aren't referencing those macros in the shell script, KM not passing them into the shell environment won't matter. If you are referencing them I think you'd have seen bigger error messages by now!
If you do need to use them in your shell script, try writing them out to a temp file then reading that into the shell script action, or setting the System Clipboard then using pbpaste.
(The automatic inclusion of everything, including the kitchen sink, is a legacy of an earlier – pre Keyboard Maestro 11 – approach to making KM variables available in other scripting environments. The new approach is both more efficient and more secure, and worth taking advantage of as fully, soon, and thoroughly as possible)
TLDR
See the menu behind the drop-down chevron to the left of the script text fields, and always prefer Include No Variables as the default, only selecting any that the script really needs.
I don't really understand the technical aspect of environments and all that, unfortunately.
A while ago I was having an issue with a macro and Peter himself suggested that that option to include all variables was the best one to use as a default. I don't know if that was just him trying to help me without being too technical, based on my inexperience with that? Maybe it was.
Thank you for offering this solution. I will update some of my favorite actions to have that disabled and I even added a comment to the script itself to remind me to just include the ones I'm using.
So maybe the fact that the "include all variables" is enabled is what makes it show that warning, even though I'm not even using those variables in any macro (on my second post)?
Maybe this is too technical for you to explain, but if I'm using variable X and variable Y on that script, why would I want to include all variables anyway? That's what seems even more confusing to me...
I will make that change to that macro and see how it goes.
So do you think that including all variables could also be the reason for me to experience this "2024-06-30 23:56:12 Task failed with status -1" almost 50% of the times I run that macro? I don't think I've ever experienced this with any other macro, to be honest.
So basically, it seems to make no sense to ever select "Include All Variables", right?
If I have this script: chmod 755 /tmp/find_them
I select "Include No Variables"
If I have this: blah blah blah "$KMVAR_Local__var1" blah blah blah "$KMVAR_Local__var2"
Then I just go and pick these 2 variables from the list, right?
If that's the case, what if I have variables with the same name, which is quite frequent with Local variables? For example a "Local__itemToSelect". I was just scrolling through the list of variables on that dropdown menu and it seems to me that KM doesn't show duplicates, right? At least I didn't see any, but I'm 100% sure I have the same local variable in different macros.
The list is simply of all variable names used in any macros.
Duplication doesn't really have a meaning in that sense – it doesn't matter if the same name is used in different places. (Unless, of course, two simultaneously running macros happen to confuse each other by binding competing values to the same global macro name).
So you mentioned the issue with global variables which I can totally understand. I have very few global macros and the ones I have are all named uniquely and I rarely use global macros anyway, so having the same global macro being use and altered in 2 macros running at the same is almost impossible (if not completely impossible, actually).
What about local variables? If I have Local__var1 being used by 2 different macros running at the same time, does KM interpret each variable as unique because they are local, even though they have the exact same name?
By the way I just ran that other macro and it seems that those messages about those two variables are no longer being shown in the log. I will keep checking the log just in case, but it seems that this solved the issue. Thanks.
My understanding is that values are bound to local_ variable names only within name-spaces that are private to a particular macros. There should be no risk of clashes.
KM doesn't know what variables you will want to access in the shell script, so it makes as many available as it can/should. These will include all your global variables and any local variables used in that macro before the "Execute a shell script" action.
But, as Peter explained, the amount of data you can pass to the instantiating shell in this way is limited -- if it will be over 100k (I assume kB) then KM will not pass some and will add log entries to let you know.
The option @ComplexPoint... erm... pointed out was introduced in KM v11 and now allows you to control which variables are made available to the shell, helping ensure that what you want included is by excluding the ones you don't.
Local variables are only available within the currently executing instance of a macro. Even if you have "Macro A" with variable local__a and have it running multiple times concurrently, each macro's local__a is separate from the others, and a shell script action within "Macro A" will only see the local__a belonging to its executing instance.
The list in the action's options isn't so much a list of actual variables, it's a list of variable names in use across the Editor. For globals that amounts to the same thing but for locals anything you pick will be the instance for that macro, because of the above scoping. If the local variable doesn't exist in your macro but you tick it in the options and then try to use it in the shell script it will evaluate as an empty string, ''.
Which just leaves:
If you've got round this by eliminating all your unneeded variables, or you aren't actually using Local__files in your shell script -- no worries. But if you do need to use it and it's a mahoosive list of file paths, you'll need to present it to the shell in a different way or run the risk of errors whenever the list breaks the 100k limit.
So I guess one of the approaches would be to create the macro with "Include All Variables" to make sure that we can make changes without having to pick the variables and removing them as we change the script, but then once the macro is fully done and working, go and just check the variables we are using?
So I guess that the 100k is related to the variable's content, not what the variable represents, the file itself. For example a variable to a path such as ~/path/to/audio.mp3, KM is evaluating the variable as text, and that will be Xk, not the size of the file that the path is targeting, which would obviously be way more than 100k.
That's why you mentioned below "mahoosive list of file paths".
If that's the case, then I now understand the whole thing when it comes to size, which was making it more confusing to me.
And that's why ticking "Include All Variables" will not do anything to the script, if no variables are being used in that script, but it's then adding dozens/hundreds of empty strings, one per variable in that list, right?
Ok, so if a macro is showing me that error and I do need a certain variable, and it's the only variable selected on that list, that means that the variable alone is above the 100k. If that's the case, then using the method to read from a script file will fix this?
Let's say I have indeed the "mahoosive list of file paths" and that exceeds the 100k limit. If I set that list as a variable and then use it in the script, I get that error. But it I add the list directly to the script, without the variable, then I won't see that error, right? So the issue is the variable, not the script itself, and that's why using the script from a file won't be a problem. Correct?
No. The action will only pass on the variables that are "in scope" at that moment. That will include:
All global variables
Any instance variables used in the macro or its calling macro(s) that have been instantiated before the action executes
Any local variables used in the macro that have been instantiated before the action executes
It won't, for example, include password variables (I think! I only did a quick test), or local variables defined in other macros.
You can see the environment variables for an executing shell script action with a simple
You only need to get the variable's contents in using some other method, you don't need to do the whole script. You could set the System Clipboard in KM and use pbpaste in your script:
You could write the data to a temporary file and read it in as part of your script, and I'm sure there are loads more options. There's no one right answer, it'll depend on the data, how you want to process it in the script, etc.
It doesn't actually have to be that big a list. Say our tame user John Jones-Smith wants to process some files with a KM script and the paths are:
I don't use instance variables, so that's not the case.
The issue with your statement "Any local variables used in the macro that have been instantiated before the action executes" is that one of the variables that was initially showing that message (KMVAR_Local__files) is not being used by that macro, or not even by any of the other macros I have on any other group (but according to your statement, is seems that it would only affect local variables in the current macro, right?)
I tried this and I noticed that local variables weren't shown on that list, but neither did the global variables. Is that normal, because you said that global variables were included?
But that would only work if I was only using a single variable, right?
In case I want to use pbpaste in a longer script, assuming that the content is a finder path with spaces, with variables I would add "KMVAR_Local__myVar", using the double quotes because of the spaces in the path.
When using pbspace should I add it exactly how you used it $(pbspace) or should I use the double quotes as well like this "$(pbspace)"?
Unfortunately this is already too complex for me. I will stick to the other options for now, but I will save this in my Notes for future reference. Too much information for now haha
So assuming that the list is indeed that long, and let's not think of the pbpaste for now, just so I can understand what the best approach is:
Would it make sense to save each line as a temporary file, read from it, delete it (so one temporary file for each line at a time), or create unique files for each line, maybe inside a folder, and at the end of the script just delete the whole folder?
It's fewer steps (deleting the temporary file vs just deleting a whole folder with all the temporary files)
You'll know your own macro best -- but don't forget that the variable will be called Local__files in KM, the KMVAR_ prefix is for the shell environment variable.
They will be -- unless you're using the options @ComplexPoint pointed out and deliberately excluding them.
Yes -- but you could combine multiple variables into one clipboard entry and then separate them again in the shell script.
It depends -- different shell utilities have different ways of handling arguments, so there's no one correct answer.
No. It would make sense to save the whole list as a temporary file, then work through that list item by item in the shell script.
If you are watching TV and want some crisps/chips, would you keep the packet in the kitchen and then go back and forth between sofa and kitchen one crisp/chip at a time? Or bring the packet to the sofa so it's in arm's reach?
A silly analogy, but the first is like looping through a variable calling a shell script each time and the second is like passing the list then looping within the shell script. And either approach can be the right one, depending on circumstance -- to torture the analogy still further, you might go to the kitchen for one crisp at a time so you don't gets crumbs on the sofa...
