Variable use in Shell script issue

Hi! I have a large macro that sorts and processes files in a selected folder based on their file extensions, mostly using 'find' and 'rsync'. It's working great, but it's tedious to modify the shell script commands individually to include or exclude additional file extensions. I had to idea to instead use variables with a simple list to "simplify" (haha) editing the macro later for myself and other less experienced users.

Rsync works fine, as i'm able to use txt files and the --include-from and --exclude-from options.

I'm having trouble with the find command.

My working command is as follows

find "$KMVAR_debug_Package" -type f \( -name "*.mp4" -o -name "*.mov" -o -name "*.mxf" -o -name "*.m4v" \)

And here's the macro:

When I view the "local_Video_Find" variable in Keyboard maestro, or echo it from a shell script, the string is identical to the contents of the original find command. I have a feeling it has to do with improperly interpereting the quotes, but I can't for the life of me figure it out.

Quotes in a shell command are special. Quotes in a variable in a shell command are just quotes. You don't need the quotes here, so let's ignore them :wink:

Some searching took me to this example of processing the terms as an array. And, after a bit of fiddling:

Find by Extension.kmmacros (3.1 KB)

Image

This'll work because your search base path has no spaces in it -- with spaces we're back to the same quoting problem as before.

Usual caveat about programmatically creating a command that you then run unseen applies -- be very sure of your inputs!

1 Like

Interesting, as they say, there's always another way to skin the cat.

Based on what you said...when I tried just removing the quotes from the append variable actions in my macro, it works, even when my path has spaces in it. I didn't realize the syntax didn't require the quotes in the find command.

I still don't really understand why quotes are "different", but I recognize that the method you posted trades some simplicity in the keyboard maestro domain for for complexity in a bash script. I feel better about having some additional complexity (the for loops) in my macro since I'm much more comfortable troubleshooting the keyboard maestro actions.

Thanks for your help!

Which path? Your "search base" will be OK because it has no spaces but will (I think) break if it does. Your "names ending with" are OK as is but, again, will break when looking for names with spaces. So it'll work for what you're doing now, but isn't very general.

It's the order of evaluation (I think!). When the shell sees

find "/Users/username/Desktop"...

...it gets to the first quote and knows that, from there to the next quote, any space should be treated as a literal space and not an argument separator.

The same if you have the KM variable Local_path containing /Users/username/Desktop and the shell script

find "${KMVAR_Local_path}"...

But if Local_path is "/Users/username/Desktop" then

find ${KMVAR_Local_path}...

...the variable is expanded and the quotes in that are treated as literals -- it's "too late" to treat the as the special "these spaces are actual spaces" indicator. So the find command tries to base from "/Users/username/Desktop" but there's no top-level directory " and it fails.

Similarly with

find "${KMVAR_Local_path}"...

The shell now knows to treat the coming spaces as spaces, it then expands the variable, but the variable starts with " -- again, no such top-level directory.

Quoting gets messy, especially when quotes in variables are involved, and I don't really understand it -- I just keep bullheadedly trying different combinations until something works! Maybe someone will step in with a better explanation...

1 Like

See: Execute a Shell Script, Quoting Strings.

It describes how the arguments to a command (find) in this case is not a big long string, but an array of strings, with each one being an individual argument.

If you use "$KMVAR_Var" then that single argument will be the contents of the Var variable. If the Var variable contains quotes or *s or spaces, those are just characters in the argument.

If you use $KMVAR_Var, then the contents of the Var variable will be split by spaces, and each part will be an argument. Any quotes or *s it may contains are still just characters in the arguments. So if Var contained β€œ"abc def ghi"” that would but three arguments, "abc, def, and ghi".

It is extremely difficult to build up a legitimate full command in a variable and expect it to work, so almost always what you want to do is have the full command in the Execute Shell Script action.

When writing the command on the command line, if you say ls *.txt, it is actually the shell that processes that, the *.txt is not passed to the ls command, instead the ls command receives an array of arguments, each one being a file name that matches the *.txt pattern.

This is why when you write it in the find command, you need the "*.mp4", because you don't want the shell to expand that to a list of all the matching files in the directory. So you have to quote it to stop that happening.

Now, lucky for you your extnesions will never have spaces, and you don’t want the * expanded, so if you have a variable that contains -name *.mp4 -o -name *.mov -o -name *.mxf, and you pass it as just $KMVAR_Var (without any quotes) you get exactly what you want - the components separated by spaces all sent as arguments with no further processing.

However your path definitely might have spaces in it, so it does not have that luxury. But now you can build up the -name string, and then your command is:

find "$KMVAR_debug_Package" -type f \( $KMVAR_local_Video_Find \)

and that should work fine.

Note that if you include the ( ) in the KMVAR_local_Video_Find variable, then they would not have the \ for the same reason - it is there to stop it from meaning something to the shell, which would not be required if it was within the variable.

2 Likes

And a thought came to me last night -- if we use AppleScript to build the command string and run it with the do script verb we can avoid a lot of the quotes and evaluation order issues. That'll let us generalise both the search base and the match terms so we can include spaces -- we can do stuff like "find everything on the Desktop whose name starts with Screen Recording or end with *.mov...".

Demo macro:

Find by Extension (AS).kmmacros (3.2 KB)

I'd normally avoid the "using KM to run AS to run a shell" nesting but, given that the extra AS instantiation time is nothing compared to how long the find will take, how much easier this is makes it worthwhile.

2 Likes

Thank you! Between re-reading that section of the wiki, and your very generous and helpful explanation here, I have a much better understanding of how to approach these types of situations going forward.

And @Nige_S thank you for the alternate approaches! It's always so helpful to have this knowledge for future macros and for others.

2 Likes