Failing Wildcard Expansion in KM Shell Action vs. Terminal

Questions & Suggestions
, ,
1

This action is the first to execute in a KM folder action (macro triggered by a folder trigger):

image
image.png1182Ă—480 36 KB 

read DocumentsFolder f
[[ -z "$f" ]] && f=( $DocumentsFolder/*.txt )
grep -iH 'http' "${f[@]}"

This action is currently failing, and the culprit is this expression: f=( $DocumentsFolder/*.txt ). It's supposed to obtain all the text documents at the specified folder path and store their paths as an array in $f. However, it's storing the literal string value "~/Documents/*.txt".

If I run the action using this code:

read DocumentsFolder f
[[ -z "$f" ]] && f=( ~/Documents/*.txt )
grep -iH 'http' "${f[@]}"

the wildcard expansion occurs as one would expect.

I then ran both versions from inside a terminal, and both versions return the wildcard expansion as I would have hoped.

What might I be overlooking here with regards to the KM shell action implemented using the $DocumentsFolder variable ?

2

My guess is that you are running a different shell (or the shell is acting in a different way) when running in the Terminal from that run without any #! line in Keyboard Maestro.

Keyboard Maestro will be running /bin/sh, whereas you are probably running bash in the Terminal.

You could try just #!/bin/bash at the start of your script, that might be enough.

3

That didn't resolve the problem, although I'm keeping that line in there anyway. You are right that I use a different shell on my system than /bin/sh or /bin/bash—my default shell is FiSH (/usr/local/bin/fish), although the error being returned from the action makes me confident that FiSH is not the shell being used in Keyboard Maestro, as FiSH wouldn't be able to get passed && before throwing an error, let alone understanding bash variable assignments and array references.

When I echo the value of the $SHELL variable from within KM, however, it does come back as /usr/local/bin/fish.

But all the KM shell scripts are sh/bash scripts, and work as I would expect with the exception of this issue.

4

Keyboard Maestro is running sh and passing the script. sh sees the #! and decides what to execute the script with.

So if you start your Keyboard Maestro macro with #!/usr/local/bin/fish then Keyboard Maestro will execute sh which will execute fish which will process the script.

5

Then there must be something else at play here, otherwise, as you said, #!/bin/bash would have resolved the problem.

For now, I’ll just omit the variable and simply insert the path directly to the folder.

6

That would only resolve the problem if the syntax you are using is used by bash.

But you say you are using fish, not bash as your script. So unless I misunderstand, the script you have written is in fish script.

Keep in mind there are hundreds (thousands probably) of different script languages:

  • sh
  • bash
  • csh
  • tcsh
  • fish
  • perl
  • python

etc etc. If you write code in one of them you need to specify to the system what language you are using, and that is what the #! does.

7

I'm familiar with all of that. I think you misunderstood or I didn't explain myself well. All of my shell scripts in KM are always written in bash script. My default shell in macOS is FiSH, but that was only a side point, which wasn't wholly relevant to the issue.

I think my point about FiSH was just that, when retrieving the $SHELL environment variable using a KM shell script action, it does oddly return the path to the FiSH shell; however, the error being returned by the script I'm having the issue with is not a FiSH error, and is certainly a bash one.

Although I've omitted the bash shebang in KM action shell scripts in the past, they've always been executed correctly using bash. Therefore, I wasn't surprised that inserting the bash shebang didn't resolve the issue, even though my shell script is most definitely in bash syntax.

8

How did you run them in the Terminal?

I believe the issue is to do with ~ expansion. This is the thing that I noted:

$ DocumentsFolder=~/Documents
$ echo $DocumentsFolder
/Users/peter/Documents
$ f=( $DocumentsFolder/*.txt ); echo "${f[@]}"
[lists files]
$ DocumentsFolder='~/Documents'
$ echo $DocumentsFolder
~/Documents
$ f=( $DocumentsFolder/*.txt ); echo "${f[@]}"
~/Documents/*.txt

So the issue is that ~ expansion is happening before variable substitution, and that afterwards, the path ~/Documents/*.txt is not actually a valid path/glob, whereas /Users/peter/Documents/*.txt is.

9

This would seem to imply that Keyboard Maestro itself is passing arguments through to stdin and quoting them quotes. If you refer to the very first post, we have a situation where an Execute Shell Script action is taking input from:

%Dictionary[dict, key]%

and the shell script retrieves and utilises this like so:

read path
f=( $path/*.txt )

the outcome of which fits with the diagnosis you described, i.e. expansion before substitution. The dictionary value itself doesn't contain any quotes. Are you able to reproduce this on your system ?

10

No, Keyboard Maestro passes the arguments in through stdin, but there is no quoting involved.

And Keyboard Maestro is token expanding that, and I presume the result is something like ~/Documents.

And then ~/Documents ends up in the $path shell variable after the read command.

At which point the line:

f=( $path/*.txt )

is parsed. First any ~’s are expanded, but there are none. Then the $path is interpolated to ~/Documents, then the ( ~/Documents/*.txt ) is executed, but because ~ is not actually valid as a path at this point (or more accurately, it represents a directory in the current directory with a name “~”, which does not exist), all you get back is “~/Documents/*.txt” which ends up in the $f variable.

You need to expand the “~” yourself somewhere. See:

Something like:

path="${path/#\~/$HOME}"
1 Like
11

Ahh. I got the order of expansions wrong. This explains it perfectly, thank you.

I came up with what I considered as an interim solution a while ago, which is to force the expansion using eval. It felt to me like a bit of a cheap and dirty way to do it, but having skimmed the StackOverflow link above, it appears—if going by popularity and the number of separate answers that employ it—that it is the accepted way to do it.

:+1:t3:

13

The problem with eval is that it will revaluate the whole thing. So if your path was something like ~/Documents/My$path/something.txt, then it will expand the ~, sure, but it will also expand the $path. Now certainly one could argue that having a $ in the folder name is dumb, but that is just one example of what could happen. For example, I don't now how eval will treat spaces, or * or ? or even brackets, etc. Basically it is a big heavy hammer with a lot of moving parts and complexity to involve just to try to expand the ~.

I would try to avoid eval unless I really wanted something evaluated.

1 Like