Question about variable substitution inside shell script

In a nutshell, I have a variable named kmVar with contents:

Hello $x

and, an Execute a Shell Script action:

x="World"
echo "$KMVAR_kmVar"

Why isn't variable substitution performed there ?

I would expect the substitution:

x="World"
echo "Hello $x"

Output:

Hello World

For my use case, the current behaviour is better, but I am wondering if I am missing something. Does anyone (or @peternlewis) know why is that the case ?

Download Macro(s): Test- Variable substitution inside shell script.kmmacros (1.9 KB)

Macro-Image

Macro-Notes
  • Macros are always disabled when imported into the Keyboard Maestro Editor.
    • The user must ensure the macro is enabled.
    • The user must also ensure the macro's parent macro-group is enabled.
System Information
  • macOS 13.1
  • Keyboard Maestro v10.2

You're expecting shell expansion of the string in the KM variable to occur, but that string never exists in the eyes of the shell.

What you're actually getting is just the KM variable expanding to a pure string.

Expansion in the shell is not a straightforward as many people think.

Here's one way to do what you want:


Download: Test- Variable substitution inside shell script.kmmacros (2.2 KB)

Macro-Image

Keyboard Maestro Export

Macro-Notes
  • Macros are always disabled when imported into the Keyboard Maestro Editor.
    • The user must ensure the macro is enabled.
    • The user must also ensure the macro's parent macro-group is enabled.

System Information
  • macOS 10.14.6
  • Keyboard Maestro v10.2

1 Like

I think it's a "stacked operations" problem, and not particular to KM variables -- the shell resolves variables once, and doesn't attempt to then resolve the product of that resolution. But you can force it with an eval command:

2 Likes

Thank you both @ccstone and @Nige_S.

Mmm...but, if I execute the following:

I get:

Hello World

But, that pure string contains an unescaped dollar sign. I think that would be interpolated by the shell.

Is it the case that KM variables are passed quoted with singles quotes ?

Yes, because it is evaluating the variables in sequence, one per command:

  1. x is set to "Hello"
  2. y is set to the evaluation of "$x ", ie "Hello "
  3. z is set to the evaluation of "$y World", ie "Hello World"
  4. Then you echo the evaluation of z

That's different to a single statement that recursively evaluates a variable until it can go no further, which is what we'd both hope would happen but obviously doesn't.

It's the same in eg Terminal (here escaping the $ so that $y is the text "$x")

% x="Hello";y=\$x;echo $y
$x

...and you can see that $y is evaluated but the shell doesn't then say "Hang on, we've now got $x, let's evaluate that too before we echo".

Compare that to

% x="Hello";y=\$x;eval echo $y 
Hello

...where we force a second evaluation.

1 Like

No, although you can think of it that way to predict what will happen.

Interpolation occurs at the point when the variable is created. Once you already have a string no interpolation can possibly take place.

The KM variable is a preexisting string, and is unavailable to interpolation – just as if it was a single-quoted string – even though it's not.

You'd think this would interpolate, but the same rules apply:

image

Change that last line to:

eval echo $bpDir

And it will work.

Like I said:

1 Like

Keyboard Maestro variables are not passed quoted by any kind of quotes.

The environment variables are set explicitly to the variables contents without any substitution.

So the KMVAR_kmVar environment variable contains Hello $x.

And when the shell interprets "$KMVAR_kmVar" it gives the value of that environment variable without any further interpretation.

You would get the same result with this shell code:

$ myvar="Hello \$x"
$ echo $myvar
Hello $x
$ x="World"
$ echo $myvar
Hello $x

Expansion happens once, it does not expand the variable, and then expand the contents of that variable. Imagine if you did:

$ myvar="Hello \$myvar"
$ echo  $myvar

All that happens is the single expansion and Hello $myvar being printed.

3 Likes

Worth a detour to fully internalise the core evaluation processes, I think.

See, for example – very inexpensive on Kindle: How to Automate Command Line Tasks Using Bash Scripting and Shell Programming: Cannon, Jason

1 Like

Thank you guys !

Thank you Peter.

I see, in your example, that the dollar sign is quoted.

So, the contents are set explicitly but the dollar signs are escaped to prevent interpolation ?

So, in a nutshell, I learned that this has to do with the order of the operations:

KMVAR_kmVar="Hello \$x"
x="World"
echo "$KMVAR_kmVar"

but I don't understand why I have to escape the dollar sign inside the script, but I do not have to when I declare a kmVar variable in Keyboard Maestro.

Why would you? To KM "$x" is just text, not a variable to be evaluated. In the shell, in the variable KMVAR_kmVar, it is still the string "$x". And when you reference $KMVAR_kmVar inside double-quotes there's only one "round" of evaluation, giving you "$x".

The analogous situation in KM would be trying to set a variable to "%Variable%Local_x%" -- if you wanted that literal text to be stored you would have to turn it into "something not to be evaluated".

Here's the KM version of the problem. Try it as-is, then enable the "Filter" action -- that's like adding the eval command in the shell script.

Variable in Variable.kmmacros (2.9 KB)

Image

Shell variable behaviour can be non-obvious, especially to a programmer who is used to "proper" variables :wink: I'll echo @ComplexPoint and suggest a bit of background reading -- if you don't fancy a book then a quick web search will find you plenty of tutorials, and shell scripting is well covered by LinkedIn Learning/Udemy/Code Academy/etc if you're subscribed to any of them.

Thank you, @Nige_S.

I'll definitely check the resources both you and @ComplexPoint shared.

I did a different test to understand a bit better:

I am setting test_kmVar to the string "$x World". So, I think the result of both calls to echo should be the same; however, the output is:

$x World
 World

In both cases, string interpolation occurs once. And both variables contain the string $x World. What am I missing ?

Download Macro(s): Test.kmmacros (2.1 KB)

Macro-Image

Macro-Notes
  • Macros are always disabled when imported into the Keyboard Maestro Editor.
    • The user must ensure the macro is enabled.
    • The user must also ensure the macro's parent macro-group is enabled.
System Information
  • macOS 13.1
  • Keyboard Maestro v10.2

The best way to develop intuition for a command line parser is to try writing a simple one.

A Parsec project, perhaps ?
The first step, is, of course, to reflect on the output of the parser.


See, for example, the Parsec source code of:
ShellCheck – shell script analysis tool

Haskell Hackage: ShellCheck

1 Like

No, you aren't. The string is double-quoted, so $x is replaced with the contents of variable x before the string is assigned to test_kmVar.

But x is undefined, so the replacement is an empty string. In pseudo-programmy language you are doing

test_kmVar = "" + " World"

...which is exactly the output you are getting.

You'd get what you expect if you single-quoted the string in the shell because then string is literal text and the variable isn't expanded:

% test_kmVar='$x World';echo "$test_kmVar"
$x World

I get the feeling you're still thinking like a "real" programmer, where you define a variable then do string interpolation by preceding the variable name with a $ sign. That's not how the shell (generally -- there's lots of shells and lots of variation) behaves. test_kmVar is just a string except in certain situations -- like when it is separated from an "operation" by a single = and nothing else, or when it is preceded by a $:

% test_kmVar="Hello"
% echo test_kmVar  
test_kmVar
% echo $test_kmVar ## expansion because of the $, even without quotes
Hello
% echo '$test_kmVar' ## note how single-quotes "turn off" expansion
$test_kmVar
% echo $test_kmVar" World" ## concatenation with a string
Hello World
% echo "$test_kmVar World" ## expansion within a string
Hello World
% echo '$test_kmVar World' ## turning off expansion, but still concatenating
$test_kmVar World
1 Like

This looks like an attempt at a systematic account of string-expansions in shell command line parsing:

Command-Line Processing - Learning the bash Shell, Second Edition – O'Reilly

1 Like

Thank you everyone.

I did some experiments, and I can't replicate it. If I add at the bottom of my ~/.zshrc file the following two lines:

x="Hello"
ENV_TEST=$x World

and then source ~/.zshrc in a Terminal session, I get an error:

command not found: World

Now, adding

x="Hello"
ENV_TEST="$x World"

and executing echo $ENV_TEST:

Hello World

The only way I could replicate that using single quoting, as in:

x="Hello"
ENV_TEST='$x World'

and now, executing echo $ENV_TEST :

$x World

So, apparently, KM environment variables are set with single quotes (or another method) which prevents string interpolation. Does that make sense ?

Otherwise, it seems impossible to get the literal contents of the environment variable.

Think about what is happening when you source these lines...

Hint: Try adding the following instead

x="Hello"
ENV_TEST="$x World"
x="Goodbye Cruel"

...sourceing, then doing echo $ENV_TEST.

Clearly, ENV_TEST is going to be set to the string "Hello World".

My point is: imagine x wouldn't have been defined there, then I should get a space instead of $x. There has to be single quotes around to prevent string interpolation, I think.

I get the same output.

Exactly. ENV_TEST is set to "Hello World" so echo $ENV_TEST returns "Hello World". You are not setting ENV_TEST to "$x World" and then resolving two expansions in a single echo. That's further proven by the second snippet.

But why would any of that apply to KMVAR_kmVar? Or are you assuming that KM sets environment variables via source?

Honestly, I'm getting a bit confused as to why this even matters. But I am enjoying the conversation!