In the attached macro, I have two versions of a shell script.
One has the path explicitly specified in the shell script, one has the path indirectly specified via a KM variable.
The first one works, but the second one gives me a curl error that ~ resides on a read-only file system.
Shouldn't the script work either way?
I spent 10 minutes looking at it. I didn't spot a problem. But here are three things I would do to troubleshoot.
That's the best I can do at the moment. By the way, do you have any read only file systems?
I believe the difference is that the ~ is not being expanded in the latter case.
Use the full path in the first action.
~ is a shell construct, not part of the file system, and so the ~ expansion happens in the shell in the first case, but does not happen in the second case. So the second case, the ~ is passed to curl verbatim, and that is not a valid path.
If all else fails, a brute force way to make sure all expansions are happening would be to eval the entire string.
Thanks.
@Airy I do not have any read-only file systems.
@peternlewis Interesting. Before posting, I did try changing the ~ to the full path, and wasn't able to get it to work.
But now I tried a few more variations and was able to get it to work IF there are no spaces in the path. It seems that even escaping spaces with a backslash is problematic and gives a "failed to extract a sensible file name from the URL to use for storage" error.
And also when I use a path with an escaped space and the "Include Errors" flag on, it looks like it returns the "raw data" of the image it's trying to download.
e.g. if I try the macro on this page, here's part of the results...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (6) Could not resolve host: Downloads
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 7134 100 7134 0 0 171k 0 --:--:-- --:--:-- --:--:-- 174k
PNG
IHDR@@ªiqÞe¥IDATxÚÍ{ çuÞûû{fwv±÷û^Jâ7h"HB$HÚ² ãË*ªDª¤¬,0Êv%1§G¦bÚ%J@%´$@H n,ÅâØû¾ç¾ûÏû{föÂb± ltOOwOß{ï{ßÿ÷,¡Âx|·wíÚE:÷Óâçßÿþ÷» ø _ä³"f@Ö´!ñH#§¯ q)øÃDII$îIyáX#hR5[î~qu5@ýÆO.É°µ m"ÝÐÐ
My path was /Users/myusername/desktop\ Downloads
And from "Could not resolve host: Downloads", it looks like the shell interpreted the "Downloads" part of the path as part of the url. I guess the shell is stripping or ignoring the backslash.
Also, I ended up tweaking the macro so the variable only contains the path and url to download, and changed the shell script to: curl --create-dirs -O -J --output-dir $KMVAR_local_curlString
@johns I'm not familiar with eval, but after Googling, this is the proper usage, right?
variable="$KMVAR_local_curlString"
eval "$variable"
I wasn't able to get that to work.
Just checking, but you did use two backslashes in your KM text field, didn't you? Eg /Users/Me/desktop\\ Downloads
Another way to deal with spaces in paths is to double-quote -- note that you either have to use the full path or put the ~ before your KM variable:
There's an example of using eval with a KM variable here. Note that you don't wrap the variable in quotes but you do (in your case) include the curl part -- you're telling the shell to "evaluate the string curl --create-dirs....
Out of interest, why curl? What's that getting you that KM's "Get a URL" action doesn't?
Example for a single image file:
...and easy to change to a "For each" loop based on your JavaScript results.
Yes, that is going to fail for the same reasons.
Unix commands when dealing with arguments on the command line do not themselves deal with any of this:
All of that is handled by the shell. The command is simply given an array of arguments. You can see exactly what the arguments would be by using this perl shell command:
#!/usr/bin/perl -w
foreach my $a ( @ARGV ) {
printf '"'.$a.'"'."\n";
}
(assuming you have perl).
And exactly how the shell handles command line expansions is extremely complicated.
A lot of this is covered in the Quoting Strings section of the Execute a Shell Script documentation.
If you think about it as being passed an array of parameters, it's pretty obvious that you cannot give the shell an environment variable with space separated fields and a path with spaces in it and expect it to work. And you cannot include quoting or backslashing to solve the spaces issue because none of that is processed after the environment variable is expanded.
Read the documentation on quoting, and then solve this by not doing this. Output to /tmp/YourDestination and then move it afterwards.
You are going to go down an endless rabbit hole if you try to figure out levels of quoting to make this work.
After spending some time going down the rabbit hole...
I ended up creating a soft link (alias didn't seem to work) to a path with no spaces so now I can have the actual destination folder anywhere.
i.e. ln -s /path/to/actual/folder /Users/me/Downloads/placeholder
Then use /Users/me/Downloads/placeholder as the destination in the macro. No spaces. Problem solved.
Thanks everyone!
I used curl just out of habit.
But also because I've never used the Get Url action.
And the [-1] in the variable looks neat. I'll have to learn how to use that. I would've tried to use a regex pattern.
Me too! Also when I need curls options.
Here's a demo that shows how to get your ~/, space-containing, destination folder path into the "Execute a Shell Script" action and an easy way to work through an URL list if you present it to the shell as a linefeed-delimited list. Because it only instantiates the shell once it's more efficient than looping through the KM variable and executing the action for every line.
I've used a list of Forum images -- just replace that action with your "get images with JavaScript, save to a KM variable as a linefeed-delimited list" routine.
curl a List.kmmacros (3.2 KB)
It's just the standard "treat text as an array, split on the string after the ]" method, except we're using a negative number to count backwards through the array -- -1 is the last item. So it's "the text after the last / in the path".
It's very clever is what it is!
You don't need to use regex for splitting paths, there is an Split Path action specifically for this purpose.
It seems like there's always more than one way to do things in KM!
@Nige_S thanks for the updated macro. I'll check it out.
@peternlewis @Nige_S I did know KM can handle "arrays", but I wasn't aware you could specify the delimiter like that and I didn't know you could use -1 as the last item.
I guess I need to spend a lot more time in the wiki.
Neat stuff!
