CURL KM Var with Percent Sign

Questions & Suggestions
1

I am making a request to get a page, then in that response, I grab the needed data and then grab the token that is also returned in the response and use it in the API to get the next page. The token to get the next page is a parameter in the data body for CURL, the parameter (token) is also URL encoded and the token value is set to a variable named "local body"

Please note when I take the token needed, filter it URL encoded, and then manually paste the encoded token into the shell script where it is needed, it works perfectly.

With that being said, this is 100% an issue with my lack of understanding for Quotes and shell script along with token processing for % signs with Keyboard Maestro.

I tried to replace the % signs with %% and that didn't work either

Here are my requirements

  1. the token must remain encoded here is an example of a token

&__OSVSTATE=XGAIfDCWDWQTahZXUKIimkQxsqd7Y4Wj%2B9VtGuFOcr0K6HfLEbro2c%2BNGL6EYx%2Ben4P0P9gDNcLUg2Qcmmc7Q3JJ2aaf7lJ9uRyzJ%2FSikaR24%2F%2BC5m%2FOpm5wsIXRjP2ut%2FZGVlEsh29xOR3FRKrqlZEQomO1DnPw4mYCFwWjNr8MAtphqhFiyCWhU2GxeYS4XE6aFaHSBfmIzdmphB%2FEEAdntzboTE9bbXKlftUYZTdUZO6WISO2%2FHv3VDExOu7frN7X%2Bb7D2aBWMGvYnVLXbVdUa%2Bf%2Fd9HTf4mdy6SRsBti4v9Fs5Bn2cXxR%2FRsXuIWgvJLknj3dpRnqNvkB5Pn17uQMMTllfyYrFm2gvoVZU4Cq%2FWR4O4XZt0iaupZ2kzpTyn3JbZQzGabBVLCnKzgd2bjhkHg60YS3VY5kAb47ZrJu8YxFDh%2FNyXoeUk6tAwKDd0lwNg6rke5j3LWrjWHLOfQpw%2BC6QJF7esaI8kWa1CKuC9aKxDLYsFgEEAhDcpRaiJ%2BBrVRVOc4r1vNgR0Hxux3x8xnrYknA3maHfBJMe6VCgKfjegywtYv6vCXRLK9Q4p4nWlmkCe7gmV%2Fbr%2BT9xW%2B9jN9hOkpzWCOQi --- TOKEN continues this is only so format is understood

  1. The Data field is as follows

--data '__EVENTTARGET=RichWidgets_wt58%24block%24wt28&__EVENTARGUMENT=&__OSVSTATE=XGAIfDCWDWQTahZXUKIimkQxsqd7Y4Wj%2B9VtGuFOcr0K6HfLEbro2c%2BNGL6EYx%2Ben4P0P9gDNcLUg2Qcmmc7Q3JJ2aaf7lJ9uRyzJ%2FSikaR24%2F%2BC5m%2FOpm5wsIXRjP2ut%2FZGVlEsh29xOR3FRKrqlZEQomO1DnPw4mYCFwWjNr8MAtphqhFiyCWhU2GxeYS4XE6aFaHSBfmIzdmphB%2FEEAdntzboTE9bbXKlftUYZTdUZO6WISO2%2FHv3VDExOu7frN7X%2Bb7D2aBWMGvYnVLXbVdUa%2Bf%2Fd9HTf4mdy6SRsBti4v9Fs5Bn2cXxR%2FRsXuIWgvJLknj3dpRnqNvkB5Pn17uQMMTllfyYrFm2gvoVZU4Cq%2FWR4O4XZt0iaupZ2kzpTyn3JbZQzGabBVLCnKzgd2bjhkHg60YS3VY5kAb47ZrJu8YxFDh%2FNyXoeUk6tAwKDd0lwNg6rke5j3LWrjWHLOfQpw%2BC6QJF7esaI8kWa1CKuC9aKxDLYsFgEEAhDcpRaiJ%2BBrVRVOc4r1vNgR0Hxux3x8xnrYknA3maHfBJMe6VCgKfjegywtYv6vCXRLK9Q4p4nWlmkCe7gmV%2Fbr%2BT9xW%2B9jN9hOkpzWCOQi&__VIEWSTATE=&__VIEWSTATEGENERATOR=8EA4AF55&wt85=&wt79=&wtStatusComboBox=999&wtStatusComboBox2=__ossli_0&wt69=__ossli_1&wtDateFrom=01-01-2022&wtDateTo=02-23-2022&wtListAppointmentTable%24ctl02%24RichWidgets_wt30%24block%24wttxtTableRecordsId=wtListAppointmentTable&wtListAppointmentTable%24ctl02%24RichWidgets_wt83%24block%24wttxtTableRecordsId=wtListAppointmentTable&wtListAppointmentTable%24ctl02%24RichWidgets_wt21%24block%24wttxtTableRecordsId=wtListAppointmentTable&wtListAppointmentTable%24ctl02%24RichWidgets_wt72%24block%24wttxtTableRecordsId=wtListAppointmentTable&wtListAppointmentTable%24ctl02%24RichWidgets_wt24%24block%24wttxtTableRecordsId=wtListAppointmentTable&wtListAppointmentTable%24ctl02%24RichWidgets_wt3%24block%24wttxtTableRecordsId=wtListAppointmentTable&__AJAX=2002%2C1498%2CRichWidgets_wt58_block_wt28%2C1463%2C1914%2C349%2C0%2C1947%2C1482%2C'

  1. I have tried to replace the __OSVSTATE value with the following $KMVAR quote combinations

'info&__OSVSTATE="$KMVAR_local_body"moreinfo'
syntax: open with single quote - databefore param - double quote VAR double quote - more data after
VAR - single quote to end

'info&__OSVSTATE=''"$KMVAR_local_body"''moreinfo'
syntax: open with single quote - databefore param - single quote (to end previous literal) single quote (open new literal) double quote VAR double quote single quote single quote - more data after VAR - single quote to end

'info&__OSVSTATE='"$KMVAR_local_body"'moreinfo'
syntax: single quote open - info - single, double - VAR - double, single - more info - single quote to end

I have also tried to reverse the quotes by opening and ending the data body with double quotes.. and I have tried no quotes along with setting my value in a text variable and passing it in as the entire body itself.

I hope this was not confusing, I would love to understand this quote madness. Thank you

Screen Shot 2022-02-24 at 9.23.57 AM
Screen Shot 2022-02-24 at 9.23.57 AM1728×2496 371 KB

2

I am no master of quotes either. I simply experiment until it works.

But I have a simple solution to all my quote problems. And that is, instead of using any quotes at all, simply put the text that I want to execute into a KM variables, write the variable to a file, and execute the file instead of executing a string.

It's still a bit early in the morning for me to be explaining myself. (I just woke up.) So if this isn't clear, ask again and I'll try again.

3

I thought about doing that too but didn't really understand the script file with shell script

So If I have a CURL Post script

Save the entire script to a file with the new values

and then use shell script, script file, execute?

ill try it now

4

Oh i remember, I did try. it wouldn't let me upload the file, I assume the extension was wrong or something

5

It appears that you understand my suggestion. Let me know what happens.

6

There's NO uploading of any file! You save the file containing the CURL statement in a folder of your own.

I can probably post an example in a few minutes when my eyes are more widely open.

7

Screen Shot 2022-02-24 at 10.22.21 AM
Screen Shot 2022-02-24 at 10.22.21 AM2104×318 55.1 KB

the directory of a file that contains the POST info would go here?

8

No, you have to save the whole curl statement. And execute the curl statement with an Execute Shell Script action. Maybe that's what you meant, I'm not sure.

9

One moment please Sleepy, I really need this to work let me map this out, thank you sir

10

I'm trying to help. (With my eyes still half open, so it's hard.) I can create or upload an example in a few minutes.

11

Screen Shot 2022-02-24 at 10.29.11 AM
Screen Shot 2022-02-24 at 10.29.11 AM2110×1532 191 KB

I have to block the text out because its work stuff but this is what I understood and it did not work

12

Well, here's what you need to do now. Copy the command from your text window (everything inside the window that contains the red block) and paste it into a terminal window. If it fails, then you did something wrong.

Maybe you are missing the germ of my idea, My idea is to execute a string that actually works in the Terminal window, it does not solve any problems with quotes or percentages. If you don't have a command that works on the command line, my idea doesn't solve anything.

Also, you may have to change a permission on the file you created. So that it can be executed. I forget if I do that or not.

13

Okay, It does work when I manually paste it, but I want to automate it and it grab page after page, so I need the token to be dynamic, let me try the permissions

do I have to change the extension?

also I see in the screen shot I have

input from file, but I turned that off and put to nothing and it did not work either way

on it now

14

I guess I can just have KM open terminal window, paste new code, save result etc lol

15

I'm glad you are making progress. As for the dynamic part, you have two solutions.

  1. Use a KM Variable, by passing the variable to the executing shell script that you created.
  2. Recreate the file with the value of the new variable.

I forget whether I use a .sh extension or not. It may be safer to do that.

16

okay thank you i forgot about .sh

17

In the past I sometimes had to change the permission on the file to "allow execution". If you need that, we'll change the permission.

18

it says PERMISSION DENIED, changing the settings now..

how do I change the settings when I save it with KM?

19

I was warning you about that permission issue. Let me look up the command to fix that. Give me a second. There's probably more than one way to do that and I'm trying to decide which way to use here.

20

I believe the command in your original post didn't work because the you wrapped the argument to --data in single quotes, which prevented the interpolation of $KMVAR_local_body. A better solution would be to build the entire argument string in a Keyboard Maestro variable and pass that to --data as a single chunk.

curl <other options> --data "$KMVAR_data_string"

That said, @Sleepy's suggestion to put the entire curl command into a file and execute it is even better. Run the command without the execution step first, to make sure the command is being built correctly.