Bug: Keyboard Maestro does not accept '%252C' escape codes

How is this subverting any standard? It's a well-formed URL that has a defined resolution. You don't like it but I don't understand your aversion to it. Anyhow, thanks again for helping clarify and I continue to wait for the developer of KM to chime in so I can get more information about what is technically happening and we can leave the realm of arguments over standards and their meanings. :slight_smile:

Then colour me very surprised. Since you're in touch with the developer, you might like to remind him to update his own documentation -- the web page I've repeatedly directed you to includes:

Ensure that your values are properly URI encoded. For example, forward slash characters / must be encoded as %2F and space characters must be encoded as %20.

...so even the developer thinks this is the "proper" way to encode such characters. And there's loads of examples, including:

  • obsidian://open?vault=my%20vault
    This opens the vault my vault. If the vault is already open, focus on the window.
  • obsidian://open?vault=my%20vault&file=my%20note
    This opens the note my note in the vault my vault, assuming my noteexists and the file is my note.md.
  • obsidian://open?vault=my%20vault&file=my%20note.md
    This also opens the note my note in the vault my vault.
  • obsidian://open?vault=my%20vault&file=path%2Fto%2Fmy%20note
    This opens the note located at path/to/my note in the vault my vault.
  • obsidian://open?path=%2Fhome%2Fuser%2Fmy%20vault%2Fpath%2Fto%2Fmy%20note
    This will look for any vault that contains the path /home/user/my vault/path/to/my note. Then, the rest of the path is passed to the fileparameter. For example, if a vault exists at /home/user/my vault, then this would be equivalent to file parameter set to path/to/my note.
  • obsidian://open?path=D%3A%5CDocuments%5CMy%20vault%5CMy%20note
    This will look for any vault that contains the path D:\Documents\My vault\My note. Then, the rest of the path is passed to the file parameter. For example, if a vault exists at D:\Documents\My vault, then this would be equivalent to file parameter set to My note.

...and another half-dozen more.

There is nothing in any of what you are quoting that says that the string "%252C" is not valid encoding. It does not violate the RFC nor does it contradict any of those documented examples, because none of them have % signs in them that need escaping so they don't apply. You have made your point, thank you. I disagree and have reported this as a bug to the KM developer team since they don't monitor this forum apparently. I'll let you all know what they say if anything.

Well, I don't want to sound contentious and I realize you can't see my cherubic expression as I write this but I do have to persist that you don't grasp the standard despite my quoting it (and as @Nige_S points out, as Obsidian itself emphasizes).

So I will launch myself at the windmill again.

The standard encoding, as I've referenced above, is a percent sign and two hex digits (alphanumerics 0-9 and A-F). Decoding concerns itself only with that three-character string.

So %20 will decode as a space, for example.

What happens with a URL encoded as %2520? Is the %25 decoded into a percentage sign delimiter that converts the following 20 to a space? No. Should it? No.

Why not?

Because decoders do not regress to look at the literal percent sign just decoded as a liteeral and apply it as a delimiter to the following two characters.

So while you may insist that %2520 and similar constructs are "valid encoding" you are interpreting the coding incorrectly. That %2520 can only be decoded correctly as a percent sign followed by the digit two and the digit zero, not a space. If you want a space, the correct encoding is %20.

I will get back on my mule now and ride off into the sunset.

OK, then I will return your cherubic jousting one more time. I think you are wrong that

you may insist that %2520 and similar constructs are "valid encoding" you are interpreting the coding incorrectly.

No I'm not insisting that and I think that's where you are misunderstanding me. I agree that %2520 evaluates to %20, not a space. I just disagree with you that that means it's somehow "incorrect" just because you'd prefer it to evaluate to a space, which you seem to be fixated on. It doesn't. It evaluates to %20, which if it is parsed again, evaluates to a space character. This kind of double-encoding is not only OK, it's commonly used in things like shell quoting to achieve similar protection.

What happens with a URL encoded as %2520? Is the %25 decoded into a percentage sign delimiter that converts the following 20 to a space? No. Should it? No.

Agree, and I never claimed otherwise. You have just been misunderstanding my point the whole time... :slight_smile:
The important point to realize is that the string is parsed twice. That's why it turns into a space ultimately, but again, KM should not care about the URL, it should trust me and pass it on to the program like I asked it to. :slight_smile:

Jumping in to help clear up one small point, it appears the developer the OP references here:

is not the developer of Obsidian itself, but the developer of an Obsidian plugin, which may explain why the single encoded examples in Obsidian's documentation don't line up with the plugin's use of double encodings. The relevant thread, which OP linked to earlier, is here: "Copy Advanced URI" produces extra %25 characters when copied from Obsidian file menu thingy ยท Issue #62 ยท Vinzent03/obsidian-advanced-uri ยท GitHub

where the plugin developer's response was:

I had some cases (e.g. launching from terminal Encoding | Obsidian Advanced URI), where double encoding solved the issue. I'm wondering that double encoding causes invalid urls for you.

1 Like

Sorry I thought I had made that point clear. Thanks for clarifying it for those that missed it.

the single encoded examples in Obsidian's documentation don't line up with the plugin's use of double encodings

Specifically, can you say in what way don't they line up? Obsidian's documentation does not prohibit this URL scheme either implicitly or explicitly, does it? (Neither does the RFC standard as pointed out earlier). This is perfectly valid encoding, and I still say KM is being at best inconsistent in whatever undocumented standard it is applying here and at worst this is just a little bug in the code. But I'll let y'all know what the KM folks say about it.

Dittoing the above, but I'm afraid I have to disagree. There's nothing wrong with double-, triple, or howevermany-encoding a URL. It is up to the receiving "process" how it deals with this. Mostly it expects single-encoding, but there's nothing to stop someone from writing a processor that requires a double-encoded argument so that eg Apache runs one decode then passes the now single-encoded string to a CGI.

Odd, sure! But not, IMO, contrary to any standard.

Obsidian's documentation is quite clear about what it expects you to send:

Encoding
Important
Ensure that your values are properly URI encoded. For example, forward slash characters / must be encoded as %2F and space characters must be encoded as %20.

In case that's unclear --

"...forward slash characters / must (emphasis mine) be encoded as %2F and space characters must (emphasis mine) be encoded as %20.

So actually yes, Obsidian's own documentation does prohibit double-encoding!

The plug-in developer has used double-encoding as a workaround when using the *nix utility xdc-open (do note that when he says "terminal" he does not mean the macOS Terminal app) which, for whatever reason, doesn't work with single-encoding.

All of the above suggests that single-encoding is all that should be necessary the vast majority of the time, though double-encoding may be an option when single- doesn't work.

And my suggestion would be to make life simple by doing just that...

Exactly my point this whole time. The URL is valid and has valid use cases that you can find similarly used all over the place in the real world with CGI and even bash scripts.

"...forward slash characters / must (emphasis mine) be encoded as %2F and space characters must (emphasis mine) be encoded as %20.
So actually yes, Obsidian's own documentation does prohibit double-encoding!

How do you read this to 'prohibit double encoding"? Double encoding does not violate anything stated in this snippet nor anywhere else that I can find. Can you spell it out, since it's so opaque to me? The doubly encoded URL follows this admonition perfectly. The fact that after decoding once, a URL can be decoded again does not mean it's invalid or breaks any Obsidian documentation for me. That's true even if the URL is meant to be decoded again.
Consider that any URL encoded with %25 symbols is fairly likely to create other decodable symbols by accident actually, if it's in the middle of a word or perhaps an order number or something.... how would you know that's not the case in any given string or in my string in particular, if I hadn't told you otherwise?

By the way, nobody has explained why does the %252C characters violate the rule, but not the other "doubly encoded" values like %252F? This inconsistency screams "bug" at me more than anything else happening here. If your arguments are correct, then KM should reject any four digit character sequence beginning with % that can be interpreted as two hex tuples because they can be double-decoded in theory, because that violates the documentation per your analysis. Am I understanding your ruleset correctly? Whatever rule KM is following, it's not what you seem to be claiming.

I appreciate all the comments. I remain unconvinced but this kind of discussion can really help me learn so thanks. :slight_smile:

Apologies. Yes, I misunderstood you. I was entirely too focused on URL encoding to see you were talking about support for double encoding. And wrong about that, too.

Back on the mule now.

1 Like

OK -- I got nearly to the end of this when I realised we've got two different things going on here -- there's Obsidian's URI schema and the "Advanced URI" plugins schema. My comments about the documentation, here and previously, were referring to the apps schema and not the plugins and I leave them here as evidence of my stupidity :wink: Even so, hopefully the first part, below, is of some use -- working through it certainly helped me get my head around what was going on. If you don't care then scroll down to the bottom for the last two images and conclusion!

I'm not sure if you're being deliberately obtuse or really don't understand, but here goes...

...forward slash characters / must be encoded as %2F...

If you want a forward slash character / you encode it as %2F, not as %252F. Why? Because Obsidian's documentation tells you to... And the same for all the other characters...

But this is all by-the-by. I have seen no evidence that KM rejects any "four digit character sequence beginning with %". It does flag the initial % symbol, because that indicates the start of a token or other "special thing" which isn't yet completed:

Add another character and you still don't have a correctly-formed URL, which KM continues to warn you about:

But continue on with a valid value and everything's good:

And it remains good even with your "double-encoding":

So KM isn't rejecting anything, and I can only surmise that you've some other problem with your URL. Unfortunately you've not posted your macro, so we can do no more than guess at what that might be.

"But wait!", you cry, "Yeah, the warning's gone -- but KM still isn't opening the URL the way I want!".

Well, that's easy enough to check. You seem to know a bit about URLs and you have access to a web server -- knock up a quick page that simply prints the query you've just sent. I had an shtml/Perl combo knocking around from something else, a simple

#!/usr/bin/perl
print "URI was: $ENV{'REQUEST_URI'}\n";

When targeted by a KM action using %2C:


You get:


...as expected.

And when using %2F2C:


...you get:

...and it would appear that the "Open URL" action is perfectly fine with double-encoded characters and is behaving exactly as expected.

Which just leaves Obsidian. Luckily there seems to be a free trial... A quick download, the creation of an "ObsidianTest" vault and a new "My Note" and:


...works fine, opening up the note as expected, while

...gets the totally-to-be-expected

...because Obsidian is receiving the My%2520Note argument from KM's "Open URL" action, decoding that to the literal My%20Note, and is unable to find a note of that title in it's database.

Conclusion: Obsidian does not double-decode URIs.

From what I can make out, your action (and it would really help if you actually posted it rather than posting screenshots) is trying to open a note inside nested folders. So a quick bit of folder-adding and:

By our interpretation of URL encoding, the following action should work (insert brief pause while I realise this is a plugin, go download that, and work out how to activate community plugins!):

(And I've just realised using numbers at the end of folder names makes it a lot more difficult to see what's happening -- apologies for that.)

Indeed, that works fine when run in KM, opening up the "Sub2 Note" in Obsidian.

Which just leaves your "double-encoded" version, as recommended by the plugin's developer for use with apps in which the "single-encoded" URI doesn't work:

And yes! That also works from KM. It also works if you "double-encode" the space character.

Conclusion: The Obsidian "Advanced URI" plugin appears to (at least) double-decode when necessary, but there's no requirement for you to double-encode unless single- is causing you problems. And KM is quite happy to use both "styles" in an "Open URL" action.

So everything's working "as advertised" in general. If you are still having a problem then it is something specific, either to your action or to your structure within Obsidian. So if any of my blathering above still isn't helping you track down the problem, please post your action -- "eyes on" troubleshooting is far easier than guessing at the problem!

2 Likes

One thing I would check is the text processing set under the gear menu of the Open URL action. I suspect you want to process nothing -- and that isn't the default.

1 Like

Thank you for focusing on providing a solution and not arguing with my use case! :slight_smile: That worked perfectly. Much obliged.

thanks for attempting to reproduce the issue. Whatever the problem is, it goes away if I tell KM to not do the thing I don't want it to do, which is to police the URL for me.

And here is what I'm actually doing that instigated the problem:

  1. I have a file in my vault "wealthyvault" named "Knowledge/Art, Spirituality, Psychology/Emotional Journal.md"
  2. I generated the link to it this way in Obsidian, using the Obsidian Advanced URI plugin's command:
  3. Perhaps now this url will make more sense to you:
obsidian://advanced-uri?vault=wealthyvault&filepath=Knowledge%252FArt%252C%2520Spirituality%252C%2520Psychology%252FEmotional%2520Journal.md

It says "wake up obsidian, and tell plugin advanced-uri to do stuff with what's after the question mark". I suspect that before it gets passed to advanced-uri, Obsidian detokenizes it first, then it is handed to advanced-uri, which makes another pass. Thus the double-encoding. But that's just a guess.

So if you wanted to explore further, there is a complete reproducer and background for ya. I will bow out now as @mrpasini has given me the solution in like 30 words. :slight_smile: There may or may not be a bug in KM's lexical parser, but I no longer care. :slight_smile:

I'm not sure if you're being deliberately obtuse or really don't understand, but here goes...

I'm not sure if you're being deliberately rude or just lack social skills, but you can be more helpful if you just assume I mean what I say and skip the personal comments. Maybe ask a question instead?

Thanks for trying to help! I'll follow up here if the developers respond to my inquiry, but I consider this is a closed matter.

Well to be fair we didn't really suss out the double encoding until @Nige_S zoomed in on it as being unusual.

As I said just before that...

My comments about the documentation, here and previously, were referring to the apps schema and not the plugins and I leave them here as evidence of my stupidity :wink:

The Obsidian documentation was explicit and I really couldn't understand why you couldn't understand. But that's because you were talking about the plugin, which I'd completely misunderstood. As I said -- evidence of my own stupidity, for which I apologise.

In attempt to make amends, we can now see what's happening. From the KM manual:

Convert Hex to Unicode Characters
Text token fields also process %NN% or %NNNN% or %NNNNNN% as arbitrary hex unicode characters (eg %41% is an A, %01F300% is :cyclone:).

...and, as luck would have it, you've got pairs of double-encoded characters in your URL, eg %252C%2520. KM is spotting the %252C% and treating it as a token -- the Unicode "Box Drawings Light Down and Horizontal" -- and using that in your URL instead.

@mrpasini was bang on the money -- switching to "Process Nothing" stops that from happening. No bugs anywhere (at least, not here!), everything working "as intended".

Thanks both for the fun trip -- I've got a lot from it, and will remember to watch more closely for "inadvertent" tokens from now on.

2 Likes

OMG! This is the True Solution I think. You have found it, good work ,thanks!

I ain't buying that... you're not stupid, I'm not pretending to be stupid, this was a helpful discussion, thanks. We all make mistakes and I'm glad everyone here is patient and kind about mine.

1 Like

The Final Word from the Devs, confirming what we knew and adding an interesting detail as to why it happens this way at all:

On 19 Jul 2022, at 01:57, Richard Cook <wealthychef@gmail.com> wrote:

I was wondering, if this is not a valid URL, why do Safari and the Terminal have no problem with it?

It is not valid after Keyboard Maestro processes the Text Fields (Text Fields [Keyboard Maestro Wiki]) text token, specifically the %252C%.

The "Process Nothing" is about how Keyboard Maestro processes the text field. For example, you might have a variable Search which contains text to search for, and a URL like:

https://www.google.com/search?q=%Variable%Search%

Keyboard Maestro will process that and replace the Variable token (token:Variable [Keyboard Maestro Wiki]) with the value of the search variable (which must be validly encoded for a URL field, so it cannot contain invalid characters for example) to perform the google search.

After Keyboard Maestro processes the URL and changes the %252C% into unicode character 0x252C which is โ€œโ”ฌโ€, the URL is no longer valid, because URLs cannot contain characters like โ€œโ”ฌโ€.

Side note: Obsidian is "100% free for personal use" (optional "Catalyst" pricing for keen supporters). Recommended. :slight_smile: