Rename Files Mystery

[mrpasini]

I'm baffled by the Rename Files macro in the Macro Library, which I believe is Peter's work.

I've had no trouble using it for some time to do literal search-and-replace operations but today I wanted to use a regex to move some characters around in the root filename and it insisted on replacing the filename with the replacement regex not what the regex should have generated.

For example, the filename is _MRP2345.dng and I'd like to change it to MRP-2345.dng. So I set the search regex to _(MRP)(\d+?)\. (I tried several variants, actually) and the replacement to $1-$2. but the filename is replaced as $1-$2.dng.

Can anyone confirm similar behavior with regexes in Rename Files? TIA

P.S. I did find a very useful batch file renamer called Transnomino that saved the day for me. But I'm still baffled why the macro ignored my regexp.

P.P.S. High Sierra, latest Keyboard Maestro.

[JMichaelTX]

@mrpasini

For example, the filename is _MRP2345.dng and I'd like to change it to MRP-2345.dng. So I set the search regex to _(MRP)(\d+?)\. (I tried several variants, actually) and the replacement to $1-$2. but the filename is replaced as $1-$2.dng.

First of all, @mrpasini, as a long-time member of this forum, and often provider of other user's solutions, you should know what information to give us to help you.

• Either your uploaded Macro (best), or al least a link to the source macro.

So, just guessing, did you change the Replace "for" to Regex?

It works fine for me:

For the benefit of all readers:
Tip: How Do I Get The Best Answer in the Shortest Time?

[mrpasini]

Thanks, but you didn't use Peter's macro from the Macro Library. That's the macro 'Rename Files' that I'm referring it. There's no need to duplicate it because everyone has it. But not everyone would have installed it, of course.

I'm just asking for anyone else's experience using the macro included with the distribution -- and a regex.

The example filename was just to show the results I'm getting to explain why I'm puzzled.

(I'm beginning to think nobody uses 'Rename Files'...)

[JMichaelTX]

@mrpasini

Thanks, but you didn't use Peter's macro from the Macro Library. That's the macro 'Rename Files' that I'm referring it. There's no need to duplicate it because everyone has it. But not everyone would have installed it, of course.

Some issues:

1. We have a "Macro Library" here in the forum, and there is a "Macro Library" that comes with the KM app. You didn't specific which, and It's not our job to go looking.
2. IAC you have made modifications to some macro. The problem you are seeing is most likely due to your changes, but we can't tell because we don't have the macro.
3. Finally, you didn't say whether or not my guess about your issue was correct:

@JMichaelTX

So, just guessing, did you change the Replace "for" to Regex?

Bottom line, to get the best answer in the shortest time, make it easy for us to help you by directly providing all of the necessary information. Surely you know this.

[jonathonl]

Please just use

https://www.publicspace.net/ABetterFinderRename/
this saves so much Regex pain and anguish as it shows you exactly what is going to happen

It also has a lot good stuff if you are into Exif files for photography.

[JMichaelTX]

@jonathonl

this saves so much Regex pain and anguish as it shows you exactly what is going to happen

RegEx need not be pain -- I quite enjoy it.
In this case the RegEx is simple, and not the issue.

RegEx is a great example of the old saying "use it or lose it".
So I like to use every opportunity to use it, which I do in KM, BBEdit, JXA, JavaScript, and AppleScript.

[mrpasini]

@JMichaelTX

1. We have a "Macro Library" here in the forum, and there is a "Macro Library" that comes with the KM app. You didn't specific which, and It's not our job to go looking.
2. IAC you have made modifications to some macro. The problem you are seeing is most likely due to your changes, but we can't tell because we don't have the macro.
3. Finally, you didn't say whether or not my guess about your issue was correct:

1. Sorry that confused you. That's the name Keyboard Maestro uses. If anyone else was confused, I was referring to the Keyboard Maestro distribution not the forum collection.

2. I made no changes whatsoever, so:

3. No, your guess wasn't the problem.

@peterlewis, I wonder why the replace regex is showing up as the new filename in your Batch Rename Files in the Finder macro:

[mrpasini]

Yep, I'm into Exif headers and photography but I prefer to leave camera-generated file names alone (there being some 70,000 of them here).

Interestingly, Transnomino (which is free) can also pop some Exif metadata into a file name, although ABetterFinderRename can pop more of them in.

I have to say regex is not pain and anguish for me. It's just pattern recognition, which I find fun, although I agree it does seem to plague the innocent.

Thanks again for the suggestion.

[JMichaelTX]

@mrpasini

Sorry that confused you. That's the name Keyboard Maestro uses. If anyone else was confused, I was referring to the Keyboard Maestro distribution not the forum collection.

OK, so the Macro that comes with the Keyboard Maestro.app.

@mrpasini

I wonder why the replace regex is showing up as the new filename in your Batch Rename Files in the Finder macro:

So, this Macro, from the Forum Macro Lib, is not the one you're having an issue with?

IAC, I can confirm the behavior the the Search and Replace action used in this macro:

It appears that the Action does NOT support using RegEx in variable.

@peternlewis, is this a bug or by design?

[peternlewis]

@JMichaelTX

It appears that the Action does NOT support using RegEx in variable.

There is no variable in the regex.

The replacement field is not a regex field. It supports tokens (and hence variables), and it supports $1, $2, but it does not support expanding one contained within the other. This is by design.

[JMichaelTX]

@peternlewis

There is no variable in the regex.

That's not what I said. You reversed it.
I said that the KM Variable contains a RegEx pattern, and then that Variable was used in the Replace With field.

IAC, why is the behavior by design? Setting a variable to a Regex pattern string, and then using that in a RegEx function is supported in every other language I have used.
Obviously your Macro is the perfect use case for it in KM.

Having said that, I'll admit I've rarely needed that feature in KM. So maybe it's an edge case.

[peternlewis]

Now I'm just confused.

You can use a token, including variables in the regex search field, it will be expanded and then the resulting regex will be used.

You can use tokens, including variables, and also including things like $1 and \r, in the replacement field. But its not recursive, you can't use $1 inside a token, nor will a token within a $1 result be expanded.

@JMichaelTX

Setting a variable to a Regex pattern string, and then using that in a RegEx function is supported in every other language I have used.

As it is in Keyboard Maestro. I'm confused. The regex search field can contain tokens. As can the replacement field. But once expanded, the expanded contents is not itself expanded.

[mrpasini]

@peternlewis

Now I'm just confused.

Two screenshots should show what I find baffling. Here's the prompt:

Pretty simple, all legal for a file named _MRP3233.dng. But here's what the confirmation dialog shows:

It's using the regex as a literal replacement instead of resolving the regex to MRP-3233.dng.

[peternlewis]

Right, you will not be able to do that.

Since the replace field in this case is a variable, that variable cannot contain any other expansion facilities, such as $1.

I have no solution for this. “Enhancing” the replace field to allow expansion characters within tokens would not be a good idea - tokens could already include any kind of characters and they are not necessarily intended to have meanings. For example, suppose I am trying to use a search & replace to change products to currency amounts, and the amount is in a variable as "$1.00", and that variable is used in the replacement. Suddenly that $1 has meaning where it should not.

So there basically isn't any way to support this.

You could, as an extreme, create the action from XML, after processing the variable to expand it, but there is no solution short of that that can be implemented safely.

[mrpasini]

@peternlewis

Right, you will not be able to do that.

Thanks for confirming the behavior, Peter. That's really all I was asking about.

After exploring the issue a bit, I found a little code that works around the problem nicely. I wrapped that Perl solution (the green macro) into your original macro, replacing the Keyboard Maestro search and replace action:

Rename Files with Regex Support.kmmacros (8.9 KB)

[JMichaelTX]

@peternlewis

I have no solution for this. “Enhancing” the replace field to allow expansion characters within tokens would not be a good idea - tokens could already include any kind of characters and they are not necessarily intended to have meanings.

No one is asking for expansion of ALL Tokens.

@peternlewis

For example, suppose I am trying to use a search & replace to change products to currency amounts, and the amount is in a variable as "$1.00", and that variable is used in the replacement. Suddenly that $1 has meaning where it should not.

You do this the same way you do it in every other language: Escape the "$" if you want to use it as a literal:
\$1.00

Actually, as I'm sure you know Peter, that is standard Regex.

So, I have given you a solution for this. (you really thought there was no solution??)
Other languages I have used allow you to construct a RegEx pattern as a string stored in a variable, and then use that variable in a regex function.

[peternlewis]

@JMichaelTX

You do this the same way you do it in every other language: Escape the "$" if you want to use it as a literal:

You don't escape things within other things that are expanded. Just like a shell variable that contains quotes does not mean that the contents of that shell variable are somehow quoted. You only get that kind of behaviour when there are multiple layers of processing, which is not applicable in this case, unless you want to create the XML, which would be a possibility.

@JMichaelTX

Actually, as I'm sure you know Peter, that is standard Regex.

Standard regex what? Regex says nothing about replacement strings and there is precious little standard about replacement strings. Different apps don't even agree on how bracketed groups are referenced, how escaping (if any) works, what character replacements are supported.

@JMichaelTX

Other languages I have used allow you to construct a RegEx pattern as a string stored in a variable, and then use that variable in a regex function.

This is comparing Apples to Oranges. Keyboard Maestro’s replacement string is a token text string which you can include tokens, backslashed characters and group identifiers in. Replaced tokens that include backslashed characters don't expand them, replaced tokens that include group identifiers don't replace them, replaced groups that contain tokens or other group identifiers don't replace them.

If you use another language, with some other regex, then it will have its own definition of a replacement string, and will vary from language to language. And if you pass it a string, it wont expand things within that string to other things that expand as well either. If the first group matches “$2" and you include $1 in the replace string, the result will be “$2”, not the match of the second group.

In the same way, in Keyboard Maestro, if the first group matches “%LongDate%” you replace with “$1” you’ll get “%LongDate%” not the result of that token. And if your replace string contains “%LongDate%” and the LongDate token happened to include the “$1” would would just get that date from the token, it would not include the contents of the first match.

The replacement string is a token string. It includes tokens, which may include variable - they are expanded and the results used verbatim. It includes $1, $2 etc, and they are expanded to the group they match, and those matches are used verbatim. Neither one is processed further to include other things that might need processing.

[JMichaelTX]

@JMichaelTX

@JMichaelTX

Other languages I have used allow you to construct a RegEx pattern as a string stored in a variable, and then use that variable in a regex function.

This is comparing Apples to Oranges.

OK, I'll not beat a dead horse further other than to offer this JavaScript as an example of other languages:

[mrpasini]

Here's a JavaScript version of the Rename Files macro that supports regular expressions.

Rename Files Macro (v9.0.6d2)

Rename Files.kmmacros (8.5 KB)

NB: The one in the Macro Library that ships with Keyboard Maestro does say it supports regular expressions. But it doesn't, as Peter explains.

[Steve_Solari]

Trying to adapt @mrpasini's methodology. (Thanks for the JSA script!) My case differs because I don't need the option to choose the target folder, so I'm trying to hard-code it. The problem is I keep getting an error:

File action failed because source is not a full path "/Users/<myUserName>/OneDrive - Long Unchangeable Directory Name with Spaces/t_e'%;7km*l?uy5g9r96tp#%*file.txt"

The sample file is intentionally full of garbage characters for testing purposes. The fix pattern works fine at RegEx101. It appears the problem is the spaces in the directory name. Do I need to use double quotes, or a back-tick, or multiple single quotes, or something else in the JSA section? Do I need something along the lines of a POSIX path? Would AppleScript be better suited for this?

---

The quotes around the variables at the last step were just an attempt to fix the issue but they don't help.