For Each and Search and Replace - the most confusing combination in KM

UPDATE (Jul 20, 2024)
I finally understand how the combo For Each+Search and Replace works.
Feel free to read what I just posted here


I can learn fast and understand complex concepts, but I gotta be honest: I feel pretty stupid when it comes to using these two together, because to me, it just makes no sense most of the time

So I have this list:

Jul 5, 2024, 11:51 PM|17
Jul 3, 2024, 8:47 PM|37
Jul 2, 2024, 10:41 AM|9

And what I want to achieve is a list like this:
"Jul 5, 2024, 11:51 PM|17","Jul 3, 2024, 8:47 PM|37","Jul 2, 2024, 10:41 AM|9"

So I'm surrounding each line with double quotes and adding a comma to separate them. I know I could do this:
image

but this creates a list that ends with an extra comma, which is not what I need.
But even if there's a different approach, I would like to explore the one below so I can understand what's going on and if there's something I'm missing (I'm definitely missing it, but I can't understand the issue here).

So I decided to use 2 Search and Replace actions and that's where I have no idea what's going on...
image

If I just activate the first SnR action, I indeed get the list like this:
"Jul 5, 2024, 11:51 PM|17"
"Jul 3, 2024, 8:47 PM|37"
"Jul 2, 2024, 10:41 AM|9"

But when I activate the second SnR, nothing happens.
Isn't the first action updating the source "Local__tasksList" with the new versions of each line, so isn't the list already updated to
"Jul 5, 2024, 11:51 PM|17"
"Jul 3, 2024, 8:47 PM|37"
"Jul 2, 2024, 10:41 AM|9"

when I then run the second SnR to replace each new line with a comma? :thinking:


EDIT: Ok, I think the issue in this particular case is how KM is handling the \n because I moved the second SnR action outside the For Each loop and changed it to this:
image

And it worked:
SCR-20240706-kybq

But when I change it back to this:
image
... it doesn't work, even thogh on the RegEx website, it works:

IMPORTANT NOTE: Even if this is indeed an issue with \n I still find the For Each + SnR combination to be confusing most of the time. :wink:

Solution seems to be this:
[\r\n]+

Can someone explain why this works in KM and the other option doesn't, while \n in the https://regex101.com website works?
It seems that KM is adding some characters, but for example when I run the macro and I get the original list and paste it to https://regex101.com, the website doesn't show me any extra characters.

This is a general problem with this sort of conversion, and why things like AS, JavaScript etc have built-in functions to convert lists to delimited text strings. The easiest way round here is to do what you've done, then chop of the extra, unwanted, character(s):

It all depends on the line-endings in your source text. \n is a linefeed character, \r is a return character. By using [\n\r]+ you're saying "find one or more of either linefeeds or returns" (and that will also match mixtures like \n\n\r\n.

You can also use \R to match all line endings, whatever their flavour.

1 Like

To get rid of the last comma, use this on the entire text, not line-by-line:

image

The regex is:

,\s*$

Without the (?m) option, "$" refers to the end of the entire string.

So this is basically searching for the last comma, optionally followed by whitespace (specifically in this case, a trailing linefeed).

But I think you can do it easier than that. You could probably do this in one regex expression, but I can't quite figure out how to add the first and last quote, so this will have to do:

image

If you're referring to regular expressions, it takes forever to learn them. But if you use them regularly (pun intended), you can eventually get there. My favorite website for testing them out is regex101.com, and I use it all the time to build regular expressions.

1 Like

Thanks for sharing that workaround. I never used the Sub String action before, but I can see how that will help me with some macros. Added to my notes :wink:

But is there a reason why KM is adding those and just using \n doesn't work, but when I copy-paste the exact same list to the regex website and test it, \n seems to be enough? That's what's confusing to me...

Yes, I found that today as well when I was dealing with another action. It wasn't working the way I wanted, because it was processing the whole text, not each line. Thanks for reinforcing that concept.

What is a linefeed and why is it being added?

So when you use the action "Trim Whitespace" you are pretty much removing all of those linefeed things and all that, right?
And once you get rid of those, then the \n will work as expected, correct?

I'm actually referring to the combination of the actions "For Each" and "Search and Replace", because it seems to search the collection first in the For Each, but then it targets the collection again in the SnR, when I would expect it to target the line itself.

I think I "kinda" understand it now after looking at it for a few minutes now, but I guess I just need to really get used to how it works.

Yes, that's what I use all the time. It's great!
And I keep taking notes on new things that I'm learning.

And most recently I've been using Claude.ai for some extra knowledge and it was actually where I got the [\r\n]+ from.
It's a bit frustrating sometimes, because it's not very consistent, but I learned that it's all about how to ask things and then just test them. If it works, great, and it always includes some clarification, if it doesn't work, I try to find it using Google and all that, for forums, etc.

Thank you for your reply. Learned a few more things today again.

On the Mac, when you're typing in plain text (as opposed to a word processor), it's what you get when you press the "return" key.

It's the same as \n, and in this case it could be there for a lot of reasons, but most commonly because you pasted something into a "set variable to text" action and it had a blank line at the end.

And yes, I should probably use [\r\n]+, but everything I deal with on my Mac only has linefeeds.

It only trims the leading and trailing whitespaces ("those linefeed things and all that" :stuck_out_tongue:) for the entire string, not line-by line.

so


line 1
line 2

would become

line 1
line 2

Correct. But try it out at regex101.com and see what happens if you add a blank line at the end.

I almost never use for each when I'm doing a search and replace. But if you do, remember that the variable you use for each line is only a copy of the actual line. So replacing anything in that line won't help you, unless you're combining the results into a new variable. (I hope that makes sense.)

** digression **

The line feed (\n) and carriage return (\r) are holdovers from "back in the day". They actually refer to what a typewriter would do. "Carriage Return" is moving the print head to the beginning of the line, and "Line Feed" is moving the paper up one line so you don't type over the previous text.

Nowadays, at least on the Mac (and Unix/Linux), you generally only get the Line Feed character in text files, but there are exceptions.

In BBEdit, when you save a text file, it gives you these option for line endings:

image

Line Feed is the default, but if you're sharing files with Windows users, then CRLF is the typical line ending.

1 Like

My personal guess is that regular expressions may just create additional problems here.

A shorter and less confusing route might use:

  1. JS .split("\n")
  2. the Keyboard Maestro %JSONValue% token to reference the list or any part of it.

List from lines.kmmacros (2.2 KB)

2 Likes

KM isn't doing this on its own -- they're either in your source text or you are adding them in one of your macro's actions. \r is the old-style Mac line ending, \n the new-style (OS X) line ending. While most things are on the "new" (OS X has been around for a while now!) you can still break things.

For example, if you use the Return key in a KM text field the line ending will be \n. But if you have an AppleScript like

set theList to {1, 2, 3}
set AppleScript's text item delimiters to return
return theList as text

...and save that to a KM variable (or copy the result and paste it into a KM action) then line endings are \r. Similarly with KM's tokens -- %LineFeed% is \n, %Return% is \r. Or you might have used \r in a replacement pattern, etc.

Because when you paste into regex101.com each line becomes an individual string. When it evaluates it assumes line endings are \n. If, instead, your test string was eg 1\r2\r3\r it would fail.

1 Like

A couple of questions:

  1. Is it copacetic to code the JXA this way, without enclosing it in an arrow function? I'm guessing since it doesn't define any variables, it's ok to do this?
  2. I never knew you could use "kmvar" this way in JXA scripts - I knew you could do it in Custom HTML Prompts, but not in scripts. This is really valuable - thanks!
1 Like

The secret lies in the sub-menu behind the small (KM 11+) chevron to the left of the code field (in all execute actions now, I think – not just for JXA)

When Modern Syntax is checked (now the default),
you need to explicitly return a value because the whole thing is already wrapped for you in an IIFE.

What's more, you can control the keys of the kmvar object which is now defined for you in the scope of that IIFE, either:

  1. Excluding all Keyboard Maestro variables from it (lightweight, secure) or
  2. Including them all (similar to the old arrangement, but with more convenient referencing – no need now for a getvariable method call).
  3. Choosing the ones you need – both efficient and secure (particularly for the Browser JS actions)

Kudos to @peternlewis

6 Likes

I wish I could give you two thumbs up instead of just one heart. That's awesome to know - thanks!

2 Likes

Not really contributing to the discussion as a whole here now, but it seems to me like your original task can also be achieved only using a single Search and Replace:

Return linedelimited list- as commaseparated with quotes.kmmacros (17 KB)


Edit: Only now saw that a take very close to mine had already been demonstranter by @DanThomas in Post #4, didn’t notice it at my first glance over the thread

Based on what @ComplexPoint explained, here's an incredibly simple method to do what the OP was trying to do:

image

Result:

"Jul 5, 2024, 11:51 PM|17","Jul 3, 2024, 8:47 PM|37","Jul 2, 2024, 10:41 AM|9"

The relevant action is the Execute JavaScript for Automation action, with the following script:

return `"${kmvar.Local_Source.trim().split("\n").join(`","`)}"`

NOTES:

  1. This requires KM 11.0 or higher.
  2. Make sure to click the drop-down arrow (highlighted in green) and select "Modern Syntax".

@peternlewis I have no idea how you might implement this, but it would be nice if the JXA action had a visual indicator that the "Modern Syntax" option was selected. Just a thought.

And obviously a million thanks for implementing "Modern Syntax". I keep looking for superlatives to describe your prowess, but let's go with this: You are the GOAT. And I'm not kidding, and that's not hyperbole.

I am regularly blown away at not only the great things you give us, but the inventive way you implement them.

image

8 Likes

AS version is slightly longer, with AS being more verbose, needing more work to get the variable, and lacking a built in trim:

Text to List AS Version.kmmacros (4.1 KB)

Image

4 Likes

BBEdit is another place I find very useful to "try out" Regex.

2 Likes