For Each and Search and Replace - the most confusing combination in KM

[alltiagocom]

UPDATE (Jul 20, 2024)
I finally understand how the combo For Each+Search and Replace works.
Feel free to read what I just posted here

---

I can learn fast and understand complex concepts, but I gotta be honest: I feel pretty stupid when it comes to using these two together, because to me, it just makes no sense most of the time

So I have this list:

Jul 5, 2024, 11:51 PM|17
Jul 3, 2024, 8:47 PM|37
Jul 2, 2024, 10:41 AM|9

And what I want to achieve is a list like this:
"Jul 5, 2024, 11:51 PM|17","Jul 3, 2024, 8:47 PM|37","Jul 2, 2024, 10:41 AM|9"

So I'm surrounding each line with double quotes and adding a comma to separate them. I know I could do this:

but this creates a list that ends with an extra comma, which is not what I need.
But even if there's a different approach, I would like to explore the one below so I can understand what's going on and if there's something I'm missing (I'm definitely missing it, but I can't understand the issue here).

So I decided to use 2 Search and Replace actions and that's where I have no idea what's going on...

If I just activate the first SnR action, I indeed get the list like this:
"Jul 5, 2024, 11:51 PM|17"
"Jul 3, 2024, 8:47 PM|37"
"Jul 2, 2024, 10:41 AM|9"

But when I activate the second SnR, nothing happens.
Isn't the first action updating the source "Local__tasksList" with the new versions of each line, so isn't the list already updated to
"Jul 5, 2024, 11:51 PM|17"
"Jul 3, 2024, 8:47 PM|37"
"Jul 2, 2024, 10:41 AM|9"

when I then run the second SnR to replace each new line with a comma?

---

EDIT: Ok, I think the issue in this particular case is how KM is handling the \n because I moved the second SnR action outside the For Each loop and changed it to this:

And it worked:

But when I change it back to this:

... it doesn't work, even thogh on the RegEx website, it works:

IMPORTANT NOTE: Even if this is indeed an issue with \n I still find the For Each + SnR combination to be confusing most of the time.

[alltiagocom]

Solution seems to be this:
[\r\n]+

Can someone explain why this works in KM and the other option doesn't, while \n in the https://regex101.com website works?
It seems that KM is adding some characters, but for example when I run the macro and I get the original list and paste it to https://regex101.com, the website doesn't show me any extra characters.

[Nige_S]

@alltiagocom

but this creates a list that ends with an extra comma, which is not what I need.

This is a general problem with this sort of conversion, and why things like AS, JavaScript etc have built-in functions to convert lists to delimited text strings. The easiest way round here is to do what you've done, then chop of the extra, unwanted, character(s):

@alltiagocom

Solution seems to be this:
[\r\n]+

Can someone explain why this works and the other option doesn't?

It all depends on the line-endings in your source text. \n is a linefeed character, \r is a return character. By using [\n\r]+ you're saying "find one or more of either linefeeds or returns" (and that will also match mixtures like \n\n\r\n.

You can also use \R to match all line endings, whatever their flavour.

[DanThomas]

To get rid of the last comma, use this on the entire text, not line-by-line:

The regex is:

,\s*$

Without the (?m) option, "$" refers to the end of the entire string.

So this is basically searching for the last comma, optionally followed by whitespace (specifically in this case, a trailing linefeed).

But I think you can do it easier than that. You could probably do this in one regex expression, but I can't quite figure out how to add the first and last quote, so this will have to do:

@alltiagocom

I can learn fast and understand complex concepts, but I gotta be honest: I feel pretty stupid

If you're referring to regular expressions, it takes forever to learn them. But if you use them regularly (pun intended), you can eventually get there. My favorite website for testing them out is regex101.com, and I use it all the time to build regular expressions.

[alltiagocom]

@Nige_S

The easiest way round here is to do what you've done, then chop of the extra, unwanted, character(s):

Thanks for sharing that workaround. I never used the Sub String action before, but I can see how that will help me with some macros. Added to my notes

@Nige_S

It all depends on the line-endings in your source text. \n is a linefeed character, \r is a return character. By using [\n\r]+ you're saying "find one or more of either linefeeds or returns" (and that will also match mixtures like \n\n\r\n.

But is there a reason why KM is adding those and just using \n doesn't work, but when I copy-paste the exact same list to the regex website and test it, \n seems to be enough? That's what's confusing to me...

[alltiagocom]

@DanThomas

Without the (?m) option, "$" refers to the end of the entire string.

Yes, I found that today as well when I was dealing with another action. It wasn't working the way I wanted, because it was processing the whole text, not each line. Thanks for reinforcing that concept.

@DanThomas

trailing linefeed

What is a linefeed and why is it being added?

So when you use the action "Trim Whitespace" you are pretty much removing all of those linefeed things and all that, right?
And once you get rid of those, then the \n will work as expected, correct?

@DanThomas

If you're referring to regular expressions, it takes forever to learn them

I'm actually referring to the combination of the actions "For Each" and "Search and Replace", because it seems to search the collection first in the For Each, but then it targets the collection again in the SnR, when I would expect it to target the line itself.

I think I "kinda" understand it now after looking at it for a few minutes now, but I guess I just need to really get used to how it works.

@DanThomas

My favorite website for testing them out is regex101.com, and I use it all the time to build regular expressions

Yes, that's what I use all the time. It's great!
And I keep taking notes on new things that I'm learning.

And most recently I've been using Claude.ai for some extra knowledge and it was actually where I got the [\r\n]+ from.
It's a bit frustrating sometimes, because it's not very consistent, but I learned that it's all about how to ask things and then just test them. If it works, great, and it always includes some clarification, if it doesn't work, I try to find it using Google and all that, for forums, etc.

Thank you for your reply. Learned a few more things today again.

[DanThomas]

@alltiagocom

What is a linefeed and why is it being added?

On the Mac, when you're typing in plain text (as opposed to a word processor), it's what you get when you press the "return" key.

It's the same as \n, and in this case it could be there for a lot of reasons, but most commonly because you pasted something into a "set variable to text" action and it had a blank line at the end.

And yes, I should probably use [\r\n]+, but everything I deal with on my Mac only has linefeeds.

So when you use the action "Trim Whitespace" you are pretty much removing all of those linefeed things and all that, right?

It only trims the leading and trailing whitespaces ("those linefeed things and all that" ) for the entire string, not line-by line.

so

line 1
line 2

would become

line 1
line 2

And once you get rid of those, then the \n will work as expected, correct?

Correct. But try it out at regex101.com and see what happens if you add a blank line at the end.

I'm actually referring to the combination of the actions "For Each" and "Search and Replace", because it seems to search the collection first in the For Each, but then it targets the collection again in the SnR, when I would expect it to target the line itself.

I almost never use for each when I'm doing a search and replace. But if you do, remember that the variable you use for each line is only a copy of the actual line. So replacing anything in that line won't help you, unless you're combining the results into a new variable. (I hope that makes sense.)

** digression **

The line feed (\n) and carriage return (\r) are holdovers from "back in the day". They actually refer to what a typewriter would do. "Carriage Return" is moving the print head to the beginning of the line, and "Line Feed" is moving the paper up one line so you don't type over the previous text.

Nowadays, at least on the Mac (and Unix/Linux), you generally only get the Line Feed character in text files, but there are exceptions.

In BBEdit, when you save a text file, it gives you these option for line endings:

Line Feed is the default, but if you're sharing files with Windows users, then CRLF is the typical line ending.

[ComplexPoint]

@alltiagocom

what I want to achieve is a list like this:

My personal guess is that regular expressions may just create additional problems here.

A shorter and less confusing route might use:

1. JS .split("\n")
2. the Keyboard Maestro %JSONValue% token to reference the list or any part of it.

---

List from lines.kmmacros (2.2 KB)

[Nige_S]

@alltiagocom

But is there a reason why KM is adding those and just using \n doesn't work

KM isn't doing this on its own -- they're either in your source text or you are adding them in one of your macro's actions. \r is the old-style Mac line ending, \n the new-style (OS X) line ending. While most things are on the "new" (OS X has been around for a while now!) you can still break things.

For example, if you use the Return key in a KM text field the line ending will be \n. But if you have an AppleScript like

set theList to {1, 2, 3}
set AppleScript's text item delimiters to return
return theList as text

...and save that to a KM variable (or copy the result and paste it into a KM action) then line endings are \r. Similarly with KM's tokens -- %LineFeed% is \n, %Return% is \r. Or you might have used \r in a replacement pattern, etc.

@alltiagocom

but when I copy-paste the exact same list to the regex website and test it, \n seems to be enough?

Because when you paste into regex101.com each line becomes an individual string. When it evaluates it assumes line endings are \n. If, instead, your test string was eg 1\r2\r3\r it would fail.

[DanThomas]

@ComplexPoint

A couple of questions:

1. Is it copacetic to code the JXA this way, without enclosing it in an arrow function? I'm guessing since it doesn't define any variables, it's ok to do this?
2. I never knew you could use "kmvar" this way in JXA scripts - I knew you could do it in Custom HTML Prompts, but not in scripts. This is really valuable - thanks!

[ComplexPoint]

@DanThomas

• without enclosing it in an arrow function
• use "kmvar" this way in scripts

The secret lies in the sub-menu behind the small (KM 11+) chevron to the left of the code field (in all execute actions now, I think – not just for JXA)

When Modern Syntax is checked (now the default),
you need to explicitly return a value because the whole thing is already wrapped for you in an IIFE.

What's more, you can control the keys of the kmvar object which is now defined for you in the scope of that IIFE, either:

1. Excluding all Keyboard Maestro variables from it (lightweight, secure) or
2. Including them all (similar to the old arrangement, but with more convenient referencing – no need now for a getvariable method call).
3. Choosing the ones you need – both efficient and secure (particularly for the Browser JS actions)

Kudos to @peternlewis

[DanThomas]

I wish I could give you two thumbs up instead of just one heart. That's awesome to know - thanks!

[Alexander]

Not really contributing to the discussion as a whole here now, but it seems to me like your original task can also be achieved only using a single Search and Replace:

Return linedelimited list- as commaseparated with quotes.kmmacros (17 KB)

---

Edit: Only now saw that a take very close to mine had already been demonstranter by @DanThomas in Post #4, didn’t notice it at my first glance over the thread

[DanThomas]

Based on what @ComplexPoint explained, here's an incredibly simple method to do what the OP was trying to do:

Result:

"Jul 5, 2024, 11:51 PM|17","Jul 3, 2024, 8:47 PM|37","Jul 2, 2024, 10:41 AM|9"

---

The relevant action is the Execute JavaScript for Automation action, with the following script:

return `"${kmvar.Local_Source.trim().split("\n").join(`","`)}"`

NOTES:

1. This requires KM 11.0 or higher.
2. Make sure to click the drop-down arrow (highlighted in green) and select "Modern Syntax".

---

@peternlewis I have no idea how you might implement this, but it would be nice if the JXA action had a visual indicator that the "Modern Syntax" option was selected. Just a thought.

And obviously a million thanks for implementing "Modern Syntax". I keep looking for superlatives to describe your prowess, but let's go with this: You are the GOAT. And I'm not kidding, and that's not hyperbole.

I am regularly blown away at not only the great things you give us, but the inventive way you implement them.

[Nige_S]

@DanThomas

Based on what @ComplexPoint explained, here's an incredibly simple method to do what the OP was trying to do:

AS version is slightly longer, with AS being more verbose, needing more work to get the variable, and lacking a built in trim:

Text to List AS Version.kmmacros (4.1 KB)

Image

[rlivingston]

BBEdit is another place I find very useful to "try out" Regex.

[alltiagocom]

I always use the RegEx website for that.

By the way, this is off topic, but since you mentioned BBEdit, maybe you can help me figure this out?
I have this app called Espanso and it uses YAML files which require indentation of 2 spaces. There are some settings in BBEdit that seem to allow a Tab to add the equivalent of 2 spaces, but when I use it, I always get a block with the width of 2 spaces, but not 2 real spaces, because when I try to use the arrows, it doesn't go 1 space at a time. The issue with that is that when I save the .yml file, I get an error, because it's expecting 2 spaces.

Now when I use VS Code and my tab is also set to 2 spaces, it indeed ads 2 spaces on each tab, but it's 2 real spaces so it never throws me an error.
Already tried the Auto-expand tabs on and off, set the Tab width to 2... no luck.
Any tips?

---

EDIT: Never mind. After messing with the settings I figure it out. It seems that the setting is saved with the file itself, whatever is in Edit>Text Options.
Then the settings in the actual Settings window, will only affect new documents, not old ones. Kinda weird behavior, but... it's all working now

[DanThomas]

@alltiagocom

I always use the RegEx website for that.

That's actually regex101, which is what I use. And I use it for more than just testing. Sometimes when I have things to change in some code in VS Code, I just find it easier to use the website for it.

@alltiagocom

Kinda weird behavior, but... it's all working now

This is a common thought for me, with BBEdit. I think it's a great program, but it certainly is quirky, to say the least. Still, I use it all the time, though usually not for anything fancy.

I know it has features I haven't even come close to finding.