KM8: How To Use Search and Replace RegEx to Remove a LIne

[Joel_Rendall]

Hi Peter and fellow macro ninjas!

I noticed that one of my macros using a regex wasn't working properly, using Search/Replace action. I created a simple macro to demonstrate the behaviour, with the following regex:

^(.+?)\n

I get the same result with: ^([^\n]+?)\n

Unless my regex understanding is faulty (which is possible!), it seems that this regex may be looping through multiple times. As far as I understand, either of these expressions should only remove the first line, but it seems to be removing all lines up until the last line.

Again, I'm not regex-pert but thought I should mention this just in case it's a bug, as this expression is behaving differently than in KM7. Thanks in advance!

Joel

ps. as a successful workaround, I wrote the following expression to split everything into 2 groups using the "search variable" action – 1) the first line and 2) all the following lines

^([^\n]+?)\n((?:.|\s)+)

---

Keyboard Maestro 8.0.2 “Regex Test” Macro

Regex Test.kmmacros (1.9 KB)

[JMichaelTX]

@Joel_Rendall

I noticed that one of my macros using a regex wasn't working properly, using Search/Replace action. I created a simple macro to demonstrate the behaviour, with the following regex:

^(.+?)\n

I get the same result with: ^([^\n]+?)\n

Unless my regex understanding is faulty (which is possible!), it seems that this regex may be looping through multiple times. As far as I understand, either of these expressions should only remove the first line, but it seems to be removing all lines up until the last line.

I'm not sure I'm entirely clear on your objective, but here are some thoughts:

1. Search and Replace is GLOBAL by default.

• So that means ^(.+?)\n will match every line, EXCEPT for the last line, because it does NOT end with a \n

2. When you want to match a line, you need to allow for no EOL characters (\n, \r, \R, etc)

• So you need to use $ as an alternate indicator of EOL.
• This will generally work to use EITHER the EOL characters or $
  (?:\r?\n|\r|$)

So, if you want to do a Search and Replace action (KM Wiki) to remove only the first line, then you can use something like this:

• Search For: (?:.+?(?:\r?\n|\r|$))(?s)(.*)
• Replace With: \1
• For RegEx Details, see: regex101: build, test, and debug regex

@peternlewis and @ccstone are the master RegEx gurus around here, so they may have a better solution.

Here's an example macro to remove the first line.
Please let us know if this answers your question, and feel free to ask any follow-up questions.
RegEx is a complicated topic, and we all (certainly I do) learn from our discussions.

Example Results

##Macro Library   RegEx Search and Replace [Example]

####DOWNLOAD:
<a class="attachment" href="/uploads/default/original/2X/8/8992c63fa33a64768391af93d4d5bdbe7f2bf9f7.kmmacros">RegEx Search and Replace [Example].kmmacros</a> (7.6 KB)
**Note: This Macro was uploaded in a DISABLED state. You must enable before it can be triggered.**

---

#### Purpose
* Demo How to Remove First LIne using RegEx

---

#### REQUIRES:
1. **KM 8.0.2+**
  * But it can be written in KM 7.3.1+
  * It is KM8 specific just because some of the Actions have changed to make things simpler, but equivalent Actions are available in KM 7.3.1.
.
2. **macOS 10.11.6 (El Capitan)**
  * KM 8 Requires Yosemite or later, so this macro will probably run on Yosemite, but I make no guarantees.  :wink: 
---

<img src="/uploads/default/original/2X/c/cc5b118e53e8e292f534ec6163df3bba5f262c6b.png" width="488" height="818">

[JMichaelTX]

@Joel_Rendall, I hope you don’t mind that I changed the Title of your Topic to better reflect the contents, and aid others in searching for it:

Original Title: "Keyboard Maestro 8.0.2 “Regex Test” Macro"
New Title: “KM8: How To Use Search and Replace RegEx to Remove a LIne”

BTW, the behavior you observed did NOT change with KM 8+. Search & Replace has worked that way at least since KM 6, maybe earlier.

Thanks for posting this question. I’m sure it is one that will benefit many users.

[peternlewis]

@Joel_Rendall

Unless my regex understanding is faulty (which is possible!), it seems that this regex may be looping through multiple times. As far as I understand, either of these expressions should only remove the first line, but it seems to be removing all lines up until the last line.

It does loop through multiple times because it is a global search and replace, but you're not wrong, the behaviour is incorrect because the ^ should only be matching at the start of the entire string unless you use the (?m) flag.

I have resolved it for the next version.

[JMichaelTX]

@peternlewis

but you're not wrong, the behaviour is incorrect because the ^ should only be matching at the start of the entire string unless you use the (?m) flag.

I have resolved it for the next version.

Interesting. Has it always been this way, or just introduced with KM8?

However, if you just had one line without an EOL character, then the match would still fail.

[peternlewis]

@JMichaelTX

Interesting. Has it always been this way, or just introduced with KM8?

@Joel_Rendall's search and replace for "^(.+?)\n" with "" worked to remove just the first line in v7, so that at least worked that way in v7 and will again in 8.0.3.

@JMichaelTX

However, if you just had one line without an EOL character, then the match would still fail.

Correct.

Although what it means to remove the first line if there is only one line is not entirely clear.

[Joel_Rendall]

Wow, thanks so much, @JMichaelTX and @peternlewis. Always so impressed by JMichael’s clear explanations and all the time spent with examples, I’ve learned a lot already by reading it a few times through. This kind of support and problem solving would be a great service to offer as a recurring, premium membership, just sayin’ KM is essential to a lot of what I do to make a living, so great to have this support.

No problem about changing the title, the original default one was an oversight on my part. I scrapped my original forum post with a better title after deciding to use KM’s “Share” feature.

Peter mentioned the (?m) flag. Anyone know off hand a good explanation of these and their usage in KM? I tried a forum search but couldn’t find anything relevant, and I’m not sure what the general term for these things are.

Thanks again!!

[peternlewis]

@Joel_Rendall

Peter mentioned the (?m) flag. Anyone know off hand a good explanation of these and their usage in KM? I tried a forum search but couldn't find anything relevant, and I'm not sure what the general term for these things are.

The definitive reference is the one you get from the Help ➤ ICU Regular Expression Reference

There are other sources of information, but this one is definitive since the ICU regex library is what OS X uses and thus what Keyboard Maestro uses.

?m is defined in the Flag Options sextion.

m – Control the behavior of "^" and "$" in a pattern. By default these will only match at the start and end, respectively, of the input text. If this flag is set, "^" and "$" will also match at the start and end of each line within the input text.

[Joel_Rendall]

Great, thanks Peter. This is a powerful tip that will simplify some of my existing regular expressions.

The only missing piece I needed about these flags’ syntax inside KM, I found here (for others who stumble across this post): https://wiki.keyboardmaestro.com/Regular_Expressions

Just need to prepend the flag inside brackets, example: (?m)^\s*\d+[\t]+

[Joel_Rendall]

Peter, I’m sure you hate this question, but with 0% pressure and 100% humility, do you have rough ETA for 8.0.3? Wondering if it’s worth me adjusting all of my frequently-used macros that are affected by this temporary ^ behaviour. Even knowing if it’s a matter of days, weeks or months would be enough for me to make a decision

Thanks!

[peternlewis]

@Joel_Rendall

The only missing piece I needed about these flags' syntax inside KM, I found here (for others who stumble across this post): Regular Expressions [Keyboard Maestro Wiki]

In the Flags Options:

The following flags control various aspects of regular expression matching. The flag values may be specified at the time that an expression is compiled into a RegexPattern object, or they may be specified within the pattern itself using the (?ismx-ismx) pattern options.

You can do things like:

(?m)^\s*$

or

(?m:^)(.*)(?-m:$)

That latter one will capture the last line.

[peternlewis]

@Joel_Rendall

Peter, I'm sure you hate this question, but with 0% pressure and 100% humility, do you have rough ETA for 8.0.3? Wondering if it's worth me adjusting all of my frequently-used macros that are affected by this temporary ^ behaviour. Even knowing if it's a matter of days, weeks or months would be enough for me to make a decision

I'd guess at days. Probably this week.

[Joel_Rendall]

Holy moly, talk about overdelivering! I just saw the update, thanks