The %Return% token

[Tom]

###In BBEdit…

…interestingly, with the inversed order (CR first, then LF) I do not get any incompatible chars:

set sourceStr to "CR" & return & "LF" & linefeed
set the clipboard to sourceStr

[JMichaelTX]

@Tom

…interestingly, with the inversed order (CR first, then LF) I do not get any incompatible chars:

Confirmed -- same results here. Very strange.

EDIT:

And now, BBEdit \r finds the match:

[Tom]

In BBEdit \r always finds both (LFs and CRs).

In the first example (with the unrecognized char) the CR does not work as line break, since the editor can’t handle two different kind of line breaks in one document.

The same in TextMate makes it a bit clearer:

The CR ist recognized as CR but does not “function” as a CR. The same in BBEdit: when you open the Character palette it is clearly identified as CR (Ascii 13), but BBEdit simply does not have a way to display it like TextMate.

But still interesting that the order decides if the CR gets converted into a LF or not. As you see in the screenshot, TextMate does not care about the order, it always preserves it as (non working) CR.

[JMichaelTX]

@Tom

In BBEdit \r always finds both (LFs and CRs).

Is it that, or has BBEdit converted all of the CRs to LFs?

Unexpected, and unwanted, unless I set an option somewhere in BBEdit to do so.

[Tom]

@JMichaelTX

has BBEdit converted all of the CRs to LFs?

Yes, that’s what I tried to say in the last paragraph. (In contrary to TextMate, which preserves them.) Unless the CR comes first, then it does not convert it, for unknown reasons.

I think \n and \r are (almost) synonymous in BBEdit. Try to replace all \n by \r. You will still have linefeeds. It’s the document setting (at the bottom of the window) that decides over the type of linebreaks.

See also page 148 and 173 of the BBEdit manual.

[Tom]

Quoting my self…

@Tom

It’s the document setting (at the bottom of the window) that decides over the type of linebreaks.

More precisely:

The line break setting (at the bottom of the window) decides which kind of line break is written to file.

In the editor (the open document) there are always LFs, even if the line breaks are set to “Legacy Mac (CR)”. However in the document on disk you’ll find CRs then.

To see what I mean:

1. Create a document consisting only of a line break.
2. Set the document to “Legacy Mac (CR)”.
3. Save it to disk.
4. Make a Hex dump of the document (File menu> Hex Dump Front Document)

• You’ll get 0A (=LF)

5. Make a Hex dump of the file (File menu > Hex Dump File)

• You’ll get 0D now (=CR)

[JMichaelTX]

@Tom

More precisely:

The line break setting (at the bottom of the window) decides which kind of line break is written to file.

In the editor (the open document) there are always LFs, even if the line breaks are set to “Legacy Mac (CR)”.

Yep, I was just about to post this, when I saw your last post:

So, what that means, is that there is no apparent way, using BBEdit, to find/replace all \r with whatever, correct?

Confirmed. I just tried this, and it replaced BOTH \r and '\n`:

Bummer. I'm really surprised that BBEdit would behave this way. I'm hoping that Chris (@ccstone) will jump in here soon to shed some light on this.

[Tom]

@JMichaelTX

that there is no apparent way, using BBEdit, to find/replace all \r with whatever, correct?

Just change the setting at the bottom of the window to the type of line break you need. This will change all line breaks. (In the file on disk.)

[JMichaelTX]

@Tom

@JMichaelTX

that there is no apparent way, using BBEdit, to find/replace all \r with whatever, correct?

Just change the setting at the bottom of the window to the type of line break you need. This will change all line breaks. (In the file on disk.)

I think you're missing my point. I don't want to change all of the line breaks.
I may just want to change wherever I have a string like "some text\r" to something else.

KM can do this, so I guess I'll just have to do it there.

[Tom]

I think a text document with mixed line breaks is not a very sane document. So, honestly I’m very happy with the way BBEdit works in that respect.

Some years ago I had to work with Windows people and those produced very weird stuff sometimes. I highly appreciated that BBEdit made it so easy to “normalize” that.

But I understand that there might be needs to selectively treat CRs. Though I cannot think of a concrete example at the moment.

[peternlewis]

@JMichaelTX

Peter, I'm not sure what you mean by this. Are you saying that, somehow, the System Clipboard is changing LF characters into CR characters when they are copied to, or pasted from, the Clipboard?

No, that is not what I said.

I said that Mac applications use Return (CR 0x0D) characters in clipboards.

If you copy text in Mail or Pages, the text has Carriage Returns in them (except in the UTF8 flavor apparently).

But yes, the system does mess around with them when it generates different flavors.

Do not use BBEdit as an example, it does many different things and has deep handling for line endings far exceeding (and different) to most applications.

Basically, you cannot rely on the line endings in the clipboard being anything in particular, and for most text flavors it is more likely to be CR than LF, even today.

[JMichaelTX]

@peternlewis

I said that Mac applications use Return (CR 0x0D) characters in clipboards.

Actually, you didn't say anything about applications. You said:

@peternlewis

However, traditionally on the Mac, the clipboard uses Return (CR) characters even in OS X.

But what you said in correction makes more sense, and is what I was expecting.

IAC, I take your main point: Any searches need to check for either CR or LF (or even both). That's where the RegEx \R is so powerful. Unfortunately it isn't supported on all RegEx engines and/or all macOS.

So, to be safe, we could use something like
(?:\r?\n|\r)

That should match any of these:
CRLF
LF
CR

OR, if you're looking for EOL, then maybe:
(?:\r?\n|\r|$)

I'm sure you can improve on that.

But I still have one question: While using KM, what would be the advantage of using %Return%? Seems like LF (%LineFeed%) (shouldn't that be either "Linefeed" or "linefeed"?) is overall a better choice.

OK, let me answer my own question: Here's one good(?) reason to use %Return%: Microsoft Word 2011.

• Word does NOT generate any kind of line break when you paste a LF. <shrug>
• And it uses CR for line breaks when you copy text from Word.
  I guess the unexpected should be expected with Word WRT the Clipboard.

[peternlewis]

@JMichaelTX

Actually, you didn't say anything about applications. You said:

Quote the whole paragraph:

@peternlewis

However, traditionally on the Mac, the clipboard uses Return (CR) characters even in OS X. That remained for many years after the transition to OS X (when native Classic Mac OS applications ran along side native Mac OS X applications). In fact there were many years when a LF on the clipboard was highly likely to cause problems.

Also, in Classic Mac OS there were nothing except applications. Besides, 99% of all clipboards copied on the Mac are copied from Applications.

@JMichaelTX

I guess the unexpected should be expected with Word WRT the Clipboard.

No, CR on the clipboard is to be expected. That is perfectly normal for Mac. It seems the UTF8 flavor typically has line feeds, but other text flavors typically have CR, so it will depend which one is used in any given situation.

@JMichaelTX

Seems like LF (%LineFeed%) (shouldn't that be either "Linefeed" or "linefeed"?) is overall a better choice.

"line feed" is two words, tokens are Camel Case, and case sensitive, so it is %LineFeed%.

[JMichaelTX]

@peternlewis, OK, thanks for the info and the lively discussion.

Just so I'm clear, and maybe it will help others, here's a summary of my understanding.

2017-10-27 14:28 GMT-0500
• Correct use of \R per Peter's post.

Summary of Use and Handling of CR and LF

Submitted for your review, comment, and/or revision.

1. CR -- Carriage Return

• ASCII Decimal: 13
• Hex: 0D
• RegEx: \r
• KM Token: %Return%
  .

2. LF -- Line Feed

• ASCII Decimal: 10
• Hex: 0A
• RegEx: \n
• KM Token: %LineFeed%
  .

1. When using KM Actions with RegEx, it is best to allow for both CR and LF as newline characters. So a RegEx of something like this would be useful:

• (?:\r?\n|\r|$)
• \R will work macOS 10.11+

2. What is put on the Clipboard is dependent on the Application where the copy command is done. Newline character could be either CR or LF, but most likely CR. Again, we need to allow for either (see #1).
3. KM \n will match only LF in a Search and Search/Replace.

• I thought I had seen cases where it matched CR, but I cannot reproduce that with Keyboard Maestro 8.0.3 (8.0.3) on macOS 10.11.6.
• Peter, please confirm/correct.

4. BBEdit 12+ will match either CR or LF with a \n
5. When you paste source text to RegEx101.com, it converts all CRs to LFs.

• I have reported this as a bug to the developer.

My Test Results and Macro

###MACRO:   Return (CR) and Linefeed (LF) KM Test

~~~ VER: 2.0    2017-10-26 ~~~

####DOWNLOAD:
Return (CR) and Linefeed (LF) KM Test.kmmacros (9.0 KB)
Note: This Macro was uploaded in a DISABLED state. You must enable before it can be triggered.

---

[peternlewis]

@JMichaelTX

\R will work for KM 8+ and macOS 10.10+

\R will work in macOS 10.11+, in (I believe) any version of Keyboard Maestro.

@JMichaelTX

KM \n will match only LF in a Search and Search/Replace.

Correct.

@JMichaelTX

BBEdit 12+ will match either CR or LF with a \n

BBEdit uses \n LF line endings internally, despite whatever the line endings of the text file are. It converts on load/save to the line endings of the file on disk. You can see this by changing the line ending popup, saving to disk, and using both File ➤ Hex Dump Front Window and File ➤ Hex Dump File. The former always shows \n (0x0A), while the latter shows whatever is configure. And it seems that its regex has somehow been configured such that \r and \n both match the internal \n characters. \r will match line endings, and \n will match line endings, and \r\n will match only if there is two line endings in a row (ie, a blank line). Someone from Bare Bones would have to comment to know more for sure.

And no, I wont be messing around with the regex in Keyboard Maestro, \r is CR, \n is LF, \R is the best thing to use in Keyboard Maestro as long as you'll never see OS X 10.10 (which kind of makes me wish I had made Keyboard Maestro 8 require 10.11+), otherwise (?:\r\n|\r|\n) or the equivalent is necessary.

[JMichaelTX]

@Tom

So, I think it would be a good idea to place a big fat warning on the KM Wiki article:

“Do not use the %Return% token for anything, except you really need it for some obscure reason or legacy texts. Do never use it when feeding the variable to the shell! Use \n or %LineFeed% instead.”

I agree Tom.

@peternlewis, any objections or comments?

If not, I will make the changes, and the ones from my corrected summary, to:
token:Return [Keyboard Maestro Wiki]

[JMichaelTX]

@peternlewis

\R will work in macOS 10.11+, in (I believe) any version of Keyboard Maestro.

Thanks. I have corrected my OP.

[peternlewis]

No there should not be a “big fat warning” not use %Return% or \r.

There can be a note to ensure you know which line endings you want to use, and that generally scripts and files will need %LineFeed% or \n, while clipboards probably want %Return% or \r, and anything you write should generally try to handle either.

With minor amendments, the info in your Summary previously is appropriate for the token pages.

[JMichaelTX]

@peternlewis, would you agree that it is better to use a LF unless you have good reason to do otherwise, and always use LF in Shell Scripts?

[peternlewis]

Nope, I use %Return% frequently (I use it in 37 different macros). I don’t use %LineFeed% anywhere.

%Return% is generally fine in any macro except when dealing with variables that will be used in shell scripts.

For Insert Text by Typing or Pasting, or for setting the clipboard, %Return% is fine, and probably more correct than %LineFeed%.