Replacing Duplicates Using RegEx

[theandouz]

I have been trying to use RegEx to search and replace the duplicates of a list but with my limited knowledge of RegEx, I'm not sure if it is possible. I need to keep this formatted this way as it keeps the proper spacing when pasting in a Google Sheet. The copied text is constantly changing so I can't use a set Search for (OH) or (IMAGE). I appreciate any help and let me know if any clarification is needed.

Copied Text in Variable
OH
OH
OH
OH
OH
OH
IMAGE
IMAGE
IMAGE
IMAGE
IMAGE
IMAGE

Format Needed
OH
%return%
%return%
%return%
%return%
%return%
IMAGE
%return%
%return%
%return%
%return%
%return%

[JMichaelTX]

@theandouz

I have been trying to use RegEx to search and replace the duplicates of a list but with my limited knowledge of RegEx, I'm not sure if it is possible

RegEx is NOT well-suited for this task.

I would use either ASobjC or JXA -- both have a SET object that automatically removes dups.

JXA:
let uniqueName = [...new Set(allNames)];

Use the KM JavaScript for Automation to run the script.
This also shows you how to get a KM variable from JXA, and how to output back to KM.

[rlivingston]

@theandouz

The copied text is constantly changing so I can't use a set Search for (OH) or (IMAGE). I appreciate any help and let me know if any clarification is needed.

So it might be
CAT
CAT
CAT
MOUSE
MOUSE
MOUSE
MOUSE

---

or

LORIS
LORIS
MONEY
MONEY
MONEY
FROG
FROG

Are these all possible things you might need to deal with?

[rlivingston]

You might try this. You could do this in BBEdit which is a text editor that you should own IMO. BBEdit supports RegEx Find and Replace.

This approach requires applying three search and replace patterns in RegEx sequentially.

First, settle upon a character that does not normally occur in your text. For the sake of an example, let's assume that "@" does not occur in the text you are trying to deal with. I will use "@" in the proposal below.

---

1. Find: ^(.*)$\n^(\1)
   Replace: \1\n@\1

2. Find: ^(@?(.*))$\n^(\2)
   Replace: \1\n@\2

3. Find: ^@.*
   Replace: %return%

I will not try and argue that this is the most elegant way to deal with the task at hand. But in my trials, it seems to work. Depending on how often you have to do this task, this approach might be sufficient.

[tiffle]

Hi @theandouz,
Since I've done something similar to this in the past I'll give you a purely KM way of solving your problem without the use of regex or any other programming language or tool.

Here's the macro for you to download:
Test Unique Lines.kmmacros (7.6 KB)

You'll need to enable it and set up a trigger for it (if you want). I also don't use the clipboard simply as it makes testing easier. Instead I keep the text to be processed in the variable LocalInputText and the output goes into the variable LocalOutputText. I use a Display Text action to show the end result.

This is what the macro looks like:

and the output from it looks like this:

I hope this helps - obviously you can change it to suit your needs.

EDIT: simplified the macro.

[theandouz]

Yes that is correct those are possible options

[theandouz]

Thanks for your reply! I'd love to use BBEdit but unfortunately I'm using my work computer and I can't install any programs on the computer

[theandouz]

Thank you! This is great I'm just running into a weird issue on my end where this macro works on my laptop (running KM 9) but not on my desktop (running KM 8). Really weird but I'm gonna try and see if something is conflicting

[tiffle]

It could be that the Append to Variable action is only available in later versions of KM (I think)

[theandouz]

I didn't notice the append in the photo but yes that is what is causing it to fail

[tiffle]

You should be able to install KM v9 on your desktop if you own the license for it on your MacBook.

Or replace the Append actions with ones like this:

and

[theandouz]

Yea I have to talk to my IT apartment about it but I appreciate your help!

[tiffle]

Re-read my previous reply as I've just edited it with a suggestion.

[theandouz]

Perfect! It works now thank you!

[JMichaelTX]

@JMichaelTX

RegEx is NOT well-suited for this task.

I would use either ASobjC or JXA -- both have a SET object that automatically removes dups.

JXA:
let uniqueName = [...new Set(allNames)];

Use the KM JavaScript for Automation to run the script.

@theandouz, I realize that if you are not familiar with JavaScript and JXA, then that might not be as simple as I implied.

So, here's a complete Macro that performs this task.
You can use the JXA script it contains as a reusable script. All you need to do is set a KM Variable "Local__SourceStr" to your data, and call the script.

MACRO: Remove Dup Items in List [Example]

[theandouz]

Thanks JMichaelTX! Yea I have been trying to learn a bit of JavaScript and JXA for macros when needed but can be challenging. Thanks for your help with the script!

[ikenassi]

Write the data to a temp file. then use the shell command "uniq", and read back the result.

[ccstone]

@ikenassi

Write the data to a temp file. then use the shell command "uniq"

There's nothing wrong with using uniq, but why would you want to write to a file?

Bash ⇢ Uniq v1.00.kmmacros (5.7 KB)

Caveat:

Uniq requires the inputted duplicated lines to be adjacent to one another.

If they're not adjacent then you need to use sort -u instead of uniq.

-Chris

[ikenassi]

You can redirect from the clipboard.

nassi@nassi.com
www.nassi.com/
+1-408-390-8281

Beware fat thumbs

[ccstone]

@ikenassi

You can redirect from the clipboard.

The Clipboard can easily be piped into the shell:

pbpaste | uniq

-Chris