Check for duplicate lines in a variable and remove

[sputernz]

Let’s say the variable “MatchList” contains the text:

Application “A”
Application “B”
Application “C”
Application “A”

Now, there are 4 lines here, but only 3 different ‘Applications’.
So, my question is, can I search the variable for multiple instances of ‘Application “A”’ and do something with that? (i.e. delete that line from the variable)

the text will be different each time, so i'm not sure how I would do it with the 'Search & Replace' action.

Thanks in advance!

[JMichaelTX]

@sputernz

Let’s say the variable “MatchList” contains the text:

Application “A”
Application “B”
Application “C”
Application “A”

Now, there are 4 lines here, but only 3 different ‘Applications’.
So, my question is, can I search the variable for multiple instances of ‘Application “A”’ and do something with that? (i.e. delete that line from the variable)

Here is one simple approach that should work OK for a short list (say < 100 lines).
It works fast on 5-6 lines, but will probably bog down with hundreds/thousands of lines since it replaces the entire string when it finds a dup -- not very efficient.

But it provides a KM only solution that requires only 2 Actions.

EDIT: See VER: 2.0 2020-06-23 below which provides a much better JavaScript solution.

Below is just an example written in response to your request. You will need to use as an example and/or change to meet your workflow automation needs.

Please let us know if it meets your needs.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Example Output

MACRO:   Remove Dups from List [Example]

-~~~ VER: 1.0    2020-06-23 ~~~
Requires: KM 8.2.4+   macOS 10.11 (El Capitan)+
(Macro was written & tested using KM 9.0+ on macOS 10.14.5 (Mojave))

DOWNLOAD Macro File:

Remove Dups from List [Example].kmmacros
Note: This Macro was uploaded in a DISABLED state. You must enable before it can be triggered.

---

[JMichaelTX]

@sputernz

So, my question is, can I search the variable for multiple instances of ‘Application “A”’ and do something with that? (i.e. delete that line from the variable)

Here is another alternative, using the power of JavaScript Arrays and Sets in JavaScript for Automation (JXA).
This is my preferred solution, and should work well even for thousands of lines.

Below is just an example written in response to your request. You will need to use as an example and/or change to meet your workflow automation needs.

Please let us know if it meets your needs.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Example Output

MACRO:   Remove Dups from List [Example]

-~~~ VER: 2.0    2020-06-23 ~~~
Requires: KM 8.2.4+   macOS 10.11 (El Capitan)+
(Macro was written & tested using KM 9.0+ on macOS 10.14.5 (Mojave))

DOWNLOAD Macro File:

Remove Dups from List [Example].kmmacros
Note: This Macro was uploaded in a DISABLED state. You must enable before it can be triggered.

---

[August]

@JMichaelTX

Here is one simple approach that should work OK for a short list

I think there's a mistake in @JMichaelTX's macro above. His original Regular Expression search:

has four tagged sequences but he only uses three of them in the replacement string. I think his replacement string will end up deleting everything from the second instance of %Local__Line% to the end of the list.

The fix is simple, the replacement should be:

\1\2\3\4

which will insert:
\1: The beginning of the list, before the first instance of %Local__Line%
\2: The first instance of %Local__Line%
\3: Everything up to the second instance of %Local__Line%
\4: Everything after the second instance of %Local__Line%

I hope this helps. It's a great example of using Regular Expressions.

[griffman]

I posted a macro a few years ago that rapidly de-dupes a list, but also extracts the dupes:

MACRO: Quickly Remove Duplicates From a List in a Variable

There have been some posts on this (not many) over the years, but most were beyond my skill level, involving JXA scripting. Or they relied on another third-party app (BBEdit) to do the sorting. I was looking for an all KM solution, but it had to be fast and (relatively) easy to implement. The following may be of use if you have long lists and need them de-duped. Every Mac user has a great de-duping tool built in on the Unix side of their OS: The combination of sort and uniq. The sort command do…

Using the long list in that demo (240 words), it takes it about 0.3 seconds to process and display all three lists (original, dupes, and de-duped). No regex at all, no Javascript, just two Unix commands (sort and uniq).

-rob.