Check for duplicate lines in a variable and remove

[griffman]

I posted a macro a few years ago that rapidly de-dupes a list, but also extracts the dupes:

MACRO: Quickly Remove Duplicates From a List in a Variable

There have been some posts on this (not many) over the years, but most were beyond my skill level, involving JXA scripting. Or they relied on another third-party app (BBEdit) to do the sorting. I was looking for an all KM solution, but it had to be fast and (relatively) easy to implement. The following may be of use if you have long lists and need them de-duped. Every Mac user has a great de-duping tool built in on the Unix side of their OS: The combination of sort and uniq. The sort command do…

Using the long list in that demo (240 words), it takes it about 0.3 seconds to process and display all three lists (original, dupes, and de-duped). No regex at all, no Javascript, just two Unix commands (sort and uniq).

-rob.