Filter duplicate lines

Is there a built-in way to filter and remove duplicate lines in a variable or clipboard such as in Process duplicate lines in BBEdit?

(I am running Keyboard Maestro 6.4.6 on Mac OS X Version 10.9.4 (Build 13E28))

It’s possible to do in Keyboard Maestro, but quite tricky. You could use the For Each action to iterate through all the lines in a file/variable/clipboard, and then keep track of the last line, and if it is different, add it to the end of a growing variable.

Easier would be to use a BBEdit Text Factory, and execute it from Keyboard Maestro.

Thanks. I’ll go the BBEdit route. I was already using that. Didn’t know you could automate with KM.

Hey Michael,

As you're now aware KM can run BBEdit Text Factories, but BBEdit should be running when you do this. (I experienced quite a bad crash while testing when BBEdit was not already running).

What you might not know is that process duplicate lines is also available via BBEdit's AppleScript dictionary.

This handler works with any text as input. Of course you can change the code to target a BBEdit window or selection quite easily.

AppleScript will typically out perform Text Factories - especially if you run a script-file rather than a text-script.

on processLines(_text)
  tell application "BBEdit"
    set uniqueLines to unique lines of ¬
      (process duplicate lines _text output options {deleting duplicates:true})
  end tell
end processLines

processLines(the clipboard) # Or a KM variable, or whatever.

The shell will make short work of this job if you're willing to sort the lines. The -u switch means unique.

A similar macro using Perl will let you remove duplicates without sorting the output.

#! /usr/bin/env perl -w
use strict;
chomp(my @array = grep { /\S/ } `pbpaste`);
@array = do { my %seen; grep { !$seen{$_}++ } @array };
$, = "\n";
print @array; # NOT SORTED
# print sort {lc $a cmp lc $b} @array; # SORTED CASE-INSENSITIVE

If you're willing to install a small AppleScript extension (Satimage.osax) AppleScript can do this with ease, although like the shell you have to sort the text.

set _text to "one two three four five one two three four five"
set _list to splittext _text using " "
set sortedList to sortlist _list with remove duplicates

So now you have a few more choices.

-Chris

* Edit: had to change the Perl again to get it properly unsorted. 2014/09/13 21:41 CDT.

1 Like

Hi,

If KM has a variable with this result how could I filter it to only remove the extra duplicates but to keep one copy of name in the variable

aa, ab, aa, ab,

so as a result I should only end up with

aa, ab

Then If i wanted to go further and get the count of this how could I do that? the result should be 2 because there are only 2 unique results.

Here's one way, there's plenty of others:

GetUniqueLines.kmmacros (6.0 KB)

Hey Ali,

If I knew I would be dealing with only ASCII variable names I'd do something like this:

-Chris


Delete Dupes and Count.kmmacros (4.7 KB)

2 Likes