Thank you! This is great I'm just running into a weird issue on my end where this macro works on my laptop (running KM 9) but not on my desktop (running KM 8). Really weird but I'm gonna try and see if something is conflicting
It could be that the Append to Variable action is only available in later versions of KM (I think)
I didn't notice the append in the photo but yes that is what is causing it to fail
You should be able to install KM v9 on your desktop if you own the license for it on your MacBook.
Or replace the Append actions with ones like this:
and
1 Like
Yea I have to talk to my IT apartment about it but I appreciate your help!
Re-read my previous reply as I've just edited it with a suggestion.
1 Like
Perfect! It works now thank you!
1 Like
@theandouz, I realize that if you are not familiar with JavaScript and JXA, then that might not be as simple as I implied. 
So, here's a complete Macro that performs this task.
You can use the JXA script it contains as a reusable script. All you need to do is set a KM Variable "Local__SourceStr" to your data, and call the script.
MACRO: Remove Dup Items in List [Example]
1 Like
Thanks JMichaelTX! Yea I have been trying to learn a bit of JavaScript and JXA for macros when needed but can be challenging. Thanks for your help with the script!
Write the data to a temp file. then use the shell command "uniq", and read back the result.
1 Like
There's nothing wrong with using uniq, but why would you want to write to a file?
Bash ⇢ Uniq v1.00.kmmacros (5.7 KB)
Caveat:
Uniq requires the inputted duplicated lines to be adjacent to one another.
If they're not adjacent then you need to use sort -u instead of uniq.
-Chris
2 Likes
You can redirect from the clipboard.
nassi@nassi.com
www.nassi.com/
+1-408-390-8281
Beware fat thumbs
1 Like
The Clipboard can easily be piped into the shell:
pbpaste | uniq
-Chris
Next up: Peter and his solution using Perl
3 Likes
You can do the job with a 1-liner, but getting it to be fault-tolerant takes more work.
With this script I wanted to be sure to leave items in their original order.
There's a very slick way of removing duplicates using a hash, but it will reorder items according to its own whims. That makes it useless unless companioned by a sort routine (IMO).
-Chris
1 Like
Hey Folks,
I figured out how to do this much more compactly.
#!/usr/bin/env perl -sw
my %lines;
while (<>) {
if ( m!(^\S.+)! ) { print if not $lines{$1}++; }
}
This does use a hash, but not in the way I refer to above – so the sort order is preserved.
Blank lines are removed, and it's tolerant of a non-terminal linefeed.
-Chris
1 Like
Great! But how on Earth would we use that in KM?
I wanted to take a stab at this myself, so here's my version. It's keyboard maestro only, no shell scripts or javascript and uses a regex. Finds duplicates even if they aren't next to each other, retains order of lines and puts a blank line where each duplicate was found.
Basically loops through the source text a line at a time. Checks (via regex) if that line has already been added to the results, adds a blank line if it has, adds the test line if it hasn't.
I think the only bit of the regex that might be unusual is that it needs to treat each line in the result independently. the global flag (?m) at the beginning of the regex does this. See Regular Expressions [Keyboard Maestro Wiki] for more global flags (notably change the global flag to (?mi) will make the regex case-insensitive as well.
I added some entries to the test text given in the first message. Wanted to make sure partial line matches weren't being used. And test the case-sensitivie/insensitive regex.
Results
Macro Image
Macro:
Get Unique Lines.kmmacros (9.7 KB)
1 Like
If you look at my previous post you see exactly how.
-Chris
As a footnote to all this, a quick reminder that meaning of duplicate can vary a bit between particular contexts and jobs.
- Are two names duplicates if they vary only in case ?
- Are two dateTimes duplicates if they vary only by milliseconds ?
- are 4 and 4.0 duplicates of each other ?
That will vary a bit with the task in hand. Scripted solutions can make it easier to define and adjust slightly more flexible equivalences when they are needed.
3 Likes