How to remove repeated words in a variable

[Tinker]

If I have a keyboard maestro variable, which has been taken from the html page title of an Amazon product page, there are often many REPEATED words:

for example:

Smile you are on camera 24 HOURS CCTV MONITORING CCTV SIGN PVC plastic film stickers size 70mm x 80mm shop cctv sticker sign

this words is repeated 3 times: cctv

What can I do ONLY have it appear once?

(The cause of the problem is SEO spam!)

I have tried using the substring step to reduce the title length, and while it's quite good, it often chops up words in the middle, which looks ugly.

[noisneil]

Try this:

Remove Repeated Words.kmmacros (20 KB)

Macro screenshot

[Tinker]

Ooh you’re using something called shell script!
I’m very intrigued. Before I try it, I’m inclined to look at some basic shell script commands to try and get a very basic understanding of what your code is doing.
(If this works then you’re a genius!)

Edit: the output is

Smile you are on camera 24 HOURS CCTV MONITORING SIGN PVC plastic film stickers size 70mm x 80mm 
shop sticker

Wow well done!

[Tinker]

Can it be further refined to exclude the word Joseph? this is because there is a brand called Joseph Joseph. This is a super rare thing and not really worth worrying about

[ComplexPoint]

In a variation using an Execute JavaScript for Automation action, you could pre-include joseph in the dictionary of words already seen:

Expand disclosure triangle to view JS source

(() => {
    "use strict";

    // Word duplications pruned.

    const main = () =>
        kmValue(
            kmInstance()
        )(
            "local__Input"
        )
        .split(/\s+/u)
        .reduce(
            ([seen, pruned], w) => {
                const k = w.toLocaleLowerCase();

                return k in seen
                    ? [seen, pruned]
                    : [
                        Object.assign(seen, {[k]: true}),
                        `${pruned} ${w}`
                    ];
            },
            [{joseph: true}, ""]
        )[1];


    // --------------- KM LOCAL VARIABLES ----------------

    // kmInstance :: () -> IO String
    const kmInstance = () =>
        ObjC.unwrap(
            $.NSProcessInfo.processInfo.environment
            .objectForKey("KMINSTANCE")
        ) || "";


    // kmValue :: KM Instance -> String -> IO String
    const kmValue = instance =>
        k => Application("Keyboard Maestro Engine")
        .getvariable(k, {instance});


    // MAIN ---
    return main();
})();

[noisneil]

For ease of use, you could set your exclusion words to a variable, like this:

Remove Repeated Words.kmmacros (29 KB)

Macro screenshot

[Tinker]

Much obliged thank you so much.
When testing with Amazon.co.uk there is another repeated thing - & that I want to exclude

Am I entering the local exceptions correctly, with a return between each word?

The raw input is :

Joseph Joseph Salad. Toy Salad Preparation Set For Children Aged 2+. With Chop2Pot Cutting Mat & Sound Effect Salad Dressing : Amazon.co.uk: Home & Kitchen

the output is :

Joseph Joseph Salad. Toy Salad Preparation Set For Children Aged 2+. With Chop2Pot Cutting Mat & Sound Effect Dressing : Amazon.co.uk: Home Kitchen

So at the very end, it should say Home & Kitchen. It's a tiny problem - I can live with Home Kitchen

My eternal thanks for this btw

UPDATE - the & thing works now!