Open many URLs

I have nearly 1000 URLs in a CSV file. When the URL is run, a file will be downloaded. But not all the URLs will be correct (in some cases 'objKey=2022' will be ''objKey=2021'', but I do not know which is which).

I need to run all the URLs and need a way of finding out which have run successfully. How might I approach this?

species,https://download/v1/fetch?objKey=2022/species/web_download/wbspet1_range_2022.zip&key=xx
species,https://download/v1/fetch?objKey=2022/species/web_download/bbspet1_range_2022.zip&key=xx
species,https://download/v1/fetch?objKey=2022/species/web_download/wanalb1_range_2022.zip&key=xx
species,https://download/v1/fetch?objKey=2022/species/web_download/wanalb3_range_2022.zip&key=xx

What kind of result do you see if the result is not correct? We could read that in order to determine which runs ran correctly.

One way is, of course, to use a Keyboard Maestro Execute JavaScript for Automation action which partitions the URLS into two lists:

  1. Problems – the urls which could not be opened
  2. Opened – the urls which have been successfully opened.

The intention, in the sketch below, is that you read the full text of your CSV file into a variable called local_urlsCSV

After trying to read each url, it aims to return two lists – problems and successes.

Partitions of url CSV- Lists of 1. Unopenable and 2. Successfully Opened.kmmacros (7.3 KB)


Expand disclosure triangle to view JS source
return (() => {
    "use strict";

    ObjC.import("AppKit");

    // ---------------------- MAIN -----------------------
    const main = () => {
        const urls = lines(kmvar.local_urlsCSV || "");

        return [
            ...bimap(
                xs => ["Problems:", ...xs.map(x => `  ${x}`)]
                    .join("\n")
            )(
                xs => ["Opened:", ...xs.map(x => `  ${x}`)]
                    .join("\n")
            )(
                partitionEithers(
                    urls.flatMap(s => {
                        const url = s.trim();

                        return 0 < url.length
                            ? [openURL(url)]
                            : []
                    })
                )
            )
        ]
            .join("\n\n");
    };


    // ----------------------- JXA -----------------------

    // openURL :: String -> Either String IO String
    const openURL = url => (
        // A url, wrapped in an attempt to open it.
        // ObjC.import('AppKit')
        $.NSWorkspace.sharedWorkspace.openURL(
            $.NSURL.URLWithString(url)
        ) ? (
            Right(url)
        ) : Left(`Could not open "${url}"`)
    );

    // --------------------- GENERIC ---------------------

    // Left :: a -> Either a b
    const Left = x => ({
        type: "Either",
        Left: x
    });


    // Right :: b -> Either a b
    const Right = x => ({
        type: "Either",
        Right: x
    });

    // Tuple (,) :: a -> b -> (a, b)
    const Tuple = a =>
        // A pair of values, possibly of
        // different types.
        b => ({
            type: "Tuple",
            "0": a,
            "1": b,
            length: 2,
            *[Symbol.iterator]() {
                for (const k in this) {
                    if (!isNaN(k)) {
                        yield this[k];
                    }
                }
            }
        });

    // bimap :: (a -> b) -> (c -> d) -> (a, c) -> (b, d)
    const bimap = f =>
        // Tuple instance of bimap.
        // A tuple of the application of f and g to the
        // first and second values respectively.
        g => tpl => Tuple(f(tpl[0]))(
            g(tpl[1])
        );


    // bindLR (>>=) :: Either a ->
    // (a -> Either b) -> Either b
    const bindLR = lr =>
        // Bind operator for the Either option type.
        // If lr has a Left value then lr unchanged,
        // otherwise the function mf applied to the
        // Right value in lr.
        mf => "Left" in lr
            ? lr
            : mf(lr.Right);


    // either :: (a -> c) -> (b -> c) -> Either a b -> c
    const either = fl =>
        // Application of the function fl to the
        // contents of any Left value in e, or
        // the application of fr to its Right value.
        fr => e => "Left" in e
            ? fl(e.Left)
            : fr(e.Right);


    // first :: (a -> b) -> ((a, c) -> (b, c))
    const first = f =>
        // A simple function lifted to one which applies
        // to a tuple, transforming only its first item.
        ([x, y]) => Tuple(f(x))(y);


    // fmapLR (<$>) :: (b -> c) -> Either a b -> Either a c
    const fmapLR = f =>
        // Either f mapped into the contents of any Right
        // value in e, or e unchanged if is a Left value.
        e => "Left" in e
            ? e
            : Right(f(e.Right));


    // lines :: String -> [String]
    const lines = s =>
        // A list of strings derived from a single string
        // which is delimited by \n or by \r\n or \r.
        0 < s.length
            ? s.split(/\r\n|\n|\r/u)
            : [];


    // partitionEithers :: [Either a b] -> ([a],[b])
    const partitionEithers = xs =>
        // A tuple of two lists:
        // first all the Left values in xs,
        // and then all the Right values in xs.
        xs.reduce(
            (a, x) => (
                "Left" in x
                    ? first(ys => [...ys, x.Left])
                    : second(ys => [...ys, x.Right])
            )(a),
            Tuple([])([])
        );



    // second :: (a -> b) -> ((c, a) -> (c, b))
    const second = f =>
        // A function over a simple value lifted
        // to a function over a tuple.
        // f (a, b) -> (a, f(b))
        xy => Tuple(
            xy[0]
        )(
            f(xy[1])
        );

    // --------------------- LOGGING ---------------------


    // showLog :: a -> IO ()
    const showLog = (...args) =>
        // eslint-disable-next-line no-console
        console.log(
            args
                .map(JSON.stringify)
                .join(" -> ")
        );

    // sj :: a -> String
    const sj = (...args) =>
        // Abbreviation of showJSON for quick testing.
        // Default indent size is two, which can be
        // overriden by any integer supplied as the
        // first argument of more than one.
        JSON.stringify.apply(
            null,
            1 < args.length && !isNaN(args[0])
                ? [args[1], null, args[0]]
                : [args[0], null, 2]
        );

    return main();

    return sj(
        main()
    );
})();

Are you just triaging the list, or do you want to actually download the file when it's a valid URL?

2 Likes

Want to actually download the file when it's a valid URL.

I also want to download the file when it's an invalid URL...but I do not know the valid URL without actually going to the URL.

And how are you downloading the file? What's the difference in result between a file that can be downloaded and a file that cannot?

Asking these questions because it looks like you aren't trying to directly download a file -- the URLs above suggest that you're querying an API, in which case we may not be able to use "normal" methods like the result in a curl response header. But it should mean you get a proper API response for a non-existing file, and you can use that...

When I click the URL, the file downloads to my computer. I changed some things in the URLs I posted in original post, so they didn't contain my key etc.

Nige (and I) asked you what is the difference between a successful result and an unsuccessful one.

That's great. So tell us what you see when you specify an invalid URL. That may be the key to solving this.

So you're doing this via a web browser? And will be quite happy for KM to automate this in a browser?

You also stripped the domain, so even people who may be able to access the resource aren't able to try.

Not at my desk to try it out, but I reckon For Each action that opens each URL is all I need. This works in Shortcuts app on my iPhone.

Only tricky bit is that if a URL is invalid, the macro needs to move on to the next URL.

And why would it not? KM doesn't care if Safari can open an URL or not:

image

...will open each URL in turn, with the middle one resulting in the usual "Can't find the server..." page.