For some reason none of these are working for me anymore. I am on Safari 14 with "Allow JavaScript from Apple Events" checked in the Develop tab. Any idea? It use to open in several tabs and then for a while it would go on endlessly and I would have to stop the script from running and now it just doesn't work.
I tried the code from the later posts to no avail. Web Browser - Safari Macros.kmmacros (129.3 KB)
I'll try to take a look over the weekend. I seems to remember a shift in the API for tabs, but in any case, I'll report on what I find.
Thank you!
Mmm ... I may have to defer to @peternlewis for up-to-date advice on sources of delay in Execute JavaScript in Safari actions,
but this (simple renovated test below) seems to execute:
unless under Safari > Develop > [Device name] I have checked:
Automatically Show Web Inspector for JS Contexts
in which case background activation of the debugger causes a delay of several seconds.
(Is it possible that that option is checked on your system ?)
In the draft below it just displays the MD links in a window, but you could, of course, redirect them to a clipboard or KM variable.
If we can sort out the delay issues first, then perhaps we can see if anything else needs renovating ?
First 10 links on Safari page as Markdown.kmmacros (2.7 KB)
(() => {
'use strict';
const main = () => {
const
intLinks = 10,
r = document.evaluate(
'//a', document,
null, 0, null
);
return until(
pair => intLinks <= pair[0].length || (
!pair[1]
)
)(
pair => {
const node = pair[1];
return [
Boolean(node.href) ? (
pair[0].concat([
[
node.text.trim() || node.href,
node.href
]
])
) : pair[0],
r.iterateNext()
];
}
)([
[], r.iterateNext()
])[0]
.map(([label, link]) => `[${label}](${link})`)
.join('\n');
};
// ------------------- GENERIC -------------------
// until :: (a -> Bool) -> (a -> a) -> a -> a
const until = p =>
f => x => {
let v = x;
while (!p(v)) v = f(v);
return v;
};
return main();
})();
Thank you for looking into this. Yeah I don't have "Automatically Show Web Inspector for JS Contexts" checked. Hopefully @peterlewis has an idea of what is stopping this from working. This has been such a great command that I use dozens of times a day.
Was the test macro above also slow ?
It turns out that Google have changed the HTML structure of their results pages.
We used to be able to distinguish between the actual search result links and the less interesting menu links by looking for a node class labelled 'r', on the path to 'a' nodes (hyperlinks)
//*[@class='r']/a
but looking at their search result pages, I see that they are no longer using that class name.
If I search for something like "Category Theory", and then inspect the link of the result labelled What is Category Theory Anyway ?, its path on the HTML of that Google search results page turns out, these days, to be:
//*[@id="kp-wp-tab-overview"]/div[4]/div[2]/div[2]/div/div[1]/a
which we could abbreviate to:
//*[@id="kp-wp-tab-overview"]/*/a
but I'm not sure how general or reliable that id value will prove to be. I'll experiment a little.
I think, however, that we may be up against a concern on their part (probably quite understandable) to make page scraping much harder. The paths now contain quite a significant number of unpredictable identifying strings, and they seem to be evading the usual pattern of .href etc node fields.
Second test :: does this one seem to list the set of links that you would want to open in tabs ?
Show first 10 links on a Google results page.kmmacros (4.1 KB)
(() => {
'use strict';
const main = () => {
const
numberToSkip = 0,
harvestSize = 10,
infixAvoided = 'google',
prefixNeeded = 'https://www.google.com/search?q=';
const
intLimit = numberToSkip + harvestSize,
r = document.evaluate(
'//a', document,
null, 0, null
);
return until(
pair => intLimit <= pair[0].length || (
!pair[1]
)
)(
pair => {
const
node = pair[1],
href = node.href.trim() || '';
return [
Boolean(href) && !href.includes(infixAvoided) ? (
pair[0].concat([
node.href
])
) : pair[0],
r.iterateNext()
];
}
)([
[], r.iterateNext()
])[0]
.join('\n')
// .map((x, i) => [x, i])
};
// ------------------- GENERIC -------------------
// until :: (a -> Bool) -> (a -> a) -> a -> a
const until = p =>
f => x => {
let v = x;
while (!p(v)) v = f(v);
return v;
};
// MAIN ---
return main();
})();
One this system (Catalina, Safari 14), this new version seems to work, opening the first 10 results in ten new Safari tabs.
Let me know if there is still a problem on the system you are using.
UPDATED (now aims to avoid any duplications in the harvest, and avoid opening duplicate tabs)
First 10 links on a Google results page in 10 new Safari tabs.kmmacros (24.7 KB)
(() => {
'use strict';
// Rob Trew @2020
// @ComplexPoint
// main :: IO ()
const main = () => {
const
numberToSkip = 0,
harvestSize = 10,
infixAvoided = 'google',
prefixNeeded = 'https://www.google.com/search?q=';
const
intLimit = numberToSkip + harvestSize,
r = document.evaluate(
'//a', document,
null, 0, null
),
harvest = until(
pair => intLimit <= Object.keys(
pair[0]
).length || !pair[1]
)(
pair => {
const
node = pair[1],
href = node.href.trim() || '',
site = siteFromURL(href);
return [
Boolean(href) && (
!href.includes(infixAvoided)
) && !pair[0][site] ? (
Object.assign(pair[0], {
[site]: href
})
) : pair[0],
r.iterateNext()
];
}
)([{}, r.iterateNext()])[0];
return Object.keys(harvest).map(
site => harvest[site]
).join('\n');
};
// ------------------- GENERIC -------------------
// siteFromURL :: String -> String
const siteFromURL = url =>
url.split('/').slice(0, 3).join('/');
// until :: (a -> Bool) -> (a -> a) -> a -> a
const until = p =>
f => x => {
let v = x;
while (!p(v)) v = f(v);
return v;
};
// MAIN ---
return main();
})();
(() => {
'use strict';
// Rob Trew @2020
// ComplexPoint
// main :: IO ()
const main = () => {
const
linkLines = Application('Keyboard Maestro Engine')
.getvariable('linkList'),
linkList = linkLines.split('\n');
return either(
msg => alert('First 10 Google results')(msg)
)(
x => x
)(
0 < linkList.length ? (() => {
const
safari = Application('Safari'),
ws = safari.windows;
return bindLR(
0 < ws.length ? (
Right(ws.at(0))
) : Left('No window open in Safari.')
)(
safariWindow => {
const
tabs = safariWindow.tabs,
alreadyOpen = tabs.url()
.reduce(
(a, x) => Object.assign(
a, {
[str(hash(x))]: x
}
), {}
);
const newlyOpened = (
safari.activate(),
linkList.flatMap(
link => alreadyOpen[
hash(link)
] ? [] : [(
tabs.push(
safari.Tab({
url: link
})
),
link
)]
)
);
return Right(newlyOpened);
}
);
})() : Left('No harvested links found')
);
};
// ------------- JXA AUTOMATION LIBRARY --------------
// alert :: String => String -> IO String
const alert = title =>
s => {
const sa = Object.assign(
Application('System Events'), {
includeStandardAdditions: true
});
return (
sa.activate(),
sa.displayDialog(s, {
withTitle: title,
buttons: ['OK'],
defaultButton: 'OK'
}),
s
);
};
// --------------------- GENERIC ---------------------
// Left :: a -> Either a b
const Left = x => ({
type: 'Either',
Left: x
});
// Right :: b -> Either a b
const Right = x => ({
type: 'Either',
Right: x
});
// bindLR (>>=) :: Either a ->
// (a -> Either b) -> Either b
const bindLR = m =>
mf => undefined !== m.Left ? (
m
) : mf(m.Right);
// either :: (a -> c) -> (b -> c) -> Either a b -> c
const either = fl =>
// Application of the function fl to the
// contents of any Left value in e, or
// the application of fr to its Right value.
fr => e => 'Either' === e.type ? (
undefined !== e.Left ? (
fl(e.Left)
) : fr(e.Right)
) : undefined;
// hash :: String -> Int
const hash = s =>
0 < s.length ? (
s.split('').reduce((a, x) => {
const
hash = x.charCodeAt(0) + (
(a << 5) - a
);
return hash & hash;
}, 0)
) : 0;
// str :: a -> String
const str = x =>
Array.isArray(x) && x.every(
v => ('string' === typeof v) && (1 === v.length)
) ? (
x.join('')
) : x.toString();
// MAIN ---
return main();
})();
Sweet that works but for some searches it seems to open the same thing over and over again. Probably because google shows links to the same location. Other searches seems to show all different links. Either way this is fantastic, thank you very much for the update and working on that!
to open the same thing over and over again
There might be a pattern there that we can make sense of and fix – any examples that look like they might be reproducible ?
I've updated the macro above.
It should, I think, now skip duplicate links and plough on until it has either harvested 10 distinct links, or exhausted the links available on the page.
It also aims to avoid opening duplicate tabs.
(If that still seems to generate duplicates in some cases, perhaps you could show me an example or two ?)
Thank you I downloaded it and tried it and I am getting an error for some reason. 
That sounds interesting. I haven't yet managed to reproduce it here, I wonder what is different about our systems (or the searches we are testing with ?)
@skillet, here is my Macro that uses the JavaScript querySelectorAll, rather than Xpath, to extract the links from the Google Search page.
I have found that usually it is much easier to use querySelectorAll than Xpath.
That is still so in this case, but Google has made it much harder with their latest changes.
My macro seems to work well (tested with Google Chrome 88.0.4324.96 (4324.96) on macOS 10.14.6 (Mojave)) and should work fine in Safari as well.
Be aware that ==this script could fail if Google changes key elements on this web page.==
Note that the results of the Chrome Search page can vary in format/layout based on what you search for. I have excluded links in the subsection of the first result, and all links in the "People Also Ask" section:
The JavaScript in my Macro returns ALL of the UNIQUE primary links found on the Search Results page.
This allows you to easily control in the Macro itself how many links you want to process.
I use the KM For Each action to open each URLs until the max number is reached. This uses the KM Open URL set to use the "Default App" (which will be your default Web Browser), but you can easily change it to open in a specific Browser, and/or to open in a new window or new tab -- whatever you prefer:
I also provided a disabled Action to display the list of links, in case you want to just examine the list or debug.
Please let me know if you have any problems with this macro.
Below is just an example written in response to your request. You will need to use as an example and/or change to meet your workflow automation needs.
Please let us know if it meets your needs.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-~~~ VER: 1.0 2021-01-27 ~~~
Requires: KM 8.2.4+ macOS 10.11 (El Capitan)+
(Macro was written & tested using KM 9.0+ on macOS 10.14.5 (Mojave))
Extract Unique Links from Google Search [Example].kmmacros
Note: This Macro was uploaded in a DISABLED state. You must enable before it can be triggered.
Author.@JMichaelTX
PURPOSE:
HOW TO USE
MACRO SETUP
REQUIRES:
TAGS: @JSIB @GoogleChrome @URL @Links @Example
USER SETTINGS:
Oh man you guys are amazing, this works marvelously I don't know how you guys know so much stuff! This works great in Safari 14 and 13.
Sorry about that I didn't realize I tried that on a computer that had Safari 13 on it. I did try your previous script you made a few days ago and it worked in Safari 13. I changed the results to "display results in a window" and it still showed up small on the side. I changed it to large and the same things. I changed the name of the macro to make sure I was getting results for this macro and I was and changed it to "Ignore Results" for the second action and it still gave me results. I think it is just following the "Notify on Failure" in the cog of the action in Keyboard Maestro. When I turn that off I get nothing.
JMichaelTX, thank you so much, this is working great for me. I am running Safari 14.1.1 on Big Sur.
Also thank you skillet for mention to turn on "Allow JavaScript from Apple Events" to allow this macro to work.
As far as meeting my needs, this is wonderful! An excelent macro to add would be a version for DuckDuckGo. I have my searching macros set up for DuckDuckGo and Google and use shortcuts depending (basically for if DuckDuckGo results are strange).
