Curl Selector Fetcher.zip (1.3 KB)
just to be sure here is the version with correct permission
Curl Selector Fetcher.zip (1.3 KB)
with updated permission.
the same as before but comprised correctly in order to preserve permissions.
after install if still having permission issues
After installation, run this to ensure correct permissions:
chmod +x ~/Library/Application\ Support/Keyboard\ Maestro/Keyboard\ Maestro\ Actions/Curl\ Selector\ Fetcher/script.sh
this plug-in captures values of specific elements on website. a piece of text or value.
to identity the element to capture it you can use an xpath or selector from inspect element on chrome .
to its very easy to caputre any value from any page. can be prices, names, status or anything you need to caputre from a page.
it does it via curl command, and uses xmlint for xpath ( requires no installation) but for css selectors require pup to be present on the machine.
it works perfect on my machine but cant make it to work on onther machines.
maybe is lack of pup maybe is permissions.
if samone can look on the code and tell me why it only work on my machine, i would realy helpfull. thanks
here is the code:
Keyboard Maestro Action.plist
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Name</key>
<string>Curl Selector Fetcher</string>
<key>Script</key>
<string>script.sh</string>
<key>Title</key>
<string>Curl Selector Fetcher</string>
<key>Results</key>
<string>Window|Briefly|Clipboard|Variable</string>
<key>Parameters</key>
<array>
<dict>
<key>Label</key>
<string>URL</string>
<key>Name</key>
<string>URL</string>
<key>Type</key>
<string>TokenString</string>
</dict>
<dict>
<key>Label</key>
<string>Selector</string>
<key>Name</key>
<string>Selector</string>
<key>Type</key>
<string>TokenString</string>
</dict>
<dict>
<key>Label</key>
<string>Selector Type</string>
<key>Name</key>
<string>Selector_Type</string>
<key>Type</key>
<string>PopupMenu</string>
<key>Default</key>
<string>CSS</string>
<key>Menu</key>
<string>CSS|XPath</string>
</dict>
</array>
</dict>
</plist>
and here is the script that does that actual scraping
script.sh
#!/bin/bash
export PATH=/usr/local/bin:/opt/homebrew/bin:$PATH
url="$KMPARAM_URL"
selector="$KMPARAM_Selector"
stype="$KMPARAM_Selector_Type"
if [[ -z "$url" || -z "$selector" ]]; then
echo "Error: URL or Selector parameter is empty."
exit 1
fi
if [[ "$stype" == "CSS" ]]; then
if ! command -v pup &> /dev/null; then
echo "Error: 'pup' not found. Install with: brew install pup"
exit 1
fi
curl -sL "$url" | pup "$selector" text{}
elif [[ "$stype" == "XPath" ]]; then
curl -sL "$url" | xmllint --html --xpath "$selector" - 2>/dev/null
else
echo "Error: Invalid Selector Type."
exit 1
fi
any help to make this work on any machine would be super helpfull
I recommend to readers of this thread that they remove the "&& killall ..." part of your command, and close the KM Engine manually instead. Killing the Engine can potentially cause corruption of the KM files where one's macros are stored.
Not additional options -- but a clear explanation of what the plugin is for, perhaps with a couple of examples, would probably increase people's interest levels.
Hi, @Nelo_Reis, thanks for your post!
Also, if your goal is to share this with others, if possible, consider packaging your logic in a subroutine macro rather than a plugin. The latter are generally more problematic when sharing with others.
Hi jims
i do this hundreds of times per day on my work.
given any URL (public, no login no paywall)
and given an xpath from google chrome ( what element to get)
the script curls the adress and extract the data of a specific element on the page, be it text or numbers. and if needed regex filtering of the received data.
extract data from xpath.kmmacros (3.4 KB)
basic example of the pipeline .
but i still prefer all of it be a single action , cleaner and faster.
granted i might have a hard time explaining this to someone not familiar with xpath or web selectors. the idea is good, works ultra fast and super useful for someone that needs to grab data from public sites, lets say booking.com or any sites that updates prices or anything else constantly.
