How to scrape a web page to input data into an Excel spreadsheet?

Questions & Suggestions
, , ,
#1

I have no idea if this is even within the realm of the possible, but if it is, I have no doubt you folks would know. I drive for Lyft, the rideshare company, and I drive between a couple and 20 rides per night. Lyft publishes (after several clicks to drill down) data for each ride that I'd like to scrape (Lyft doesn't offer this data in downloadable form... yet!) and import into my Excel spreadsheet for records and analysis.

In other words, using either Keyboard Maestro, or the macros in (macOS 2011) Excel, itself, is it possible to scrape the data from this:

Example of a Lyft ride detail page(s)

Here's an example of a single ride in video format: https://youtu.be/_sNET8bE2qM

into this:

My Excel Spreadsheet

If there is, it would save me untold consternation every morning.

#2

You might be able to create an Execute Javascript in Front Browser macro and then save the results to a variable. Then, you could append the variable contents to a file with applescript or some other way. You might be able to automate a whole spreadsheet and also automatically fill out parts of the page with javascript.

Some pages are trickier than others, so I'm sharing some basic macros I use as starting points. I've gathered the ideas from forums and different places.. hopefully they'll be useful to you! Note: many of these KM macros have a "Front Browser" dropdown, super convenient to locate the web element vs inspecting the page!

WEB UI MANIPULATION MACROS.kmmacros (11.1 KB)

image
image.png706×2726 215 KB

#3

Yes. As @jayknowstheway suggests, Execute a JavaScript in Browser action should be able to extract the data for you. Most likely each data item you want is a HTML element with a unique ID or class that you can use querySelector() to get the element and it's value.

To be more specific, we'd need the actual URL, or the HTML code for the page (uploaded in a Code Block).

Once you have the data in KM Variable(s), it should also be fairly easy to add to your Excel workbook. There are numerous examples of this in the forum. Just search on "excel".