Working with structured data

Questions & Suggestions
1

I’m finding myself often needing to work with structured data, e.g.:

  • Key/value pairs
  • CSV data
  • JSON data

I was wondering whether Peter plans any evolution of KM into this area? i.e. being able to read structure data, and the provide methods for accessing the data?

As an example, CoinMarketCap has an API that returns all currencies listed on their site:

https://api.coinmarketcap.com/v2/listings/

Another API call can get data about a particular currency, e.g. for bitcoin:

https://api.coinmarketcap.com/v2/ticker/1/

If I had a list of currencies I own, I’d like to be able to:

  1. For each currency, get their ID, so that…
  2. I could then fetch the ticket URL and pull out the current price

Thoughts?

1 Like
2

I use JSON.parse in an Execute JavaScript for Automation action for this.

( e.g. Trying to parse csv file into variables )

1 Like
3

Does that require knowing JavaScript? (I don’t, unfortunately…)

4

Probably eventually, something. Its hard to know what is appropriate.

The Dictionaries introduced in v8.0 are basically Key/value pairs. Some sort of JSON handling would be nice, it is very popular and very useful, but JSON is a very broad spec, so it's not clear how best to make it useful within Keyboard Maestro.

5

Nor do I, however with the generous help of forum members (@ccstone in this case) and a fair bit of hard work myself, I was able to pair data from a csv file and use it in any number of ways. I've had the same success with getting data from google maps on the Alfred forum. People are very generous and helpful if you give them the right information to move forward one step at a time I find. You might be able to achieve your plans without learning JavaScript!

1 Like
6

Hey Matt,

If you want to parse data you're most likely going to have to learn a few new tricks.

If you're going to work with JSON then obviously you'll want to learn how to use native tools, because ultimately they'll be more flexible and easier to use.

However...

Basic data parsing such as your original post isn't all that hard with a little knowledge of the shell and/or some RegEx.

Here's a basic Keyboard Maestro macro that uses the coin name to return the ID for a given list of coin names from your listings URL. (You could use the symbol instead, but that would take a bit of recoding.)

Web -- Scraping Currency Data (Keyboard Maestro) v1.00.kmmacros (7.9 KB)

Here's a really basic shell script that returns the info-record for a given list of coin names in a pop-up window.

It's slightly more complex than necessary, because I put the currency name list at the top – where it's easy to modify.

Web -- Scraping Currency Data (shell) v1.00.kmmacros (5.2 KB)

This macro is less basic.

It takes a given list of coin symbols and returns the complete data for each of them from the ticker API.

The report is displayed in BBEdit if you have its command line tools installed, otherwise it's displayed in TextEdit.

Web -- Scraping Currency Data (shell) v2.00.kmmacros (7.2 KB)

I'm not super happy with this due to the verbosity of the script, but I was trying to stick with sed – and it has certain limitations. Even so the script runs only 21 lines, and it's fast.

I didn't feel like taking the time to make this one quite as user-friendly, so the coin symbols are “buried” in the code.

If I feel like it later I'll take one more pass at this in the way I'd normally do it for myself.

So – now you have a few examples.

Don't worry if it all looks like Greek. When I started doing this stuff I was pretty bug-eyed at all the weird syntax of the shell.

-Chris

3 Likes
7

jq is also a good tool, which can be used from KM Execute Shell Script actions.

https://stedolan.github.io/jq/

8

Hey Rob,

Cool.

MacPorts has it, so it's easy to install.

I'll give it a try.

-Chris

9

Chris, first of all, wow — thanks so much!

I’m going to need some time to digest all this, but even your first example highlights some things I wasn’t aware of:

  • A simple one, I don’t need curl. How had I missed the “Get URL” action?

  • It hadn’t occurred to me I could use a regex in a for each to actually define the list that is being operated on. I don’t yet fully grok what your regex is doing, but I know enough to figure it out, which I’m going to try to do now.

The shell stuff with sed just looks beyond me, but I do know a little python. Is there any reason to invest the time to learn shell programming for this kind of thing, rather than calling, say, a python script?

Thanks again so much!

1 Like
10

Hey Matt,

The Get URL action was new with Keyboard Maestro 8.0 and a welcome addition indeed.

Rather than explain it to you, I've put it up on regex101.com:

Take a look at it there, and dump the data into a BBEdit 12 window and:

  • Activate “Live Search”.
  • Turn on the [Grep] button.
  • Paste the pattern in the “Search” field.

Type ⌘G to “Find Next”.

You can construct and deconstruct patterns here and watch them match LIVE.

(If you don't already own BBEdit then use the demo. it will revert to BBEdit-Lite after 30 days but will still be muy potente.)

I did need to make a slight change in the regex pattern for it to work properly on regex101.

(?mi)^\h+"id"[^\n]+?\n\h+"name": "(Bitcoin|Philosopher Stones|TEKcoin)"

And this change works fine in the Keyboard Maestro macro as well.

RegEx101 can be a bit overwhelming to new users, but don't let that throw you – it's a great forensic tool.

If after fooling with the RegEx for a while you get stuck let me know, and I'll break it down for you.

In Keyboard Maestro this is how you manage multiple found sub-strings.

In Perl you'd just match into an array like so:

Web -- Scraping Currency Data (Perl) v1.00.kmmacros (5.2 KB)

Keyboard Maestro Export.png924×890 86 KB

Note – There are only 3 functional lines of Perl code in this script. The variable declaration is a formality, and I don't consider the call to curl to be Perl code.

-Chris

11

Hey Matt,

Actually I could probably teach you enough sed in 30 minutes for it to be useful to you – maybe less. Mastery of course takes a little longer… (I’m still working on that after a decade of use.)

Well – I'd say you can do some things in the shell that you can't do in as easily in Python – but with all the modules available for Python you can probably do most anything except application automation on macOS.

Python is on my radar as a language to learn, because it's used extensively for science and for web-scraping and data analysis.

-Chris

12

Could you do that with a shell script?

13

Hey Keith,

What specifically?

I've already shown how to do @dafacto's task a couple of different ways with the shell.

See post #6.

-Chris

14

Hi Chris,

I've now made use of the first, and simplest, approach you suggested — I have a Python script that does a lot of stuff, and returns a string like the following to Keyboard Maestro:

currency1,balance1,delta1|currency2,balance2,delta2|...

And then uses the "for each in a set" action to regex for substrings. That works great! Which makes me feel good, until I remember that I'm using the simplest of the examples you provided! :slight_smile:

Thanks so much again!

15

This sounds very interesting. Unfortunately, I cannot find an reference to "for each in a set" anywhere in the KM Editor or Wiki.

Could you please provide a real example of a KM Macro that uses your approach?

Thanks.

16

He presumably means the Substrings In collection with a regex like ([^\|]+), followed by a Search using Regular Expression action to split the entry into three pieces.

17

Hi @JMichaelTX —

Here's what it looks like:

LVl1tr+1660×1030

It's a fantastic feature!

2 Likes
18

OK, thanks. I'm very familiar with that technique and have used it often. Just never saw any reference to "for each in a set".

To other KM forum readers: Keyboard Maestro does NOT have an Action by the name of "for each in a set". This is just an informal name assigned by the macro developer when combining these two Actions.

19

Hey Matt,

Feeling good about making Progress is not just allowed – it's required!  :sunglasses:

-Chris

3 Likes