Structuring long, complex macros

[Joel_Rendall]

Hi everyone! I have been searching the forum and surprised I couldn’t find any existing topics about this…

Like many of us who try to get the most out of KM, I have some HUGE daisy-chained macros that consist of groups inside groups of grouped actions, plus use of external macros via “Execute Macro” action.

I end up getting pretty lost in the macro which can create that foggy feeling in my head when it comes time to revisit and adjust a macro.

Does anyone have any tips or tricks about how to organize macros, whether it be through the use of Groups, Comments, color coding etc?

Do you have any policies that you enforce to help your future self who revisits the macro later? For example, I decided that although we can rename individual actions, I try to do so sparingly so I don’t get confused about what type of action I’ve used.

Looking forward to any tips, as I’m about to embark on the process of rebuilding a massive macro with hundreds of actions and want to do it right this time.

Thanks in advance!

Joel

[Joel_Rendall]

It would be cool to come up with some kind of best practices guide for others struggling with this same problem… similar to how we all follow the same syntax for something like Markdown, a standard set of best practices (that most of the power users in here agree on) with regards to grouping and structuring massive macros would probably be pretty useful.

[Mark]

@Joel_Rendall

some HUGE daisy-chained macros that consist of groups inside groups of grouped actions, plus use of external macros via "Execute Macro" action.

.

Joel -

Thank you for posting that.
You've brought up two important topics which interest me, too:

1. structuring complex macros
2. guidelines of best practices

You are already using "Execute Macro" action.
Would it alleviate the problem of long and complex if you used "Execute Macro" even more?
Maybe a lot more.

One practice I use is to limit each macro (subroutine) to one page (of paper or screen).
When any macro gets larger than that, I split it into multiple subroutines, using more "Execute Macro" actions.
If it won't split easily, then I reach for pencil and paper and sketch flow charts with boxes and arrows.

Doing that keeps "If" actions neat and tidy.
Would that help with your macros?
Or is there more to this problem than I realize?

Do you have any training in computer programming?
Perhaps you'd find value in books or web sites that explain best practices that have been refined by programmers over decades.
Could avoid "that foggy feeling" in the head.

(I hope I'm not "preaching to the choir".
For all I know you could be a system engineer for Goodle, or something!)

---

.

@Joel_Rendall

Do you have any policies that you enforce to help your future self who revisits the macro later? For example,

"Your future self" and mine, too, depends on lots of comments.
I expect you already know that, so this is a just a gentle reminder.

---

.

@Joel_Rendall

best practices guide

You won't get much of that on any technical forum.
Posters on this forum are wonderfully helpful, but on any forum the discussions tend to devolve to nitty-gritty details, rather than overall perspective.

For best practices, look for books or web sites devoted to that topic.
Some others here are better able to suggest those than I am.

If you decide to study any of those, I hope you will consider "curating" and reporting back here with the best you've found.
And then describe how you apply those practices to your KM development.
I expect you'll get plenty of interest and lots of critique, including from me.

[JMichaelTX]

@Joel_Rendall

Like many of us who try to get the most out of KM, I have some HUGE daisy-chained macros that consist of groups inside groups of grouped actions, plus use of external macros via "Execute Macro" action.

This is hard to address without specifics.
Care to post one of your "HUGE daisy-chained macros" that we can use as an example?

Dealing with complex processing, that might entail many steps, is really no different in Keyboard Maestro than any other "language". OK, KM does not offer some of the tools, but the basics are the same.

As @Mark mentioned, the key is to write short, focused "subroutines" (we call them "sub-macros" here) that can be written and tested independently. Then call these sub-macros from a main program (main macro), that exposes only the top-level logic.

Long ago, we had a guideline that if a routine was longer than about 100 lines of actual code (excluding comments), then it was a strong candidate for breaking up into subroutines, even if you only called a subroutine once.
The "100 lines" is not a hard and fast rule, but a guideline to make you think.

The KM "Group" Action presents an alternative to sub-macros. Since you can collapse the Group to one small "line", it has the same effect. However, the down side is you can't to unit testing with Groups (not without a lot of work).

Again like @Mark said, try to keep each macro (sub-macro) to about one virtual page (or so).

Frankly, I have very few workflows that require really long macros. I am curious as to what workflows you have that do require long macros.

The most complicated, potentially very long macro, that I'm working on is a macro that will do an intelligent web capture of code blocks from almost any source, and then provide multiple output destinations. Each source, and each destination, can require extensive customization. So this immediately is a strong candidate for sub-macros, one sub-macro for each source, and one for each destination.

This approach also facilitates adding more sources and destinations, by either myself, or by others who download the master macro.

I hope Macro Master @DanThomas will jump in here. He has vast experience in the software development and project management world. I would expect he has some "best practices". Maybe he can share a few.

Software development is best done as a journey rather than a destination. If you are curious and keep an open mind, there is much to learn over a lifetime. Good luck!

[Joel_Rendall]

Hey guys! Appreciate the words of wisdom, I'll be rereading them again now. But since you asked, first I wanted to just share a glance at the project I'm working on to show you how nuts things are getting.

I don't suggest you waste too much brainpower looking closely at the macro, it's just to give you a quick glance into the self-inflicted craziness I'm dealing with! If you don't have the mental energy to dissect all of this, no sweat. Just wanted to also share with others what kind of complexity is possible with some severe nerding out with KM.

It's kind of one of those things were my use case is so extreme that I almost need to start writing custom software.. but since my coding skills are lame and mostly limited to basic PHP / Javascript, I've been working with what I know how to use (KM).

So, the screenshots - the macro screenshot is of the main Macro that has tons of nested groups, Execute Macro actions, If loops etc. (It was too big to upload so I reduced it... not sure if the forum reduces it further or if you'll be able to read it or not..)

The other screenshot is a folder of macros that mostly support that one main macro. You can see my attempt at a naming convention. "z_" shoots the items to the bottom of the list since they are not usually triggered individually but rather via another macro. The ones with "_" prefix are the main macros that I do trigger manually.

Also including some screenshots of the spreadsheet from which the content originates, and screenshots of the finished product just to give you an idea of what is accomplished.

I'll briefly explain the goal behind all the craziness. This is for a bootstrap language learning platform (for European Portuguese learners). The site runs off a wordpress quiz plugin.

Here is the gist of what these macros accomplish. Again, don't get too lost in the details, this is just a rough overview:

• Captures a spreadsheet (will multiple sheets) from excel, saves everything into a KM variable

• Using regex and For loops, it processes each set of lines that pertain to lessons/quizzes for our site. Each line generates 5 or 6 different kinds of questions using the same set of translation pairs, eg. Drag to Sort, Multiple Choice, Fill in the blank. Some questions prompt the user to translate from English to Portuguese, and others the opposite.

• Each Portuguese phrase within each line renames a jpg file as needed from a bunch of images have in a folder, renames them accordingly. It also generates small mp3 files using speech synthesis with the help of Automator and a terminal command.

• If the macro determines that a Lesson or a Unit is new and hasn't yet been created in Wordpress, it creates the new post in Wordpress. It also creates a new Quiz (a custom post type that the plugin uses) and saves its quiz ID for later to associate the questions with the correct Quiz it belongs to.

• The macro uses sub macros "Q1, Q2" etc to generate each type of question, writing them to a .csv file.

• Once everything is done, I upload the folder of mp3 files, jpgs, then through phpmyadmin, I import the CSV files directly into the mysql database. (I used to have KM creating every individual question through Safari in Wordpress but with thousands of questions, this would take hours each time I wanted to update the content! Direct mysql import is much faster).

So as you can see, I have ducttaped together a very complex, daisychained set of macros. If I knew what I was doing, one day it would be amazing to make it so that these spreadsheets could just live inside Wordpress and have all the individual questions and content be generated dynamically instead of hardcoding everything into a file that's imported into mysql. But for now everything is working and we have happy customers making use of the quizzes.

I wouldn't have been able to dream of putting together something this complex if it weren't for KM. According to the About window, it would have taken me 2190 Years to generate all this content manually

[Joel_Rendall]

Mark, good point about using flowcharts etc. I have tried to do that and need to improve that skill for sure. I find that with such a complex set of macros, each time I need to make a change, I make it in a spot that works but isn’t the most organized, so I kind of need to go through everything and start from scratch to group together actions that are related, add more comments etc.

I could probably use Execute Macro a bit more, and like your guys’ advice of limiting groups to about one screen before splitting them off. I find that while I do use Execute Macro, they’re often nested 4 or 5 layers deep inside subgroups.

I started these macros about 2 years ago (and rebuilt them once, I think). But in the past 6 months my knowledge of Regex has improved so can likely go back and eliminate many Search / Replace commands. With the help of this forum (It may have even been JMichael who replied to a prev post of mine actually), I may also be able to make use of passing variables to the Execute Macro command to make them more dynamic.

Once I decide to redo everything I’m probably faced with a full 40 hour week of redoing everything. I’m trying to figure out if I can improve things enough for there to be a big enough pay off going forward, or if I should just live with what I’ve got and start trying to program some of this stuff dynamically using my shoddy PHP skills right onto the Wordpress site. Would be a good learning process for sure. (As a bootstrap co-founder, I have to optimize and balance my time through this, planning/recording/editing podcast episodes, member support etc. All good problems to have I guess, but I really have to limit my nerding out hours to get the biggest payoff going forward)

To answer the question about if I have computer programming training - not really. I’m a self-taught Wordpress designer (mostly front end) but can write PHP and Javascript functions to tie stuff together when needed. I can also have made very basic Wordpress plugins that even sometimes work! I have the logic of a programmer from basic high school courses and experience with putting Wordpress solutions together. But if I had to program something to replace what this KM macro madness accomplishes, I’d be looking at 2-6 months of a learning curve with uncertain results.

I do like the idea of thinking of submacros as subroutines (or functions in programming). I could probably split all of these into smaller macros, but would need to work on a clearer naming convention for each individual macro to help me know which macro triggers which when looking at them since I don’t think it will be possible to avoid still going about 4 or 5 layers deep into subsubsubsubsmacro triggering. Hope that makes some sense.

In these two massive followup posts, I haven’t really asked any clear questions. Just wanted to further the conversation for those with the patience and interest to dive in further to this first world conundrum! If any of this has triggered further ideas for how to better structure this kind of daisychainedness, I’ll be refreshing the page anxiously over the next couple days in anticipation while trying to decide the next steps towards improving the situation

Here are some more practical questions that anyone reading might have advice on:

• What is your policy for renaming actions or loops? Mine is that if I need more than 2-3 words to explain what’s going on, I’ll use a comment or Group to label it. I believe that when you rename an action it may break some automatic label updating that happens with the Set Variable As or Search/Replace commands. And I also feel like it just gets messy and could make it harder to figure out at a glance what kind of action has been used. I’ll also sometimes use keywords like “TEST5” when I’m working on a section of a macro so I can use the search to find that part of the macro again later.

• What about colour coding? I have started labelling groups or actions Green when they should be manually modified before running a macro (for example, setting pasting new content into a “Set Variable” command instead of bothering to use the “Prompt for Input” dialogue). I have also used Red for actions I MUST modify before running a macro. But overall I need to better define how these are used so I can confidently use them more often for these and other uses.

Thanks again and hope no one’s brain explodes from all of this!

Joel

[Mark]

@Joel_Rendall -

1. “Reset variables” at the beginning is good practice.
   Better practice: make it a sub-macro.

2. “Repeat Actions” is another good candidate to be a sub-macro.
   Make it better by passing a variable for the number of times to repeat the loop.

3. Lots of “Search and Replace” actions.
   Instead, have one sub-macro and pass a variable for what to replace.

4. On a flow chart, the code above would look like a long and winding road; difficult to maintain.
   Better practice: use a tree structure with lots of thin branches.
   Or hub-and-spoke structures with small, controlling macros at the center.

5. To improve macro names change to noun first, verb after.
   Instead of “Fix Quiz Options for Core Quizzes” …
   make it, “Quiz Options: Fix Core Quizzes”.
   That way every macro acting on “Quiz Options” will appear together in the macro list.
   For naming macros, nouns (targets of action) are almost always more important than verbs (what the action does).

6. Old programmers’ “Rule of Thumb” to make future maintenance easier:
   One action : One comment

Being willing to expose your work and to learn, on a public forum like this, takes courage.
My compliments on yours.

[Joel_Rendall]

Excellent guidelines here, Mark. I’ve already started rebuilding one of my main macros (similar in function to this one but for a different aspect of the project) and am trying to keep as many of these guidelines in mind. Thanks a million for the patience to read through everything and offer this advice!

[Joel_Rendall]

@Mark

To improve macro names change to noun first, verb after.Instead of "Fix Quiz Options for Core Quizzes" ...make it, "Quiz Options: Fix Core Quizzes".That way every macro acting on "Quiz Options" will appear together in the macro list.For naming macros, nouns (targets of action) are almost always more important than verbs (what the action does).

I have already been following this Noun+Verb advice but using it when naming Groups within a macro. Although the idea felt strange, I already feel like things are already more scannable... didn't expect that!

[jayknowstheway]

My problem is having too many groups. Example: Each of the keyboard keys in the block Q-F has 8 alternative functions. I needed 8 different Group folders, and then a Group to activate one, deactivate all others (x8). See the nested folders discussion here: Feature Request: Folders Within Folders (Or, Groups Within Groups)

For my configuration, however, there's a much more efficient solution: using variables. https://wiki.keyboardmaestro.com/manual/Variables

I can then switch groups and change my numeric parameters over large group of macros without having to make more folders. This helps reduce the number of groups and make the code more modular.