Hey Tony, thanks for sharing your experience. Having gone way down this road, I can confidently say that your preference mean zippo to the AI when you are working at momentum which you are in using it for Keyboard Maestro macros creation.
What works is to create "forcing functions" where what you want it to do is in the path it must take. Finding where that path is hasn't been easy and I'm continuing to discover further nuances of the process.
Here is the measurement that convinced me, from my own setup. I have two kinds of rule. Consult-me rules are the ones written down somewhere the AI is supposed to check -- a preferences file, a standing instruction, a "remember to verify" note. In-path gates are the ones that sit inside a step the work cannot go around. Over one working session I counted how each kind actually performed: the consult-me rules went 0 for 4. The in-path gates went about 6 for 6. Same rules, same AI, same day. The only variable was placement.
The 0-for-4 is worth spelling out, because every one of those four was a rule I had written carefully and the AI had read that morning:
- A tool index it was told to consult first -- unread, so it rebuilt a tool that already existed and installed a dependency against a gap that did not exist.
- A checker for UI instructions -- never run, so it wrote me three Finder walkthroughs naming a menu item that does not exist.
- A contacts lookup it was told to run before claiming anything about my address book -- unrun, so it told me a card was missing that was sitting right there.
- A standing format preference -- read at the top of the session, violated by the third paragraph.
None of that is the AI being stupid. It is momentum. Under momentum, a rule you have to remember is a rule you don't.
So what does a forcing function look like in practice? Four shapes, all of them boring, all of them things you can build in Keyboard Maestro.
1. Refuse the write, don't warn about it. My session log will not accept an entry without a status board attached, and the flag is not a checkbox -- the tool re-reads what was actually delivered and checks that two-thirds of my project names appear in it. A bare "yes I did that" is rejected. The KM equivalent: put the validation in a subroutine that cancels the macro when a required variable is empty, rather than displaying a warning nobody reads. Cancel This Macro is a forcing function. Display Text is a suggestion.
2. Make the bad artifact impossible to exist, not merely unsent. I used to ask for UI walkthroughs and check them. Now a walkthrough cannot be written as prose at all: it has to be a file with inline claims, run through a checker that greps the actual app source, and it only reaches me at exit 0 with the verification output stapled to it. The difference matters -- "remember to check it" is exactly the instruction that fails under momentum, so the check moved into the only path the output can take to reach me.
3. Positive controls, or silence means nothing. This is the one I would push hardest at this forum, because it is pure KM discipline and has nothing to do with AI. Every search my tools run has to first find something it knows is there. If the control fails, the tool raises instead of reporting. It earned its keep this morning: a contacts dump came back empty, the tool refused rather than telling me my address book had nothing in it, and the error file held one line -- Contacts got an error: Application isn't running. Without the control I would have been handed a confident, clean, completely wrong "no matches". In KM terms: before you trust a Search using Regular Expression that found nothing, make it find a known string first.
4. Assume the escape hatch will be used, because it will, by the AI, under momentum. I had a waiver comment for one of my gates -- a way to say "I know, ship it anyway". The gate then caught a false claim twice in one session, and both times the AI wrote itself a waiver and shipped. I did not remove the gate; I made that class of claim non-waivable and made the log name every line a waiver ever cleared. A gate is what it detects minus what can be talked past. Before you build an override, ask who is going to use it.
And one that speaks straight to your #3, where you said it "seems to be remembering." I had a formatting rule -- always show times in 12-hour, never 24. I wrote it into the standing instructions. It was violated three separate times. The reason is worth knowing: 17:45 already looks like a time, so nothing triggers a correction impulse on the writing side. The error is invisible from where it is made. It stopped for good the day I moved the rule out of prose and into the function that renders the times -- now there is no code path that can emit a 24-hour clock at all. The rule did not get better. It got relocated.
@Nige_S is right about the KM-specific gap, and I would add that it is the reason the forcing function has to be yours. There is a lot of shell and JavaScript in the training data and very little Keyboard Maestro, so the model reaches for Execute Shell Script where a native Action exists, and it will not know that your Local variable does not need initialising or that a compiled script inside a For Each recompiles every pass. You cannot prompt that away. But you can make it checkable: keep a short macro that opens a generated .kmmacros, counts Execute Shell Script and Execute AppleScript actions, and flags any script action nested inside a loop. Run it on every macro the model hands you, before you enable it. That is a forcing function, and it turns Nige's expertise into something that fires without either of us remembering it.
@ComplexPoint's objection deserves a straight answer rather than a dodge. He is right that iterating with the model can be time spent training it rather than yourself. My honest reading: the iterating is where the learning does not happen -- that is the part that feels productive and teaches nothing. The learning happens when you build the gate, because to build one you have to state exactly what "correct" means for your work, in a form a machine can check. I have learned more Keyboard Maestro from writing checks on generated macros than from reading the generated macros. The gate is the study.
Tony, the practical version for you, given you said you avoid bash and AppleScript: don't start by learning to read them. Start by writing down the last three things the model got wrong that you had to catch by hand, and then build the smallest possible thing that would have caught each one without you looking. Three small checks beat one big preference paragraph, every time. And I'd genuinely like to hear where this breaks for other people -- I'm still finding new nuances in where the path actually is.
