Execute Shell Script not triggering "Display Results" window or running Python script

Hi everyone, I’m working on a macro to chunk PDFs using a Python script, but I’m hitting a wall. When I run the macro, it prompts me for input, I click "OK," and then... nothing. No results window appears, and no files are created. KM should be telling me something about my mistakes, but that window is also not appearing.

Here is a human explanation of what I'm doing. I'm doing something that's way over my head. But, it works sometimes. I want to put reference books into Devonthink. If I can find specific chapters, I will be much closer to the information that I'm looking for. So I'm trying to chunk these books into chapters. I have many of these reference books, so I'm trying to make a generic chunker that will chunk many different books using a table of contents. It works enough of the time for me to keep trying to do this. I can do it for one book, but then I can't do the same thing for the next book. That's where I started making this generic chunker with the help of Gemini. I'm sure this would be easy for a human to do, but I couldn't pay a human as much as it would cost to do this. I'm just guessing.

I have tried doing this with the terminal alone, but every time there is some problem with the Macintosh terminal. A different version of Python is needed and the versions are somehow extremely complicated. That's when I started trying to do it with Keyboard Maestro. I'm learning a lot as I'm doing it, but I'm learning a lot about the weakness of AI.

If you could tell me where I'm making some mistake, I will tell Gemini and somehow make a change.

The Setup:

  1. The Goal: Pass two local variables (Local_CurrentLine and Local_EndPage) from a KM Prompt to a Python script located in a dynamic subfolder.

02 Chunker_pdf for Single book.kmmacros (10.1 KB)

TOC example

[[Acari (Mites & Ticks)]] ((1))
[[Crustaceans]] ((89))
[[Insects]] ((181))
[[Myriapods (Centipedes & Millipedes)]] ((837))
[[Scorpions]] ((857))

My Python script looks like this:

import os
import sys
import datetime
import re

# --- 1. LIBRARY CHECK ---
try:
    from pypdf import PdfReader, PdfWriter
except ImportError:
    print("\n❌ CRITICAL ERROR: 'pypdf' library not found.")
    print("Please run: /opt/homebrew/bin/python3 -m pip install pypdf")
    sys.exit(1)

# --- 2. ENVIRONMENT LOCK ---
SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__))
os.chdir(SCRIPT_DIR)
BOOK_IDENTITY = os.path.basename(SCRIPT_DIR) 
OUTPUT_FOLDER = "CHUNKS_FINISHED"

def detect_pdf():
    """Finds the main book PDF."""
    all_files = os.listdir('.')
    pdfs = [f for f in all_files if f.lower().endswith('.pdf') 
            and not f.startswith('._') 
            and f != 'rawTOC.pdf']
    if not pdfs:
        print(f"❌ ERROR: Main PDF not found in: {BOOK_IDENTITY}")
        sys.exit(1)
    return pdfs[0]

def run_chunking():
    source_pdf = detect_pdf()

    # Handshake with Keyboard Maestro
    if len(sys.argv) < 3:
        print("❌ ERROR: Keyboard Maestro failed to send data.")
        sys.exit(1)

    raw_line = sys.argv[1] 
    
    try:
        # Pulls text out of [[ ]] and page out of (( ))
        # This disregards the brackets and cleans the name
        clean_chapter = re.search(r'\[\[(.*?)\]\]', raw_line).group(1).strip()
        start_page = int(re.search(r'\(\((.*?)\)\)', raw_line).group(1)) - 1
        end_page = int(sys.argv[2])
    except Exception:
        print(f"❌ ERROR: Could not read Wiki-Line: {raw_line}")
        sys.exit(1)
    
    # FORMULA: ChapterName_BookFolderName.pdf
    full_name = f"{clean_chapter}_{BOOK_IDENTITY}"
    clean_filename = re.sub(r'[\\/*?:"<>|]', "", full_name).replace(" ", "_") + ".pdf"
    
    # Automatic Folder Creation
    if not os.path.exists(OUTPUT_FOLDER):
        os.makedirs(OUTPUT_FOLDER)

    print(f"🛡️  AUDIT: Processing {clean_chapter}...")

    try:
        reader = PdfReader(source_pdf)
        writer = PdfWriter()
        actual_page_count = len(reader.pages)

        # Safety check for page numbers
        if start_page < 0: start_page = 0
        if end_page > actual_page_count: end_page = actual_page_count

        for page_num in range(start_page, end_page):
            writer.add_page(reader.pages[page_num])
        
        output_path = os.path.join(OUTPUT_FOLDER, clean_filename)
        with open(output_path, "wb") as output_pdf:
            writer.write(output_pdf)
        
        print(f"🏁 SUCCESS: Saved {clean_filename} to {OUTPUT_FOLDER}\n")
        
    except Exception as e:
        print(f"❌ PDF ERROR: {e}")

if __name__ == "__main__":
    run_chunking()

Please correct block of code, part is displayed as message, part as embedded code, then will be easier to analyze.

The "Shell" Action is set to not include errors:

...so try changing that.

Which suggests you've installed python3 properly, the python script works, but KM isn't passing in the correct variables.

Your first "Comment" Action implies that each book will get its own folder inside TOC_container but you set Local_FilePath to point to a chunker script in a particular folder of TOC_container. You later "Pause" until that same folder exists -- if the ARTHROPOS_VERMEULEN folder isn't there your macro won't go any further.

You call your python script with:

/opt/homebrew/bin/python3 chunker.py "$KMVAR_Local_CurrentLine" "$KMVAR_Local_EndPage" 2>&1

But you don't set the variable Local_CurrentLine in your macro. Should it be Local_StartPage instead? (I don't know enough python to work out what your script will do if raw_line is an empty string!).

I used </> to imbed the python script. I hope it helps. Thank you. I have never done this before.

For code on the Forum you can wrap in single backticks to inline it. So if you type:

My macro outputs `Hello Fool` but I want `Hello World` -- what did I do wrong?

...you get:

My macro outputs Hello Fool but I want Hello World -- what did I do wrong?

For code blocks use at least three backticks on their own line to start the block, and the same again to stop:

```
if x > 0 then
   set x to x + 1
else
   set x to 1
end if
```

...gives:

if x > 0 then
   set x to x + 1
else
   set x to 1
end if

(And how did I get the backticks to show in the above? By starting and finishing the code block with four ticks!)

You can even include the language to get some syntax highlighting:

```python
import os
import sys
import datetime
import re

# --- 1. LIBRARY CHECK ---
try:
    from pypdf import PdfReader, PdfWriter
except ImportError:
    print("\n❌ CRITICAL ERROR: 'pypdf' library not found.")
```

...gives:

import os
import sys
import datetime
import re

# --- 1. LIBRARY CHECK ---
try:
    from pypdf import PdfReader, PdfWriter
except ImportError:
    print("\n❌ CRITICAL ERROR: 'pypdf' library not found.")

All this, and much much more, can be found in @_jims's most excellent Entering and Enhancing Forum Posts post.

Thanks @Nige_S. I made your changes. I am going through every line in the python script and the shell script. We are probably still missing something, but I forced AI to be systematic and explain every line to me. I will be back in a few days if it does not work out. AI does not really know the KM, but its guess is usually better than mine. Knowing is so much better than guessing.

And @nutilius's knowledge of Python and executing Python scripts from KM will be best of all! Now your code is nicely formatted, hopefully they'll be back.

In the meantime, first check that your Python script is being run. A quick way to do that is to add a print command after the import statements and comment out everything below that with triple quotes (single or double, as long as they match:

import os
import sys
import datetime
import re

print("Hello World!")

'''
# --- 1. LIBRARY CHECK ---
try:
<snip>

if __name__ == "__main__":
    run_chunking()
'''

Did you receive any message after enabling flag “Include errors” as @Nige_S suggested?

It’s hard to understand (for me) the general concept of solution.

As I understand the script receive two argument in form:

  • [[Acari (Mites & Ticks)]] ((1)
  • page-num

In script I see that arguments split to:

  • clean_chapter: Acari (Mites & Ticks)
  • start_page: 1
  • end_page: page-num

It setup BOOK_IDENTITY to folder where script exists (_file_) and later in that folder get FIRST (returned by listdir() function) PDF, select pages from start_page to end_page and save it in file Acari_(Mites_&_Ticks)_KM01.pdf

But it always returns FIRST element of listdir() - see detect_pdf() function.

In my environment script works in terminal. What kind of error do yo receive in terminal?

Should that script select file defined by 1st argument?

What is the structure of subdirectories:

You first setup %Local_FilePath to place where python scripts is:

~/Downloads/workspace_chunk/TOC_container/ARTHROPOS_VERMEULEN/chunker.py

but later setup BASE_DIR to:

~/Downloads/workspace_chunk/TOC_container

and list directories from

`~/Downloads/workspace_chunk/TOC_container/*/` -setup as BOOK_FOLDER

and NEVER use this variables later!


That why I mostly not resolve problems created by AI.

When real person define the problem, I could ask him about details. When someone ask AI (mostly many times) I don’t see their prompts and don’t know what idea say behind the AI solution. Now people asking for help have to provide their idea, history of prompts sent to AI and finally generated code - it is … Ugh.

The main reason why your Macro doesn't work is “embedded spaces in first argument”.

The spaces (and other not friendly characters for shell like [[ and (( ) may disrupt interpretation of script arguments

Change line with pyto to 2 lines:

ARG1=$(echo "$KMVAR_Local_CurrentLine" | sed -e 's/ /\ /g')
python3 chunker.py "$ARG1" "$KMVAR_Local_EndPage" 2>&1 

Now python script will see 2 arguments, not 4.

Doesn't double-quoting the variable when calling the script take care of that?

And isn't Local_CurrentLine just a number anyway?

Although, as mentioned above, that variable isn't set in the macro so will actually be an empty string. And I can't see any place where the TOC is actually read in to then be processed. But that may be my lack of Python showing!

yes - it should - and now it works - I don't know how I’ve been testing before - shame.

Anyway after changing Local_StartPage to Local_CurrentLine as below (what you suggested before) script works (now also with double quotes :slight_smile: )

The safe issuing curly braces around KM_Variables (as in your example) but without that is also OK.

But I changed configuration - in my case I’ve created python env, because I don’t want to install pypdf globally and I run script with absolute path.

What I see now, there is strange cd command in if statement - because it change directory to TARGET_PATH what is result of strange lscommand. There is probably wrong place because in the beginning the clunker.py is in /Users/ellenmadono/Downloads/workspace_chunk/TOC_container/ARTHROPOS_VERMEULEN/chunker.py and cd should go to the directory /Users/ellenmadono/Downloads/workspace_chunk/TOC_container/ARTHROPOS_VERMEULEN/ where real script exists.

I still don’t understand structure of working directories - where is the chunker.py, where are the books, where we are doing work.

I think, from the script, that each book has its own directory and each directory has its own copy of chunker.py. So there's a folder structure like:

(Unnecessary folders like "Documents" and so on omitted.)

So all resources are in the same directory as the copy of the script being run, hence the cd, the "strange ls", and so on.

Not the way you or I would do it, but I can see how you'd get there via AI prompts that started as a one-off "process this file". And it does save having to pass in a path as a script argument :wink:

The shell script's TARGET_PATH variable is then "the most-recently updated directory in "TOC_container" (the ls) and the script cds to that to run the local chunker.py -- that should be more explicitly called via

/opt/homebrew/bin/python3 ./chunker.py "$KMVAR_Local_CurrentLine" "$KMVAR_Local_EndPage" 2>&1

(Note the ./ before chunker.py.)

If it is true (it seems to be), that is the strangest thing I’ve seen in last few years. I will never do like that. The redundancy of code (bunker.py in each folder), strange TOC files (not used in script). I understand that ls -td, but that way - …

It is easier to rewrite this script than determine what is wrong in @Ellenm environment, but first your suspicions about file structure should be confirmed and additional elements (f.ex.format of first argument) should be explained.

I’m afraid that next time I will say “pass” when I see that someone ask about help In corrections of AI code. I’m happy to help, but such cases are much more problematic to resolve them. What is disturbing, that in next months we will see much more such “monsters” around. I hope the not in f. ex. health equipment.

Yes, this “monster” comes from a novice who is using AI. I can see where this kind of problem is a nuisance. But if you look at the situation from a different point of view. I'm the kind of person who gets very little done on KM. I'm encouraged to try again because of AI and a few successes using AI. Without AI, most of my efforts over many years have come to nothing. I will try to understand the first argument as you say, but later.

It would be possible to do multiple books I'm told, but I want to try to get one book done. Book 1 (folder) becomes the name of the chunk that is produced by the macro. Ideally, if I can get this set up, the chunker.PY is the same for every book. That's possible for very simple books.

Book_1.PDF is the actual book that we are chunking. But the title for the book comes from the folder that contains it. The table of contents (TOC) has been set up to give us a title for each file that identifies that particular Book which is a PDF file located inside the folder book 1. All of the files have the title that comes from the folder containing the PDF of the book. Here the example would read: TOCchapterName_Author_BookTitle. This is the resulting title of the file.

The files need to be chunked is indicated by the page number of the book. But many books have front matter that is counted in Roman numerals. So the PY is using an arithmetic formula to subtract the front matter from the page number that appears in a PDF reader, such as PDF Expert. Both the page number from the TOC and the page number from PDF Expert are needed to find the place to chunk.

Neither I nor AI understands Keyboard Meister very well. If I'm not asking the right questions, then AI just gets lost. So I'm going to do another project that is simpler but similar in format. Then I'll come back to this and maybe I'll understand a little more.

This isn't really an AI thing -- you'd have similar if you'd approached your local friendly Python programmer with a plan, had a few rounds of back-and-forth with them to get something working, and then tried to integrate that with KM without their help (because they've never heard of KM).

It's no longer your plan that's being followed because they've made changes. They're not around to explain to us the changes they made, any assumptions they've built in, and the strange decisions they've made (on top of other strange decisions!) to make it all work.

It's just that you and AI can do in 10 minutes what used to take 10 days of iteration...

So we need to go back to the code and break down every line to find the plan it's following, compare that to the plan you actually want, and resolve the differences.

While that's going on -- did you try the "comment it all out", as described here, trick to check that your Python script is actually being run?

OK -- so here's a problem. This bit of python:

raw_line = sys.argv[1] 
    
    try:
        # Pulls text out of [[ ]] and page out of (( ))
        # This disregards the brackets and cleans the name
        clean_chapter = re.search(r'\[\[(.*?)\]\]', raw_line).group(1).strip()
        start_page = int(re.search(r'\(\((.*?)\)\)', raw_line).group(1)) - 1
        end_page = int(sys.argv[2])
    except Exception:
        print(f"❌ ERROR: Could not read Wiki-Line: {raw_line}")
        sys.exit(1)

...is extracting chapter name and start page from your TOC text.

But in the shell script line that calls your python script:

/opt/homebrew/bin/python3 chunker.py "$KMVAR_Local_CurrentLine" "$KMVAR_Local_EndPage" 2>&1

...the first argument is a KM variable that you never set. What it should be is a line of text from your TOC file.

As written, your macro appears to be missing a "For Each" Action that should enclose your shell script:

It's not the most efficient way to do things since you're starting a new shell process for every line in the TOC, doing the same things again and again. But it's not much expense compared to the actual reading/writing of the PDF files, and a KM "For Each" is a lot easier than trying to do the same in either bash or python, so I wouldn't worry about it.

Which makes your initial macro dialog problematic. You use Local_EndPage, with python getting that via sys.argv[2], but you never use Local_StartPage -- which, if I'm understanding things correctly, should be your page number offset to account for any front matter.

And:

...is missing from your python script anyway. But that's something to worry about once you get the macro to produce a PDF for every chapter listed in your TOC.

Have you tried doing this the other way round -- prompting the AI with:

Explain the following python 3 code, line by line:

# code begins

<paste your python code here>

#code ends

I wish I'd tried that before my last post! But Copilot does confirm what I'd arrived at on my own:

I got a macro that worked. Thank you. I have to work on it more, but the hard part works.

2aaa patient invoice 2 .kmmacros (5.1 KB)

1 Like