Macro to create a PDF doc from a Markdown file using Pandoc

Macro Library
1

UPDATE: The issue with the original macro was solved by post #12.

Here is the final working macro:

20180607-165052

Create PDF with Pandoc.kmmacros (4.6 KB)

Dependencies

  • Pandoc
  • A Tex typesetting engine (Latex, ConText, etc) since the PDF creation chain is Md → Tex → Pdf

Note:
For the macro to work, the shell script shall be able to "find" both Pandoc and typesetting engine executables. You can set KM’s ENV_PATH variable so that it includes the right locations, e.g.

/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/Library/TeX/texbin

===============================================================
ORIGINAL POST: I built a really simple macro that run a pandoc command to create a pdf file from a markdown text file.

20180528-105957

/usr/local/bin/pandoc -r markdown+smart -w latex --top-level-division=chapter --pdf-engine=xelatex --number-sections --template=/Users/prem/Dropbox/Pandoc/templates/latex_PREM_1.template --filter /usr/local/bin/pandoc-crossref -M chapters --filter /usr/local/bin/pandoc-citeproc --csl=/Users/prem/Dropbox/Pandoc/csl/apsa.csl --bibliography=/Users/prem/Documents/bibs/PREM_Bibliography.bib -o "$KMVAR_TheOutPath" "$KMVAR_aPath"

It consists essentially in an Execute Shell Script action (the second one, disabled, is just a variation with different arguments).
The command works fine in the Terminal but gives an error when used in the macro; it seems it can't create the temporary folder needed for Tex files used in the conversion md→pdf:

40

That is probably due to the different environment in which the command is executed but I am at a loss how to solve the problem.

Any help will be much appreciated!

2

Hey @johseb,

You need to provide a working markdown file to test with.

-Chris

3

Hey @johseb,

Please also provide a macro file to go along with your macro image above.

The easier you make it to test, the more likely you'll get help.

-Chris

4

Hi Chris, thanks for your reply.
I didn't include the macro because I thought the problem was a trivial error on my part that could be solved without running the macro.
Anyway, here is a copy of the macro (on importing you will find it in the Finder test group)
image

Create PDF with Pandoc.kmmacros (4.9 KB)

please notice that I replaced the shell command with a stripped down one (a minimum working example that avoids the need of a few Pandoc filters); the command still shows the issue described in the first post and gives the same error (while running successfully in the Terminal).

The problem occurs with every markdown file so I'm simply attaching here a dummy test file

Test_file.md.zip (781 Bytes)

Other info that may be useful:

  • I'm running OS X 10.11.6
  • Pandoc (ver 2.2.1) was installed using Homebrew which puts a link in /usr/local/bin
  • Latex was installed via MacTex (http://www.tug.org/mactex/). I'm still on TexLive 2017.
  • The macro runs successfully if you generate a Latex output (modifying the output format to .tex instead of .pdf in the action that creates TheOutPath variable). As shown by the notification the error seems to happen in the conversion tex→pdf
  • When running the command in the Terminal the temporary folder needed for the conversion is created on the Desktop; I can see the folder appearing briefly on the Desktop and then being replaced by the final .pdf output file.

Thanks again for your help.

5

Hey @johseb,

When playing with paths, variables, and the shell a trivial error can be really hard to spot – especially when just eyeballing the problem.

Proper debugging nearly always requires active testing.  :sunglasses:

You generally won't get much help, unless you provide helpers with the tools to easily test – so keep that in mind for future issues.

I should have time to look at this in the next 2-3 hours, but it might be tomorrow if things around here get complicated.

--
Take Care,
Chris

6

The key to troubleshooting, is to break the problem/solution down into parts.
To eliminate any issues with KM Shell Scripts vs Terminal, do all of your testing in KM.

You have a very complicated shell script. I'd start with replacing all of the KM variables with explicit data. If you still have issues, then separate each part of the script into separate parts, separate scripts.

Finally, there are many KM forum posts about using pandoc and using shell scripts. Do some forum searches on each and examine solutions there.

Good luck, and let us know what you find with your testing.

7

Hey @JMichaelTX,
you're right, sorry for not explaining the problem thoroughly from the beginning.

That said, the downloadable macro in post #4 contains a much reduced script, just input/output formats and file paths.
As I wrote, it still generates the same error and it does so even substituting the KM vars with explicit paths.

On the other hand the macro works fine when generating .tex files instead of .pdf. The problem seems to be the creation of the temporary folder during md→pdf conversion.
I was't able to find in the forum a thread mentioning that problem so I started mine to discover if that's a generalized issue or something specific to my system. ​

8

Hey @johseb,

Ah, the tribulations of assuming...   :sunglasses:

Unless someone is familiar with scripting Pandoc and aware of how it interacts with MacTex this problem is pretty opaque.

That helps.

Installing 2.3 GBs worth of MacTex is a PITA, but without doing so you don't know for certain that MacTex changes the PATH environment variable and how it does so.

It took me a bit of research to find out where they did it.

/private/etc/paths.d/TeX

Your problem boils down to MacTex NOT being in the PATH Keyboard Maestro uses.

Once I fixed the path your macro worked fine.

See this ⇢ Path in Shell Scripts

Keyboard Maestro by design does NOT pick up changes made to the System's Shell Environment automatically.

The user has to provide them to Keyboard Maestro as necessary.

Run this:

echo $PATH | pbcopy

Paste the result into BBEdit or another good text editor.

Then run it again – but this time in a Keyboard Maestro Execute a Shell Script action.

Paste that result into your text editor below the previous path.

Compare the two.

You'll see a big difference.

You can set up an environment variable in the Keyboard Maestro Editor's variables panel with this name:

ENV_PATH

And place the entirety of your system path above into it.

This will make that full path available to ANY shell script you run in Keyboard Maestro.

OR you can do ONLY as needed in EACH shell script action using the export command:

export PATH=/Your/Path/String

Personally I set the ENV_PATH variable, so I don't have to mess around.

Let me know if you have any questions about this or any other problems.

NOTE – Once I had all the requisite pieces it only took me about 5 minutes to test and debug this problem. On the other hand getting set up to test took about a half hour and would have been much more of a headache had you not provided everything you did in post #4.

-Chris

9

Hi Chris, thanks for going to the trouble of installing all the stuff needed to test my macro.
Unfortunately I have the KM variable ENV_PATH set according to shell $PATH (I should have added this piece of info to the list in post #4) so that doesn't seem to be the issue.

I think the conversion performed by Latex executable(s) doesn't even start since Pandoc is not able to create the temporary folder on the Desktop.
By the way, do you see that folder created and then deleted on your desktop during the conversion?
Can you think of any reasons that explains the permission problem in creating a folder on the Desktop?

Addendum: all the executables mentioned in this thread was installed on my machine a few months ago and in the meantime I had the brilliant idea of changing the name of my user account. The procedure caused a long list of small issues so I'm adding the info here in case it helps.

10

Hey @johseb,

What does your ENV_PATH look like?

No. There's a moderately long pause, but I don't see the folder.

Not offhand.

What happens when you run this from the Terminal?

mkdir ~/Desktop/aaa_My_Test_Folder

Hmm...

Have you tried reinstalling Homebrew and/or Pandoc since changing your user name?

What about MacTex?

-Chris

11

To narrow down the issue a bit:

I’ve tried it with ConTeXt (-w context, instead of latex) and I can reproduce the issue perfectly:

It works fine in the Terminal, but not from KM, same error as you.
Adding the context path to ENV_PATH doesn’t change anything.

So, it doesn’t seem to be a MacTeX problem (my ConTeXt is a standalone install), and not related to the changes of your user account.

12

I think I have found the problem.

When I run the script from the Terminal, the default working directory ($PWD) is

echo $PWD
/Users/tom

that is, my $HOME directory. That’s where Pandoc is creating the intermediate folder, e.g. ~/tex2pdf.-ed2cbfc44a61f992.

However, when running a script from KM (Execute Shell Script action) the default working directory is

echo $PWD
/

that is, the root folder of the Mac.
Obviously Pandoc doesn’t have the permissions to create a folder there, thus it fails.

So, the only thing that’s needed, is to change the PWD. You can do this for example by prepending a

cd ~

to your script (or just cd). This gives it the same working directory as with the Terminal.

The script then should look like this:

cd ~ && /usr/local/bin/pandoc -r markdown+smart -w latex -o "$KMVAR_TheOutPath" "$KMVAR_aPath"

I can also use any other directory — as long as it is writable. For example the directory of the Markdown file: cd "$KMVAR_Local__Parent"

For me (with ConTeXt as PDF engine) this works. Give it a try.

13

Yes, changing the working directory did the trick!

Tom & Chris many thanks for your help.
Now that the macro is working thank to your intervention, I hope it can be useful to other fellows of the amazing KM community.

14

It is not enirely clear to me why in KM the default PWD for shell scripts is /. I imagine this can produce similar problems in other use-cases (?).

@johseb: BTW, you indeed do need the path to the PDF engine (context or latex) in KM’s ENV_PATH variable. It is not related to the present problem, but without it you get a different error (“context [or latex] not found”).

15

You're right. For Pandoc executable you may specify the full path in the script but for PDF engine you indeed need to set ENV_PATH correctly.

I made a note in the original post to make it clearer.

16

You can also set the path of Pandoc (and other Homebrew-installed tools) in ENV_PATH. That’s how I’m working all the time.

It would be painful to add the paths in each shell script action.

That’s my ENV_PATH:

~/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin

With ConTeXt it looks like this, but usually I don’t need that for KM:

~/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin:/Users/tom/ConTeXt/Beta/tex/texmf-osx-64/bin

The ENV_PATH variable does the same as if introducing each shell script action with

PATH=~/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin:/Users/tom/ConTeXt/Beta/tex/texmf-osx-64/bin

… so you can also set the path to the PDF engine in the script action itself, if you wanted so.

17

I know this is a bit of an old thread, but I just wanted to share the full directory version of the pandoc option for the latex engine, as it took me a bit of searching to find it, way back when. Just change it from

--pdf-engine=xelatex

to

--pdf-engine=/Library/TeX/texbin/xelatex

Assuming your TeX installation uses the defaults, as mine does. I put that in every pandoc command so I don't have to worry about setting the path.

18

Hey @bastionBrioche,

I would just add the appropriate directory to an ENV_PATH variable in Keyboard Maestro.

Mine currently looks like this:

/opt/local/bin:/opt/local/sbin:/Users/chris/perl5/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/Library/TeX/texbin:

It saves a lot of trouble down the line.

See:

Path in Shell Scripts

-Chris

19

Thanks; I find 'Path' a bit confusing to be honest. I like the script to contain as much information as possible so it doesn't depend upon my remembering to set a variable elsewhere. In my experience, if I can forget something, I will. I can imagine wanting to change the path in future and forgetting it was set in an environmental variable, for example. So I can see it's a good idea, but I don't think it will work for me. I just wanted to put the path up there so the information is easily available for people like me!