"Literary Toolbox" Palette

mrpasini · January 19, 2021, 6:58am

LITERARY TOOLBOX II

Just a note to say I'm providing a version of this macro that works on very large files.

I tried running a few of the analyses on an 84K file with the original macro group and it overran the environment size. So rather than reading the selection stored in a Keyboard Maestro variable, this version writes it to a temporary file, reads it into a Perl variable and deletes the temporary file, avoiding filling the environment space.

Keyboard Maestro can accept a variable of any size, as I understand it, but once the system environment is full, it deletes the largest variable, which causes the original version of this macro group to fail. This version avoids that.

Seems to work fine in my limited testing on two machines. It seems as fast as the original on an SSD but it's no slouch on a hard disk.

Update: Here it is. The temporary file is written to Users/Shared/temp.txt because that location doesn't depend on your user name. It's also deleted right after it is read. This version uses Option-F10 as the trigger (the original used F10) so you can use both.

BTW, no need to reinstall any of the supporting software. Just import the updated macro Literary Toolbox II.

Update (21 Jan. 2021): This version fixes an issue with the Concordance that did not maintain the original line endings in the report.

Update (9 Feb. 2021): Revised the location and name of the temporary file from Users/Shared/temp.txt to tmp/LT-temp.txt. It is, however, deleted immediately after it is read. Note, too, that some functions (like the grammar check) take a while to run on longer texts. This version, however, does not show a progress bar for them.

Update (8 March 2021): Updated code to handle revised Emotion module. Removed restrictions on InDesign CS5 as obsolete. Added WordNet Lookup option for selected word [undocumented beta]. And I just tweaked the WordNet Lookup with 1) a link on the headline word to the Princeton form for deeper exploration and 2) colored examples.

Update (10 March 2021): I've added the PDF of a draft of the revised manual as a separate download below.

Update (11 March 2021): A few changes:
• Added local WordNet Lookup option
• Change hot key for WordNet options
• Activated WordNet links in remote option
• Updated Spell Catcher word lookup
• Revised Spell Check Document to include some non-standard keyboard shortcuts which you can extend easily for other applications
• Revised documentation

Literary Toolbox II Macros.kmmacros (203.1 KB)

LiteraryToolbox.pdf.zip (740.2 KB)

ccstone · February 9, 2021, 6:47pm

Hey Mike,

I'd stay out of the user space and use the /tmp directory.

-Chris

mrpasini · February 9, 2021, 6:52pm

I suppose I could prompt for a location if the corresponding variable were blank rather than use a hard-coded location available on any machine. But what's the advantage? Or, to put it another way, what's the disadvantage to using the Users/Shared location briefly?

NB: I just saw a typo in the original message regarding the location of the temporary file. It's in Users/Shared not Users alone.

ccstone · February 9, 2021, 6:59pm

Hey Mike,

The /tmp directory is available on all Unix machines and is specifically designed as a destination for temporary files.

Staying out of the user space when you don't need to be in there is a basic safety measure.

-Chris

mrpasini · February 9, 2021, 7:32pm

Ah, yes, forgot all about that. I've revised and replaced the group to use tmp/LT-temp.txt. And tested it, too.

Thanks for the help!

xsmk · February 27, 2021, 1:07pm

it works like magic.
thank you a million mr pasini !

mrpasini · February 27, 2021, 4:34pm

You're welcome! Glad you find it useful.

avialan · March 6, 2021, 7:01pm

Mike, thanks for posting this amazing tool. I showed it to a journalist friend about to start a new job, and blew their mind. We just sold a couple more KM copies for Peter

I believe I have everything installed correctly, but I ran into a bug in 'Emotion Check". It spits out this error repeatedly

Use of uninitialized value in numeric lt (<) at /var/folders/zp/pp7dtsms5mgdtmsz2qz0_56m0000gn/T/Keyboard-Maestro-Script-A50D2931-D590-4C56-9E68-0B07D42CE5AB line 32.

.
I tracked it as far as the highlighted line in your perl script, but I'm not a perl guy. ("+" just added here, for highlighting)

    for my $value ( @emotions ) {
        print "<p>Sentences with the most <span>&nbsp;$value&nbsp;</span>:</p><ul>";
        for my $sentence ( @{ $opinion->sentences } ) {
+            next if $score{$sentence}->{$value} < $max{$value};
            print "<li>$sentence</li>\n\n";
        }
        print "</ul>";
    }

.
I imagine it's just an unitialized variable. I ran the tool from BBEdit, selecting a page of article text. Other content checks, such as "opinion" and "readability" work fine. I double-checked that I ran this command for installing Opinion

cpanm Lingua::EN::Opinion --sudo
here's a log.
opinion-install-log.txt.zip (987 Bytes)

Thanks for any help you can provide, keeping the 'Emotion Check' in check!

Avi

mrpasini · March 6, 2021, 8:21pm

Thanks for the kind words, Ari.

I've never seen that error message on either the original version or the version for long texts, which both use the same Perl module. I just ran an Emotion check on a 20,000 word file without a problem. It went through eight emotions with high scoring sentences displayed. I also ran it on an InDesign document's text with no problem (which, frankly, surprised me).

The error message doesn't say which side of the < is unitialized.

If we assume there's nothing special about your text, the suspect would be the Perl module itself. I haven't updated it here since Dec. 2018, so that would be v 0.1502, whereas a new install today would be v 0.1600, which involves a lot of "code changes" according to the release notes.

Let's try an experiment.

Replace the line you cited with this line:

       next if !exists $score{$sentence}->{$value} || $score{$sentence}->{$value} < $max{$value};

That change comes from the current example code.

If that solves your problem, I'll roll it into the distribution here. (I tested the code with my version of the module and it works as expected.)

Meanwhile I'll see if, after installing the current module, I can duplicate the issue here. Thanks for pointing it out.

Edit:

The revised example code has one other line that may help chase this down. Here's the entire segment:

for my $value ( @emotions ) {
    next unless $max{$value} > 0;	# new
    print "<p>Sentences with the most <span>&nbsp;$value&nbsp;</span>:</p><ul>";
    for my $sentence ( @{ $opinion->sentences } ) {
 #       next if $score{$sentence}->{$value} < $max{$value};
        next if !exists $score{$sentence}->{$value} || $score{$sentence}->{$value} < $max{$value}; # new
       print "<li>$sentence</li>\n\n";
    }
    print "</ul>";
}

I've commented the two new lines (# new)) and commented out the previous line that causes the error.

avialan · March 6, 2021, 9:10pm

That worked! - just the single line replacement, I didn't have to try your 2d suggestion. Thanks for the rapid fix.

avialan · March 6, 2021, 9:18pm

If it helps to explore the bug, here's the source text, sent with the author's permission ( © Gannet Co.). I got the same results from BBEdit and Pages.
test_article.txt.zip (5.9 KB)

mrpasini · March 6, 2021, 10:37pm

Thanks, that does help a lot. It confirms that the issue is the updated module.

On my installation with the older module (and either version of the code), the report runs as expected. So the new code, which you confirmed works, must be required by the new module.

I'll update the distribution with the new code shortly.

Thanks again for the detective work!

mrpasini · March 8, 2021, 7:44pm

I've just updated the Large File Version above with the new code. And I removed references to InDesign CS5.

I've also added a WordNet Lookup option for a selected word. Undocumented and in beta at the moment as I try to make the report a little more readable than the raw HTML. To use it, select a word before access the command from the palette. The Princeton database will be searched online to retrieve definitions and synonyms.

I just tweaked the WordNet Lookup with 1) a link on the headline word to the Princeton form for deeper exploration and 2) colored examples.

martin · March 9, 2021, 1:13am

Hi @mrpasini, thanks for sharing this tool. Looks very powerful. I'm trying to set it up.
Just need some clarifications.

You have the LiteraryToolbox.10e.zip file and the Literary Toolbox II Macros.kmmacros
Only the first one contains the PDF file. So after installed the latter, not finding the PDF file, I downloaded the former, also installed it, and followed the instructions in the PDF.

My first question is: Do we use the PDF in the first file and use the second LT II set since it is the latest?

Following the PDF, I have downloaded the file Lingua-EN-Grammarian-0.000005.tar.gz from Lingua::EN::Grammarian - Detect grammatical problems in text - metacpan.org. The PDF only asks to

Copy the grammarian_cautions and grammarian_errors ﬁles to any one of the following locations:
/usr/local/share/grammarian/
/Users/[username]/
/Users/[username]/Desktop/

So I unzipped the downloaded file and moved the two files to the first location. Now, what should I do with the rest of the files? Just delete all of them? (screenshot below)

My third question is:
You added the highlighted instructions (screenshot below):

I can see the first cmd is to make new folders and the second cmd is to copy the file begins with grammarian?
I assume these two cmds are essentially the same as the instructions as Copy the grammarian_cautions and grammarian_errors ﬁles to any one of the following locations:.

I have already manually created the folder and moved the two said files to that folder. But I don't understand what the second cmd is supposed to copy. The file is downloaded to my Download folder, after unzipping, all files are in another folder named Lingua-EN-Grammarian-0.000005.
Can you clarify?

Thanks!

mrpasini · March 9, 2021, 2:11am

Sorry for any confusion, Martin. I’m not able to access the original post to update it for some reason.

If you’re following the bouncing ball in this thread, LT II replaces the last version of its predecessor but not its documentation. It handles bothe small files as well as very large ones, works the currently available modules for new installations (as well as the original modules) and now adds the WordNet lookup.

So follow the installation instructions in the PDF but import LT II and you’ll be using the latest and greatest.

Alternately, if you only work with shorter text (don’t ask how short because it depends on your environment), you can use the latest non-II version with the new Emotion code detailed above.

I’ve been trying to convince myself to just move on with LT II but it’s only been a couple of days. It doesn’t feel slower on short texts so I don’t see a downside.

By the time I work out the WordNet function, i’ll update the package with a new PDF.

(And yes you can delete what you don’t use in that one module.)

martin · March 9, 2021, 4:16am

Thanks, @mrpasini.

Some macros use Spell Catcher. I never heard of it, so I did some search. Looks like it has been discontinued. Is it compatible with Big Sur? If so, which place do you recommend to download it? For me, the most needed feature is spell check.

I saw most apps you listed to make the macros available are text editors and browsers, but not rich text word processors such as Microsoft Word. Any reason behind it?

Also, I'm wondering how to use the Spell Check Document macro? It just has a keystroke action. I don't know what it does at all.

Some thoughts for improvement:

If you could add a "Comment" action to each macro and put some notes/description/instructions about that macro, it might be great. Information such as what it does, what app/files are required, how to use, trouble shooting, etc.

Since your macro is evolving, this forum might not be the best place to update every version. I think in this case, host your file and instructions at github might be a better choice. You can simply post a link, screenshot, etc. here to update us. Trying to find the correct version here is not an easy task.

Again, thanks a lot for sharing it with us!

martin · March 9, 2021, 4:19am

~~Is it possible that it was due to your post being edited by one of the forum moderators? (just wild guess)~~
Edit: wrong guess. Sorry.

JMichaelTX · March 9, 2021, 5:00am

No. This is incorrect.
The Forum software limits the amount of time that a user can edit his/her post.

Please refrain from making wild guesses, as this can lead to misinformation.
If you have a question like this, you can PM the Forum owner or one of its moderators, or post a question in the Forum Admin Category.

martin · March 9, 2021, 5:10am

Sorry. I’ve edited my original post to clarify that. Thanks.
Does the forum notify user that it is locked due to long time when he/she tries to edit it?
A message like that will clarify things.

mrpasini · March 9, 2021, 5:27am

Spell Catcher is a venerable application that goes way back in Macintosh history and has outlived its author by many years and several operating system updates. I am still able to use it in Mojave (using the keystroke trick) and do use the options in LT II, so I include them for anyone else with Spell Catcher. Just disable them if that isn't your case.

I don't use Word (since the fiasco of Word 3.0) but I do use InDesign and this works with that. No promises but all the modules assume UTF8 text (I believe) and the macros strip styles to accommodate them.

I do have a Comment for the undocumented and beta WordNet Lookup but I try to avoid cluttering up the Keyboard Maestro macro file by documenting the tool in an extensive PDF. Everything I have to say about it is in there.

It's not really in development. Keyboard Maestro simply provides a friendly interface for some obscure but useful tools that quite mature. What I might do is start a new threat with LT II when I'm happy with it.

Whatever the limitations of the forum software, a sequential reading of this thread will reveal all. It's a pain to find the current release, I agree, but a new threat will no doubt resolve that little problem.

Thanks for taking the time to share your thoughts. Let me know if you find some application that isn't compatible.

"Literary Toolbox" Palette

Options