Macro to *Find* a Bear Note with a RegEx?

[August]

I'm brand new to Bear in the past couple of days. If I'm going to keep using it, there's one thing that I have to be able to do: give a macro a string and find that string within the titles of all my Bear notes (probably using a RegEx for the "find within" part) and then paste some other text into that note, just after the title. And if no existing note is found, then make a new one and paste the text into it.

On the Bear Community Forum, I found this macro which creates the Bear note on the left:

In the Bear forum as well as here, every Bear KBM macro that I've found has been about creating new Bear notes, converting Apple Notes to Bear, etc. Not one macro has included the need to find an existing Bear note.

And that's what I want to be able to do. I have the feeling that with another week or two's use of Bear and some experimenting with the macros that are available, I might eventually have a clue how to approach this problem. But for now, being a Bear novice, I'm floundering.

For instance, I can send a ⌘F to Bear and enter a string, but the results are visual. How might I tell if there was a result or not? The visual image is going to depend on the search text, so to use Find Image on Screen I would have to generate the image to search for. That seems totally awkward, if not impossible.

So I'm asking, if you have an idea for how to approach this, would you please give me an outline? I'm not asking for a working macro, just the steps that someone with more experience than I at manipulating Bear with KBM might use to make it happen.

Thanks.

[kevinb]

I understand that Bear notes are plain text Markdown files. I wonder if you might find it useful to consider thinking outside the bear… For example, you could have a macro that performed a regex string on the folder that contains your notes (and of course then opened that note).

I don't use Bear so I expect there are far better methods.

[Nige_S]

@kevinb

I understand that Bear notes are plain text Markdown files.

IIRC, you can write in Markdown in Bear but the app actually keeps all the notes in an SQL database.

@August

So I'm asking, if you have an idea for how to approach this, would you please give me an outline?

Given Bear has no AS support -- take a look at its Shortcuts actions:

"At the start" appears to be "after the Title and before existing content".

Note -- not a Bear, or even much of a Shortcuts, user...

[kevinb]

@Nige_S

IIRC, you can write in Markdown in Bear but the app actually keeps all the notes in an SQL database.

Oh dear. Well I've probably said enough on that subject today.

[August]

@kevinb

@Nige_S

IIRC, you can write in Markdown in Bear but the app actually keeps all the notes in an SQL database.

Oh dear. Well I've probably said enough on that subject today.

Not to worry. The lack of separate text files was one of the things that kept from even thinking about Bear. But I've discovered it has a feature that works well for me: each separate note can be opened in a separate window and those windows can be on different Desktops, and Bear preserves the Desktop location through a reboot. And I found out this week that as of v2.0, released last year, it now folds text, so I can collapse all the text and subsubheadings underneath any subheading.

The reboot onto the original Desktops is a must-have for me and the folding has been on my wish list for years. I was finally able to upgrade past Catalina as of last month (catching up on Monterey before I move further) and sadly discovered that with the OS update, TextEdit no longer preserves windows on their Desktops. That prompted the search and with the new Text Folding feature, Bear returned to my short list.

[August]

@Nige_S

take a look at its Shortcuts actions:

[Image omitted]

"At the start" appears to be "after the Title and before existing content".

From what I can tell, Bear has some adequate support for Shortcuts. I had not realized that KBM can Execute/Edit Shortcuts too. That's something to explore. But I've gotten extremely frustrated every time I've attmpted to use Shortcuts because it seems to be missing so many basic programming operations. Or I don't know how to implement them.

For instance, in Shortcuts I can Search in Bear for a string. I can limit it to Notes that have a specific tag. And then what? Where does the result go? It doesn't seem to be passed to the next operation, or does it need to be passed to a capture operation? I'm spoiled because KBM has all that so well wired.

Bear also has an API based on the x-callback-url scheme. I've gotten that to work using both an API url in a browser and using open x-url... in a shell script or KBM Execute Shell Script window.

What I have not yet figured out in using x-callback-url is how to actually use x-success and x-error. The syntax is that in the original URL you provide both of those as arguments and each has a parameter of a URL that will then be opened on either success or error. I don't even know yet whether or not I need them or if they are the only way to capture output from the API operation. It's also a framework full of assumptions that I have not encountered before.

If anyone has any tips or tutorials for either Shortcuts or x-callback-url, I'd appreciate pointers.

[August]

I've been confused about how to handle the cycle of both success and error calling yet other URLs. I've been thinking that maybe I could do something with a URL like "file://..." but that has not been gelling.

I just found out, or got reminded, that Keyboard Maestro has a built-in web interface where I can use a URL like

https://localhost:4491/action.html?macro=<YourMacroUUID>&value=<TheTriggerValueToPass>

to call a specific macro with a value that KBM can read. That MacroUUID value is one I can get for any macro from the KBM Editor. So unless I hear other suggestions, this is the direction I'm going to try next.

I chased a dead-end for a day or so of thinking, mistakenly, that I could use AppleScript with Bear, but all that's available, I think, is System Events stuff like this that I found on https://www.reddit.com/r/bearapp/comments/12dfg9u/bear_to_things_3_quick_entry_with_autofill/:

The following AppleScript works for me:

on run {input, parameters}
  tell application "System Events"
    tell application process "Bear"
      click menu bar 1's menu "Note"'s menu item "Copy Note's Identifier"
    end tell
  end tell
end run

I then used "Get clipboard" action to get the copied note ID. Hope this works for you too.

From the Bear doc, it appears that Shortcuts is well supported, but Shortcuts is difficult to do any real programming in. At least, it is for me. What I want to do isn't supported by the existing Shortcuts actions and editing the Shortcut as I'm trying things out is such a pain, I'm giving up on that direction.

So, to wrap back around, setting up the Keyboard Maestro server and using calls to that as the callbacks for x-success and x-error seems the way to go. For now.

Does this idea make sense or seem like it has a gotcha in it?

[Nige_S]

@August

Does this idea make sense or seem like it has a gotcha in it?

IMO the gotcha is you using Bear

Not a Bear user, so could be wrong, but Bear doesn't have regex support in searches. And there's no obvious way to get all the note titles so you can do the search externally then reference the result. Which really leaves you with directly accessing the mysql database (if that's even possible).

If you want to automate a notes app in the ways you want rather than the (limited) ways the devs provide then you should look elsewhere rather than jumping through hoops. You're certainly in the "power user" bracket -- have you tried Obsidian? Plenty of opportunity to nerd out with that and, since each Obsidian note is a discrete text file within your "normal" file system, operations like the one you're trying to do here are trivially easy.

[kevinb]

@Nige_S

IMO the gotcha is you using Bear

I agree, but I didn't like to say, because @August stated very clearly what he found suitable about Bear, and so often the choice of software is the most acceptable match without being the perfect match! That said, yes, if regex searching is among the highest priorities, trying to get that done in Bear could be hairy.

Moving on from that unbearable pun… I didn't mention Obsidian because I'm sure @August must know about it by now and think he would find its very un-Mac-like handing of windows to be repellent. I wouldn't expect it to be very clever about use of "spaces" either. I use it every day but it's a cross-platform app and that does show.

[Nige_S]

@kevinb

Moving on from that unbearable pun… I didn't mention Obsidian because I'm sure @August must know about it by now and...

Good points all. And my apologies, @August, for not considering my reply more carefully.

In an attempt to make amends -- it's not that difficult to regex-search the note titles and return a match list to KM. You could then feed the accurate title ("Prompt with List" if more than one matches the regex) to the "Prepend" Shortcut.

Bear Regex test.kmmacros (4.0 KB)

Image

If you look through the database schema you might find a way to return the note's "Identifier" for more accurate targeting, essential if you have multiple notes with the same title.

[August]

@August

setting up the Keyboard Maestro server and using calls to that as the callbacks for x-success and x-error seems the way to go. For now.

I've already superceded this ides. Since I have no need to access this from other machines on my network (yet) or from elsewhere on the 'Net, I don't need the KBM server, I can just use the simpler kmtrigger://... URL trigger, documented at [https://wiki.keyboardmaestro.com/trigger/URL](trigger:URL [Keyboard Maestro Wiki]). It only works on the local macnine and can trigger a macro by name or by UUID and can pass an optional value. So this seems a good way of reporting an error or continuing actions after a success.

@Nige_S

have you tried Obsidian?

I have. While it looks like it provides some powerful and even fascinating tools, and I may eventually use it for brainstorming-type writing, etc., it does not serve for the current purpose because multiple separate windows all show up in the same Desktop after a reboot.

I have numerous tools that do not support multiple Desktops, but in this case I specifically want one that does. I had been building a system on Catalina where I coujld press a hotkey and it would take the current Chrome window (it also worked on Brave), switch to that window in Tabs Outliner where it would copy the tabs hierarch of that window, then switch to the corresponding "Tabs" RTF file in TextEdit for that Desktop and paste in the list of tabs as links. I was getting that system tuned when I upgraded to Monterey, TextEdit no longer reboots into the various original Desktops and copying the tabs hierarchy of a Chrome window in Tabs Outliner is unreliable. But I'm enamored of the per-Desktop archiving of my browser histor. So I started looking, again, for an aesthetically pleasing editor that keeps windows in their Desktops (where "aesthetically pleasing means: supports a dark mode, has different size fonts, maybe Markdown support as long as it's not dual-pane).

Bear does what I need and the ability to fold sections is an attractive bonus. Typora also meets the criteria (but it doesn't fold) but I've already dedicated it to journaling and other personal writing and I need totally different themes/styles for these two uses, and the theme/styles setting is app-wide, not per file. I haven't found another yet. And Bear has an API, Typora does not, not even AppleScript support.

If you've read this far, thanks for listening.

[August]

@Nige_S

there's no obvious way to get all the note titles so you can do the search externally then reference the result.

There's no way using the UI or Shortcuts, but there are apparently straightforward ways using the x-callback API. Unfortunately, it's x-callback, not AppleScript or JavaScript or something else I'm already familliar with. So I'm figuring this one out. It sure beats Shortcuts. It looks like I can do various data extractions, pass the info to KBM for manipulation, and generate calls to modify the data, cyclically. At the core, that should be enough. I just haven't gotten there yet. So far.

[Nige_S]

@August

There's no way using the UI or Shortcuts, but there are apparently straightforward ways using the x-callback API.

Not that I could find. There's no regex, and leaving the search term blank puts Bear into search mode with no records returned (for me, anyway, but I fully expect I'm doing it wrong!).

The only way I can see of getting all note titles is to give every note a tag you can search for.

Do have a look at the sqlite3 command in the macro above. That only took me a couple of minutes to work out, and I'm sure you'll be able to whip it into whatever shape you want even quicker. It works entirely in the background and, since it's read-only and fast, shouldn't cause problems with Bear's own db access. If nothing else it will allow you to regex to get a full title, maybe an ID, that you can then use in your /add-text callback.

[kevinb]

@Nige_S

Good points all. And my apologies, @August, for not considering my reply more carefully.

I'm sure @August will have recognised the good intention. In the same vein, I hope he will forgive me sticking my neck out a lot and airing something that concerns me, and that is how the centring of a virtual desktop approach is limiting options again.

I am an ignoramus when it comes to virtual desktops. I must be, because many skilled users continue to swear by them, and they seem a good idea, but they just never click with me in practice. So I am not trying to provoke a defence of them. @August knows what is best for his workflow. It worries me, though, when I see this:

@August

sadly discovered that with the OS update, TextEdit no longer preserves windows on their Desktops.

We know that Apple can be flakey these days with maintaining old MacOS features, so I hope that the above is not an indication that Apple has lost interest in Spaces. After all, might Spaces not undermine the choice of buying a Mac Pro and eight displays?

I hope this post won't be a distraction from the main topic.

[August]

About the research, development, and design...

@Nige_S

@August

There's no way using the UI or Shortcuts, but there are apparently straightforward ways using the x-callback API.

Not that I could find.

I may have misspoken/misremembered. I was sure that I had seem something on Reddit/bear about this, but I'm not good at saving ahead of time what I'll remember later. All I could find was someone searching for " ", , so that they would get all notes returned, with the caveat that this was better used when limited to a specific tag.

Actually, the tag limitation works for me, in this particular case, but it still seems like a brute-force approace to just grab everything and filter it externally.

@Nige_S

Do have a look at the sqlite3 command in the macro above.

Thanks you for the nudge/reminder. Yes, it looks pretty sweet and much less of a brute force approach, suitable for searching everthing and not just a manageable subset.

At this point I know nothing about SQL. I didn't even know that sql3 was on my Mac. Conceptually it seems trivial, but how would I structure returning the ID that goes along with those RegEx-matched titles? If that's more than a one-minute question, I'll ask Claude.ai or ChatGPT for a tutorial. Sometiimes for that kind of stuff they even get it right.

The Callback approach that I had been persuing is seeming to be less than straightforwardly useful.

@August

It looks like I can do various data extractions, pass the info to KBM for manipulation,

Um, no. I had been hoping that within the x-callback-url scheme, I could use the /x-success parameter to specify calling kmtrigger://... to return the data. Unfortunately, there is no way to specify the structure of what is actually passed to the callback URL. It's just a blob of JSON. It looks like there is no way in this scheme to get that JSON into the value="whatever" that kmtrigger://... needs to see. Or so I think. The doc on how /x-success is actually handled is extremely sparse. The primary X-Callback doc has only two extremely rough examples that are not well described. IMHO either it was written by an engineer who assumed that their audience already knows how it works, or it was written by someone who just copied what an engineer told them without understanding it al all. It totally assumes that the target URL process already has an implementation of x-callback and will be able to seemlessly interpret what it is given, so there is no need to go into the details. It does not even mentinon that the data is passed in a JSON format.

I did, however, get pointed to a useful GitHub utility, martinfinke/xcall, which is described as:

Call X-Callback-URLs From the Command Line. Outputs the x-success and x-error responses to stdout/stderr.

Which is extremely useful. That's where I saw that the /x-success output is JSON.

At this point, between xcall being able to dump the /x-success data to stdout and sql3 hopefully being able to query and extract with more power than Shortcuts, I seem well on the road to being able to figuring how how to do what I have been wanting to do.

I'll update if there are more plot twists.

[griffman]

@August

I'll ask Claude.ai or ChatGPT for a tutorial.

I wrote a multi part series on using it with Keyboard Maestro a while back:

Using sqlite3 databases with KM macros: Part 1

Overall objective This series is designed to help those looking to use sqlite3 databases with their Keyboard Maestro macros. It assumes minimal to no knowledge of sqlite3, and I'll do my best to clearly explain things as I go. If you have questions, please ask and I'll do my best to clarify. By the end of the series, you'll hopefully have a functional Keyboard Maestro macro that creates a database, and then interacts with that database to read data, change data, add data, etc. With the knowled…

-rob.

[August]

About the meta-topics

@kevinb

@Nige_S

And my apologies, @August, for not considering my reply more carefully.

I'm sure @August will have recognised the good intention

Absolutely. I completely understood the wry humor in the comment. I'm quite aware that my deal-breaker must-have features are not necessarily in the mainstream of what marketing departments know how to sell.

@kevinb

when it comes to virtual desktops ... they seem a good idea, but they just never click with me in practice.

I suspect that's because Apple has never had very good support for them. It's not your fault, there's been an Apple-imposed steep learning curve. And even the apps that were starting to make Spaces manageable that began to appear around OSX 10.8 have disappeared because Apple pulled the rug out from under them by removing nearly all public API support by OSX 10.11 (I think I have those numbers right, but I didn't verify).

Without getting defensive (but absolutely climbing onto my soapbox the moment I think I have an audience), they make it possible for me to work on a laptop. Period.

I've always been a bit ADHD or something. I always have numerous things going at once. I can be writing a paragraph and need to reasearch something on Wikipedia as well as several other sites, put the research into a spreadsheet, put several things onto my to-do list and expand the details for another item I find there, and then go back to the paragraph I was writing. I have never had the "discipline" (or whatever it is, the plodding slowness of mind, the embracing of drudgery...no offense intended) to use one app and then put it away and use another app and put it away, etc. Like Einstein and Napoleon, I have always had a notoriously messy desk long before computers came along to tame the problem somewhat (or at least hide it).

When I worked at Google in the Google TV group, I had a 50" TV as a monitor on my desk, because that was part of our target audience. I literally could not touch the upper corners of the monitor while staying sitting in my chair. So I could have a very cluttered desktop and still work. But when I took my laptop home, I needed another way or organizing things. I had a normal-sized external monitor and the laptop, period. When I worked at Apple and again at Netflix, I plugged the laptop into a large dual-monitor setup, and I always had the laptop open too, giving me three screens. And again I didn't have nearly that much real estate when I worked at home. And now, since I retired, my external monitor has been in storage for five years and my laptop is what I work on, period.

I've been banging on this concept for my own sanity. I got initially introduced to what could be done by an app called "Current-Key Stats" which had been developed by a guy who had numerous clients and used it to assign multiple Spaces (he called them Rooms) to each of multiple clients so he could easily keep track of what time he was spending on each client and have his timeclock automatically switch accounts when he switched screens.

I had no use for the statistics side, but being able to ignore the Mission Control Desktop Numbers and give each Desktop an actual Name, of more than a few characters, and being able to rearrange the order of the Desktops and have the names move with them (instead of, "Oh, wait, KBM is now in Desktop 5 instead of 4 because I moved Research in Desktop 7 up to position 3...") — it was doing what computers are supposed to do, it kept track of thing FOR me instead of me having to keep track in my head of what the computer was doing.

I found it so useful that I started writing a book about how to set up a system like mine, because it was really useful, even graceful, intuitive, and supportive and I could find nothing else like it and found lots of people complaining about what Apple didn't provide.

Unfortunately, CKS was a one-man show and Apple introduced changes in the OS that required he buy the newest hardware if he was going to keep supporting his users — at the same time as he became a father and had a whole lot less time and money to devote to his pet project. So he withdrew CKS.

Which meant my book idea was a bust; how useful would it be to people to tell how to use an amazing piece of software that was no longer available, with no replacement in sight? Yet I still had the motivation because it really is extremely useful, at least to me.

Eventually, I got the author of CKS to share his "trick" with me. He told me that he put an "invisible" app window on each Mission Control Desktop and the title of that window was the name of that "Room". Combined with making an Apple Script "observer" to watch for the one remaining Spaces system notification that Apple still has, I was able to start building my own replacement for CKS. When Peter added the Space Change Trigger to KBM, I had something more reliable as a foundation and have kept building.

I don't mean to slight Jim Sauer's MACROS: Desktop Spaces • Macros to Improve Navigation and Window Management, v4.0. He has a very highly evolved and robust system there, but it based on the Desktop Numbers and I have trouble keeping track of more than about seven such. Because he uses Apple's Keyboard Shortcuts, he's limited to 16, which is very likely enough for most people, but by the time Current Key Stats stopped being available, I already had over twenty. (There's some discussion between @_jims and me here in the forum archives.)

My space management system has been pretty stable for over a year now, ever since I created a pushdown stack for Deskspace names. It survived the translation from Catalina to Monterey seamlessly, as far as the basic Space naming and selecting and the window moving tools. I had been working on a per-Space browser tabs management tool that broke in Monterey, and that's what this thread is about, using Bear to keep per-Space tabs archives (and maybe eventually ToDo lists).

@kevinb

I hope that the above is not an indication that Apple has lost interest in Spaces.

Apple has never had much interest in Spaces. If they ever completely stop supporting it, I'll have to switch to Linux. As I mentioned, Peter's support for the one remaining Apple API, NSWorkspace.activeSpaceDidChangeNotification. has been extremely useful.

[August]

@griffman

I wrote a multi part series on using it with Keyboard Maestro a while back:

Thanks Rob! I'm looking into it now...

[Nige_S]

@August

All I could find was someone searching for " "

Of course, search for something (almost) guaranteed to be in every note...

@August

Conceptually it seems trivial, but how would I structure returning the ID that goes along with those RegEx-matched titles?

The trick, as with any structured data, is knowing where the data you want is within the structure. I'm no SQL guru, but a few simple commands let me go spelunking in Terminal.

Open the database in an interactive session (you can copy the path from the Finder, obvs):

sqlite3 ~/Library/Group\ Containers/9K33E3U3T4.net.shinyfrog.bear/Application\ Data/database.sqlite

You can then list the tables in the DB:

sqlite> .tables
ZSFCHANGE              ZSFNOTEFILE            Z_5TAGS              
ZSFCHANGEITEM          ZSFNOTEFILESERVERDATA  Z_METADATA           
ZSFEXTERNALCHANGES     ZSFNOTESERVERDATA      Z_MODELCACHE         
ZSFINTERNALCHANGES     ZSFNOTETAG             Z_PRIMARYKEY         
ZSFNOTE                ZSFPASSWORD          
ZSFNOTEBACKLINK        ZSFSERVERMETADATA

ZSFNOTE looks promising (luckily they've used sensible table names!) so let's look at the schema of that to see column names (much data snipped from the following:

sqlite> .schema ZSFNOTE
CREATE TABLE ZSFNOTE ( Z_PK INTEGER PRIMARY KEY, ..., ZTEXT VARCHAR, ZTITLE VARCHAR, ZUNIQUEIDENTIFIER VARCHAR, ... );

...and we can guess that ZTEXT is the text of the note, ZTITLE is the title, ZUNIQUEIDENTIFIER the UUID. I've included the text so that, if you've two identical titles, you'll have some context when choosing.

The general form to get data out of the db is

SELECT columns FROM table WHERE condition

... and the WHERE keyword REGEXP says to match by regular expression, so we'll want to start with

SELECT ZTITLE,ZTEXT,ZUNIQUEIDENTIFIER FROM ZSFNOTE WHERE ZTITLE REGEXP ourRegex

...to get the title, text, and UUID for every note that matches ourRegex. But getting all the text seems excessive just for context, so let's get the first 20 characters by using the substring function on that part of the data:

SELECT ZTITLE,substr(ZTEXT,1,20),ZUNIQUEIDENTIFIER FROM ZSFNOTE WHERE ZTITLE REGEXP ourRegex

Change the shell script action in the macro form earlier to use that command

...and, given that we search for Third and there's a note titled "A Third New Note" in Bear, the result when the macro is run is:

...and we see a couple of problems. It's worked, but those field delimiters probably won't be unique (eg shell script snippets could have pipes) and the text of the note (redundantly) includes the title and will contain linefeeds. For speed (because I've gone on long enough!), we'll deal with the text problems by upping the character count then using KM text actions to make sense of things. But the separators we can change in the sqlite3 command, and I'll go with "==myKMNew==" for record separators (otherwise differentiating between record-separator linefeeds and in-text linefeeds will be a pain) and "==myKMCol==" for the columns:

sqlite3 -newline '==myKMNew==' -separator '==myKMCol==' ...etc

All of which is far too much "tell" and not enough "show" -- but it hopefully sets the scene for the demo, if only to prove that I haven't clue and was googling most of this as I went

Bear Find Note by Regex Demo.kmmacros (15.3 KB)

Image

Perhaps someone who does know what they're doing (nudges @griffman...) will clear up any issues...

[August]

Thanks @Nige_S, that is some awesome progress.

I hate to be the bearer of bad news in response to all that work (e.g., I would probably not have thought about using specialized strings as delimeters to be able to find them again later and change them). But...

I tried it on my existing Bear notes and immediately found a bug. I have one Bear nots with a title that includes the word "Meta". That word also appears 10 times within the text of the note. That word does not appear anyewhere else in my Bear notes. When I enter "Meta" as the RE to search for, the macro responds as if it has found TWO notes, one matching the title and one matching the third instance, but displayed with a different UUID.

And I figured it out: There is a note which has that "third instance" in the text as the tite of a note that is in the Trash! So when I search in Bear, it only reports one note has the term, but the macro finds them both.

With that settled, why did you replace "\R" with " "? (I'll change that to %SPACE% so its obvious.)

And with two results, it seem as if "==myKMNew==" isi not being replaced, it appears in the Text Display Window. Maybe it's a glitch in the multiple results logic. I haven't traced my way through that yet.

Thanks again. I'm not going to mark this "Solved" yet, because we're still conversing, but this defintely solves the OP.