Proposal seeking consensus for a standard location for *some* data files

[kevinb]

This is a proposal for a standard location which can be used by files needed by a user's macro files. The aim is to discuss and arrive at broad agreement on a voluntary common practice. I would not envisage official support in the Keyboard Maestro application—at least, not for now!

I recall that the discussion of the best location for such files has come up before, but I have forgotten in which topics and I have started afresh with my thinking anyway.

I am finalising a revised set of macros that I plan to upload here soon. It will require two or more Macs on a LAN to synchronise not just Keyboard Maestro macros in the usual way, but also to synchronise some small text files.^[1] This has prompted me to think further about having a voluntary standard for folder location and structure for dealing with such user files, so that creators of macros have the option of adopting a common practice.

I propose that the base folder should be at:

~/Documents/Keyboard Maestro User/

It is undeniable that the Documents folder has long since become a de facto standard location in which many applications will create folders (many years ago I retreated from the invasion by using a subfolder for my own files). The Application Support folder is not suitable because this matter is not about supporting an application; also the folder is less accessible to the user than the Documents folder is.

Regarding the folder structure, I currently have only one subfolder to suggest:

~/Documents/Keyboard Maestro User/Keyboard Maestro User Sync/

This is the folder to be synchronised between Macs. This leaves the option of adding further folders which would remain local to each Mac. I recommend Keyboard Maestro User Sync rather than just Sync in order to prevent user confusion when setting up file synchronisation software.

Within that folder would be the folder name for the macro, or suite of macros, that you have made, followed by a UUID. For "Example Macro" would look like:

~/Documents/Keyboard Maestro User/Keyboard Maestro User Sync/Example Macro (E43004EF-0B71-4091-ACF9-A0F54EFE2408)

The UUID is in case there are ever several macros (or set of macros) called "Example Macro", or perhaps different forks or versions of the original macro. Of course, it would be up to the macro's creator to supply the UUID.

I recommend the same model for any macro's folder that is outside the Keyboard Maestro User Sync/ folder.

---

1. The latter is not supported by Keyboard Maestro, but can be achieved by using synchronisation applications such as SyncThing, which is what I have been testing with. ↩︎

[griffman]

While I think this is an admirable goal, the problem you're going to have is people like me.

I despise keeping anything in the Documents folder, for the very reason you mention—it's become home to a huge mess of crud from every app I've seemingly ever installed.

In addition, I don't like it when a macro (or any app) forces me to store user-saved data in a pre-defined non-changeable location. I want to be in charge of my storage and how it's used, not the macro/app.

For me, I want all such files on my other SSD, the one I'm fully in control of. And by letting users specify both the location and the name of any such required files and/or folders, you avoid the need for UUIDs in filenames, which also make a visual mess of things, given their length. Users are responsible for insuring there are no name collisions, and they choose where their files are going to live. (There are times you have to use a UUID, but that should be an exception, not a rule. And the macro author can determine when those things are required.)

To me as a user, forced file/folder storage locations are too limiting and feel "iOS-ish." It's my Mac, and I want to save files where I want to save files.

As a macro author, it's up to me to allow my macros' users to specify locations and names, at least as much as feasible, and to make my macros work with those locations. (In some of my macros, the name is fixed, but if I save data, I always let the user choose the location.)

For this to work, I think it would have to be supported by Keyboard Maestro, in the form of a "Default macro data storage location" that one could set in Keyboard Maestro's settings. The user could pick, and the app would be aware of their choice, and maybe macros could use it via a token, %UserMacroDataFolder% or whatever. But I wouldn't want that to be the required storage location, just one that's available if you wanted to use it.

But I think that's unlikely to happen, which means people like me will make it very difficult to agree on a standardized storage location :).

-rob.

[kevinb]

Thanks for engaging with the idea. I don't think your objections are well founded, and I hope I can put your mind at rest here. The idea is to add user convenience, not to restrict user or developer options, which would, in any case, be impossible.

@griffman

I despise keeping anything in the Documents folder, for the very reason you mention—it's become home to a huge mess of crud from every app I've seemingly ever installed.

That's illogical, Captain. ("Highly illogical" doesn't make much sense, does it?).

You agree with me that "the Documents folder has long since become a de facto standard location in which many applications will create folders", and you agree with me that it is no longer a good choice for a user to keep their own files (other than in a subfolder). How does this in any way argue against the Documents folder being the best choice for the suggested purpose?

In addition, I don't like it when a macro (or any app) forces me to store user-saved data in a pre-defined non-changeable location.

"Forcing" never comes into it in any macro: a user can change a path in any macro, if they know what they are doing—and if they don't, they shouldn't, or should be given careful guidance.

In my forthcoming macros, the default path is explicitly changeable and accompanied by readme files that explain how to do so, while also cautioning about possible consequences. However, that is really just a further convenience for the user, and I am in two minds whether it's for the best.

For me, I want all such files on my other SSD, the one I'm fully in control of.

Go ahead. It must be obvious that nobody would be stopping you. I'm not suggesting an internment camp for everyone's macros, but a voluntary standard or common practice, to help facilitate the dissemination of macros in a way that does not require complicated setting up for each and every user.

As with yours, most of my data is on external drives. That's not really relevant to the question of the best location for macro data that is to be synchronised or which otherwise would be convenient to have in a widely known location.

If we may revisit this part:

@kevinb

I am finalising a revised set of macros that I plan to upload here soon. It will require two or more Macs on a LAN to synchronise not just Keyboard Maestro macros in the usual way, but also to synchronise some small text files.[1] This has prompted me to think further about having a voluntary standard for folder location and structure for dealing with such user files, so that creators of macros have the option of adopting a common practice.

Imagine you have a few independent macros that require synchronisation with other Macs (this may be a rare situation now, but I think it will happen more, and that we should facilitate an easy mechanism before we get there).

• Macro 1 stores its auxiliary files in ~/Example Macro/.
• Macro 2 stores its files in Volumes/SSD1/KM stuff/Well it worked for me/
• Macro 3 leaves the file location empty.

That's 3 macros for the user to change the path in (possibly in multiple places, even though that should not be necessary) and/or worry about setting up synchronisation software (with all its foibles) for, for up to three separate paths.

Compare that with having one "standard" location to sync once only for use with any compliant macro. There is no competition.

And that is just concerning synchronisation. I haven't thought about possibilities beyond that, but I would imagine that others might have ideas.

And by letting users specify both the location and the name of any such required files and/or folders,

They're allowed to, as I say. Also, the advanced user will know tricks such as using a symbolic link to the sync folder. Really, nothing is lost.

you avoid the need for UUIDs in filenames, which also make a visual mess of things, given their length.

I considered this, but the "visual mess" would be in a folder that one would rarely need to visit, and UUIDs are good standard practice. I considered alternatives, including author subfolders (e.g."kmforum/griffman/") and Nano ID, but decided to stick with the widely supported (including by Keyboard Maestro itself) UUID. Visual mess is not as important as having unique files, especially in a folder that is rarely visited by humans but frequently visited by software.

Users are responsible for insuring there are no name collisions

Do they have to be? I am trying to make macros that require negligible configuration for users (rather than just working on my system – I had things working like that weeks ago!) and also trying to anticipate possible, if unlikely, clashes.

And the macro author can determine when those things are required.)

That's the nice thing about establishing best practices rather than laying down an unenforceable law. There would be no protocol police banging on the doors if one decided not to use UUIDs. On the other hand, every time you use a UUID, you know that if "File compress" and "File compress" are both in use, any data loss won't be your fault.

For this to work, I think it would have to be supported by Keyboard Maestro

Why "have to be"? It might be optimistic of me to think that enough other makers of macros might find the idea appealing, but I would consider it much more fanciful to think that an app developer would implement such a system in the application without proven usefulness. Besides, it's surely at that point that practices become less optional.

But I wouldn't want that to be the required storage location, just one that's available if you wanted to use it.

This seems to be the bugbear, and I hope that I have shown that worries about "requirements" are intrinsically misplaced.

Thanks again for engaging and I hope that all illuminates matters a little!

[griffman]

@kevinb

You agree with me that "the Documents folder has long since become a de facto standard location in which many applications will create folders", and you agree with me that it is no longer a good choice for a user to keep their own files (other than in a subfolder). How does this in any way argue against the Documents folder being the best choice for the suggested purpose?

I guess it boils down to understanding what's in this folder structure you're proposing. I read it as "any time your macro saves data, it should use the standardized location and naming." But then you say this folder won't be visited very often, whereas for my macros, those folders are visited a lot. MacroBackerUpper backups are written to a folder and designed to be visited in Finder. The forthcoming forum macro collection tool will let users create a Finder-stored library of downloaded macros, also designed to be visited in Finder.

That's why I'm against the Documents folder: I do not ever open my Documents folder, nor do I ever want to open my Documents folder. Yet unless I'm reading it wrong, that's where you'd have all my macros store their data?

@kevinb

to help facilitate the dissemination of macros in a way that does not require complicated setting up for each and every user.

I think this is also where I'm getting stuck. The setting up part need not be complicated (Open Folder action, set global to the folder the user selected), and puts the user in charge of where their data is stored.

@kevinb

That's 3 macros for the user to change the path in (possibly in multiple places, even though that should not be necessary) and/or worry about setting up synchronisation software (with all its foibles) for up to three separate folders.

If I have a macro that requires data synchronization, then I imagine (having not yet written such a thing) that I would simply require the user to select a cloud drive, and have it deal with all the synch issues. So yes, they'd have to pick the same cloud folder on each Mac for a given macro, but two different macros could use two entirely different cloud providers.

If we use standardized locations, hardcoded into variables in the macro, then we're forcing users to edit the macro to change the path if they don't like the chosen location. And that means any time you update the macro, the user would have to re-enter their preferred path, unless you're storing it in a global ... and if you're doing that anyway, why bother with a standardized location?

I guess I just don't see the need for a standardized location, but that may be due to my non-use of macros that require any sort of data sync. The forthcoming forum macro collector will let users keep the database and library on a cloud folder, but I'm not doing any syncing work, just relying on the cloud provider to keep things in sync.

-rob.

[kevinb]

@griffman

I guess it boils down to understanding what's in this folder structure you're proposing.

I believe so. We all come to new ideas with preconceptions, personal experience, needs, aims and so on, and indeed write from them. So when I write something, I have to try to judge whether I am sufficiently pointing out what is to me the obvious aim without waffling on. Please therefore excuse some more bold print, just to emphasise where I am coming from on all this!

I read it as "any time your macro saves data, it should use the standardized location and naming."

I don't think you would be able to quote anything to back up that impression. My point was perhaps not explicit enough to be clear to you. Here comes the bold print to pick out the salient parts.

@kevinb

I am finalising a revised set of macros that I plan to upload here soon. It will require two or more Macs on a LAN to synchronise not just Keyboard Maestro macros in the usual way, but also to synchronise some small text files.[1] This has prompted me to think further about having a voluntary standard for folder location and structure for dealing with such user files, so that creators of macros have the option of adopting a common practice.

The phrase "such user files" may not be as clear as I thought. I refer to files, such as those "small text files", that contain data to be synchronised between Macs, not all of the user's macro files. I haven't argued for the latter anywhere. I have speculated that there may be other uses for such a common folder standard than file sync, but that is just speculation, and may I emphasise "uses".

But then you say this folder won't be visited very often, whereas for my macros, those folders are visited a lot.

So, yes, you keep them somewhere else. That's right. Do whatever you like. If, however, you subsequently want to release a macro to the forum, and it needs to sync (or otherwise share? Speculation again) data, what's best for everyone: to choose an arbitrary or empty path which the user will have to correct, or to use a common location for data files that? It's a bit like arguing against any shared resource folder. It's good to let people find things automatically or in a standard way, as the default.

Yet unless I'm reading it wrong, that's where you'd have all my macros store their data?

I'm really surprised that you took it in that sense, nope!

I think this is also where I'm getting stuck. The setting up part need not be complicated (Open Folder action, set global to the folder the user selected), and puts the user in charge of where their data is stored.

Should we do the same for other things in the Finder, such as Preferences, or have a Preferences folder as default? "It just works" isn't a bad aim.

If I have a macro that requires data synchronization, then I imagine (having not yet written such a thing) that I would simply require the user to select a cloud drive,

Now you are setting requirements! Leave the preferred sync method to the user. I have been testing with SyncThing, which works across the LAN.

and have it deal with all the synch issues.

What could go wrong? And how much set up would there be for multiple folders?

If we use standardized locations, hardcoded into variables in the macro, then we're forcing users to edit the macro to change the path if they don't like the chosen location.

If you don't do that, you are forcing users to edit the macro whether they like the chosen location or not!

In the forthcoming set of macros that I mentioned (and they do have to be a set rather than just one macro, unless more complexity is desired), you will see that the default path is set in a very accessible location.

And that means any time you update the macro, the user would have to re-enter their preferred path

No it doesn't. That would only be necessary if you decided to have a separate data subfolder for each new version, which obviously you wouldn't.

unless you're storing it in a global

Global variables have nothing to do with it. They are not even synchronised by KM.

... and if you're doing that anyway

I'm not. You'll see soon.

I guess I just don't see the need for a standardized location, but that may be due to my non-use of macros that require any sort of data sync.

Perhaps so. My forthcoming macros depend upon data sync, and I don't think they are unique in that respect, and I suspect we will see more such macros in time.

I'm not doing any syncing work, just relying on the cloud provider to keep things in sync.

That's fair enough, and you come from that perspective. My angle is that I have a set of macros that worked fine on my system but which I want other people to be able to use without jumping through hoops and, while I'm generalising solutions, the same parent file path could be used for any other macro that requires the sharing of data across Macs. That's less work for the end user, one less variable when troubleshooting and less indecision for all—and all for free, because if you don't like the model, you don't have to use it.

[griffman]

@kevinb

The phrase "such user files" may not be as clear as I thought.

Yep, that's what threw me off—I read "user files" as "files that a user creates," not "files a KM macro creates to help the user use the macro."

@kevinb

to choose an arbitrary or empty path which the user will have to correct, or to use a common location for data files that? It's a bit like arguing against any shared resource folder. It's good to let people find things automatically or in a standard way, as the default.

With the understanding that these are actually KM data files, not user files, then most of my issues go away.

@kevinb

Should we do the same for other things in the Finder, such as Preferences, or have a Preferences folder as default? "It just works" isn't a bad aim.

Again, for me that comes down to what types of files these are, and my argument was based on misunderstanding the term "user files." Background files needed by the macro to do what it needs to do can be completely under the macro's control. I often use /tmp for this, because it's writable without any sort of security restrictions.

@kevinb

Now you are setting requirements! Leave the preferred sync method to the user. I have been testing with SyncThing, which works across the LAN.

That's what I meant by requiring a cloud drive, which was probably the wrong term. "Any service that provides data sync across computers." To me, that's cloud, even though SyncThing is cloud-free (and I use it a lot and love it).

@kevinb

If you don't do that, you are forcing users to edit the macro whether they like the chosen location or not!

They don't need to edit the macro at all; they just need to respond to some one-time setup prompts. I hate telling users they have to edit a macro :).

@kevinb

No it doesn't. That would only be necessary if you decided to have a separate data subfolder for each new version, which obviously you wouldn't.

Again, this comes down to my misunderstanding. If they're macro data files, then a hardcoded path is fine. But I thought you were arguing that the path for files the user may create using the macro should be hardcoded. And if that's the way it's done, then they would have to edit the macro with every release, because the new release would have to contain a default path.

@kevinb

Global variables have nothing to do with it. They are not even synchronised by KM.

Which is actually a huge advantage to storing paths in them—they can be unique per Mac for the same macro. But again, user versus background data file is the cause of the disagreement.

@kevinb

That's less work for the end user, one less variable when troubleshooting and less indecision for all—and all for free, because if you don't like the model, you don't have to use it.

Again, I'm not against the model for background data files; I'm very much against it for user files, as in files the user creates with the macro, which is what I thought you were suggesting from the get-go.

Sorry for the misunderstanding;

-rob.

[kevinb]

First off…

@griffman

Sorry for the misunderstanding;

Not at all, it's my responsibility to be clear, and (not to lay the humility on with a trowel! ) to learn from misunderstandings and be clearer or more emphatic in future.

I often use /tmp for this, because it's writable without any sort of security restrictions.

Thanks, yes, using /tmp did occur to me, but several considerations weighed against using it for this.

even though SyncThing is cloud-free (and I use it a lot and love it).

It's good once it's working. This was the first time in quite a while that I had added a folder to SyncThing and there was a problem that took me a while to figure out ("auto accept" was on, and that means the synced folder appears in the home folder on the receiving Mac!). My thinking is that I would rather end users didn't have to muck about with sync software for the data files of every relevant macro. Once should be enough.

They don't need to edit the macro at all; they just need to respond to some one-time setup prompts. I hate telling users they have to edit a macro :).

It's a good point. Currently the accompanying Readme tells the user they can delete the default file path from the variable box and drag in its replacement from the Finder. I'm not sure there will be much difference in ease of use, but I shall experiment with Prompt for File instead.

[kevinb]

Perhaps things would have been clearer if I had posted the macro set first as a "proof of concept". It will follow fairly soon.

[kevinb]

The title probably misled you before you got to the body of the text! I have tweaked it!

[kevinb]

@griffman

There are times you have to use a UUID, but that should be an exception, not a rule.

I now agree that UUIDs in the folder names would not be necessary. After all, if any macro tried to create a folder and it already existed, the macro should handle that situation in a safe way, I suppose... If a UUID were ever needed in this sort of situation, the folder could be uniquely identified by, say, the presence of an invisible (dot) file containing a UUID.

[kevinb]

@kevinb

Global variables have nothing to do with it. They are not even synchronised by KM.

I must correct myself there. What I meant was that runtime changes to global variables are not synchronised.

Anyhow… I am, after all, dropping sync from the forthcoming V. 1 of the macros that I mentioned.

The sync itself was no trouble (other than being something for the user to set up) but it is only needed as part of a solution that far exceeds my own simple use case. Fiddling with all the "moving parts" has been leading towards a convincingly flexible system that is more than I need, and which I have no reason to presume that anyone else needs either! It's become more like app prototyping than making a macro and has been a real time sink (no pun intended), albeit a fun and educational one (I do like Keyboard Maestro ). Like us all, I have more important things to do, and I'm therefore stripping the system back down to two little macros that are good enough for me. I will post them in the forum soon.

I do still like the idea of ~/Documents/Keyboard Maestro User/Keyboard Maestro User Sync/ as a standard location for macro resources that are to be synced, and when I experiment further with such things, I'll use that unless better ideas appear.