How to List All Windows of One App That Are Open in All Desktops

On Big Sur at least, @CJK's very helpful (application prototype extension) code lets us generate a JSON listing of windows for named applications in this kind of Keyboard Maestro pattern:

Windows listed for named applications.kmmacros (4.0 KB)


Expand disclosure triangle to view JS Source
(() => {
    "use strict";

    // Rough sketch by Rob Trew,
    // using CJK's Application prototype extension code.

    // Ver 0.2 (fixed a Width ⇄ Height swap)

    const main = () => {
        const
            appNames = Application("Keyboard Maestro Engine")
            .getvariable("appNames");

        return (
            extendPrototype(),
            JSON.stringify(
                lines(appNames).map(
                    k => ({
                        appName: k,
                        windowList: Application(
                            k.trim()
                        ).getWindowList()
                    })
                ), null, 2
            )
        );
    };

    // ----------------------- JXA -----------------------

    // Using code by CJK

    const extendPrototype = () =>
        Application.prototype.getWindowList = function() {
            ObjC.import("CoreGraphics");

            const
                pids = Application("com.apple.systemevents")
                .processes
                .whose({"bundleIdentifier": this.id()})
                .unixId();

            return ObjC.deepUnwrap(
                ObjC.castRefToObject(
                    $.CGWindowListCopyWindowInfo(16, 0)
                )
            )
            .filter(
                e => pids.includes(e.kCGWindowOwnerPID) &&
                e.kCGWindowLayer === 0 &&
                e.kCGWindowAlpha === 1 &&
                e.kCGWindowStoreType === 1

            )
            .map(x => ({id: x.kCGWindowNumber,
                name: x.kCGWindowName,
                bounds: [
                    "X", "Y",
                    "Width",
                    "Height"
                ].map(z => x.kCGWindowBounds[z])})
            );
        };


    // --------------------- GENERIC ---------------------

    // lines :: String -> [String]
    const lines = s =>
    // A list of strings derived from a single string
    // which is delimited by \n or by \r\n or \r.
        Boolean(s.length) ? (
            s.split(/\r\n|\n|\r/u)
        ) : [];


    // MAIN ---
    return main();
})();


2 Likes

Thanks CP/Rob (what do you prefer to be called, @ComplexPoint ?),

This looks like a great start, not only for my project but as a trailhead into JXA. I’ve started learning AppleScript to use with KBM and I have some familiarity with shell scripts, I’ve used SED with KBM to massage text lists, but I’ve had no inkling that JXA was available within KBM. Thanks for your great Wiki articles on the subject. I’m in a similar position to where @JMichaelTX was:

I used Acrobat JavaScript 20 years ago to manipulate comments in PDF files, but I have no experience using it in browsers.

Again, thanks, I look forward to trying this out.

Nice! I like your formatting (indentation) much more than mine.

I've just noticed that, for whatever reason, I've weirdly ordered the ordinates of the bounds property as X, Y, Height, Width] instead of `[X, Y, Width, Height. I'll make the correction to my post.

1 Like

Thanks ! Switched above too, now (in the macro and the listing).

formatting

( It's just the ESLint and Prettier ESLint extensions in VSCode )

I got a chance to download an install "Windows listed for named applications.kmmacros" and what I get when I run it is:
image

I'm running on Mojave (on the verge of upgrading to Catalina). Would that make a difference?

How do I capture the rest of that error message? In a shell, I could redirect STDERR to a file. Should I have the shell execute the JavaScript to capture that, or is there a KBM-native way?

Ah HA! I found the logs folder /Users/amohr/Library/Logs/Keyboard Maestro/ and the Engine.log file.

Here's the result of running the macro:

2022-08-07 18:00:44 Execute macro “Windows listed for named applications” from trigger Editor
2022-08-07 18:00:45 Action 11176861 failed: Execute a JavaScript For Automation failed with script error: text-script:1137:1174: execution error: Error on line 44: Error: Ref has no type (-2700)
2022-08-07 18:00:45 Execute a JavaScript For Automation failed with script error: text-script:1137:1174: execution error: Error on line 44: Error: Ref has no type (-2700). Macro “Windows listed for named applications” cancelled (while executing Execute JavaScript For Automation).

It identifies line 44.

Here's lines 43-47 of the function text:

            return ObjC.deepUnwrap(
                ObjC.castRefToObject(
                    $.CGWindowListCopyWindowInfo(16, 0)
                )
            )

I'm not sure what "Ref has no type (-2700)" means, but I'll try running the script without the "strict" setting.

No difference. It's probably not just a JavaScript error then, I guess it's coming from ObjC. Stack Overflow has nothing for that error text.

Any suggestions?

It looks like it's taking issue with whatever gets returned by the function $.CGWindowListCopyWindowInfo(16, 0). Thus I'd want to know what value it's returning.

What gets returned by the following code:

ObjC.import('CoreGraphics');
let wInfo = $.CGWindowListCopyWindowInfo(16, 0);
wInfo;

// ObjC.castRefToObject(wInfo);

Subsequently, if you remove the two slashes at the beginning of the last line (this uncomments the line, which at present doesn't get executed), what does the code return now ?

[ I'd suggest running this in Script Editor: you can create a blank document and set the language to JavaScript at the top of the document (there's a dropdown menu). Also, from the bottom of the document, in the status bar, there are three icons on the left. Click the third one to reveal the event log. The reason to use Script Editor here is that we're dealing with rather opaque values that KM won't be able to display natively. The ObjC.castRefToObject function is integral to converting an opaque value into one that, firstly, JXA-ObjC can understand, before this gets unwrapped into something JS can understand, before it's manipulated to produce an array of dictionaries that KM should be able to understand. ]

1 Like

As a first try, I opened Script Editor, opened a new File window, set it to JavaScript, and pasted in the lines above. I opened the Events log and ran it (with the Play button). That produced 113038 characters of output in a single "line". The line starts out with "$([" and the matching "])" is at the end, 113035 characters later.

So I tried uncommenting that fourth (fifth) line above, When I did I got:
image

And in the Events pane I got:

Error -2700: Script error.

That looks familiar.

So, any idea what's going on? Why did you ask for that line? It's not in the original function, is it?

Back at the test script with the line commented out, all 113038 characters of it --
Within that overall ([...]), it looks like there are repeated blocks of

$({"kCGWindowLayer": ... }),

So for readability, I'll add linebreaks after all those commas, using a VIM global.

:g/}),$({"kCGWindowLayer":/s//}),^M$({"kCGWindowLayer":/g

That results in 337 lines, one per kCGWindowLayer.

Any idea what I should be looking for in there?

Aside:
97 of those lines include

"kCGWindowOwnerName":$("Logi Options Daemon")

That's for my bluetooth mouse, which isn't even on right now. There is only one entry for that in the Activity Monitor (All Processes), so why so many entries here?

Thanks!

I’m curious where you got those parameters, (16, 0).

The Apple doc for CGWindowListCopyWindowInfo describes them as:

static var optionAll: CGWindowListOption

var kCGNullWindowID: CGWindowID { get }

Where did you get numeric values?

A brief footnote or marginal scribble:

If automation is a significant part of what you need from macOS, then it may be prudent to lag a little (or even significantly) behind the newest versions.

osascript support has a very low priority at Apple these days, and every new version of macOS inevitably degrades or breaks it just a little, even if only inadvertently.
Sometimes fixes are made retrospectively, but typically with a lag of several months, rather than a few weeks.

Better to install a macOS version at the end of its life-cycle (when it's as fully cooked as it's ever going to get) than at the rough and creaking start of its life-cycle.

In short – macOS upgrades do have a cost (especially, now, for automation) and it's always worth pausing to seriously consider whether it's really worth paying that cost, and how exactly your work might benefit from it.

That's kind of the position that I'm in. I'm currently running Mojave, which Apple no longer supports, planning to upgrade soon to Catalina, a single-step upgrade. I'm looking for all the things that I can anticipate Catalina breaking -- I've removed all my 32-bit software. I'll miss being able to OCR directly in a PDF using Acrobat Pro X, but there are workarounds.

1 Like

IMO that's a little too at the end of it's life-cycle!

The OS should be well-baked by the time the next full version is publicly released. While I don't (can't!) stop our users jumping to the latest'n'greatest as soon as they feel like it, we generally recommend "production" machines hold back for a year, so they'd have upgraded to Catalina when Big Sur came out, Big Sur when Monterey dropped etc. (Assuming the hardware/software was compatible with the new OS, obvs.)

That keeps things reasonably up to date with a manageable risk of things going pear-shaped. More importantly for us it minimises the number of machines that aren't receiving security updates -- Catalina will probably go end-of-life a couple of months after you've installed it.

1 Like

I agree with your advice, but we have an odd situation here in that the problematic JXA/JSObjC is arising in earlier versions of macOS (Mojave), whereas it works seamlessly in Monterey. It's unexpected, to say the least.

1 Like

Thank you for the description of the output. You actually told me exactly what I needed to know here. That output is perfect, so we know that there's no problem with the actual function that retrieves the window data, $.CGWindowListCopyWindowInfo().

So the problem is arising when we try to take the opaque core foundation C-data types and recast it into more usable Objective-C foundation types.

I'm going to have to experiment tonight to see whether i can bind the opaque data type to something else that can act as an intermediary to bridge the values between C and JS. I just need to find out exactly how the $.CGWindowListCopyWindowInfo() function is declared.

I have to admit, something feels a bit off but i cant put my finger on it, but my attention is also divided right now, and im not at a computer.

Ha, this is because of how we conventionally use the term "window" to refer to a very specific type of element that we interact with, namely a framed, titled, content viewer that acts as a moveable container for other types of elements (documents, file icons, or whatever). But this is semantics, and in the language of Core Graphics and a session manager, a window is...well, actually, everything is a window: buttons, icons, labels with text, ... These are fundamentally all the same thing, with certain parts visible or invisible, and certain functionality added in or taken out. But they're all windows, including a lot that are completely invisible but are stilk very much present. I suspect your bluetooth device windows make up some of the invisible contingent, as well as the ones that will form part of the bluetooth menu bar menu (which is also a window).

If you examine my first script, those values were represented by constants that I was accessing the same way as I access the function. The 0 value is the value returned by $.kCGNullWindowID, representing a null value. 16 is the value of the constant $.kCGWindowListExcludeDesktopElements, which excludes desktop icons, the desktop itself, and some other "windows" that can't be seen. If you opt to include these, the list is absurdly massive, if you consider a single desktop icon is, itself, comprised of two or three windows—one for the icon, one for the label, and i think maybe one two group those two elements together.

So bear with me, and I'll get back to you with my success or failure.

I found the line in that script. Thanks.

With that line dumping all the window info, my own approach from here would be to massage that data dump with other tools:

  1. Break the data into separate lines, as I did above, probably doing it in a pipeline rather than with VIM.
  2. Filter for the lines that identify the given parent app, e.g. "kCGWindowOwnerName":$("TextEdit")
  3. Extract the name of the window from the lines that have one, e.g. "kCGWindowName":$("Xera Fluids Ordering, etc.rtfd")
  4. Format that for output.

I may be missing something, but that could be a minimal answer to my question in the OP. When I originally asked that question, it was just the the first part of my quest. I additionally eventually want to know which Desktop Workspace is each of those windows in.

I don't find anything about Desktops / Workspaces represented in the output of CGWindowListCopyWindowInfo. Is that tracked totally separately by Mission Control or is it in data that would be returned by a different first parameter (not 16, $.kCGWindowListExcludeDesktopElements)? Seems to me it might be a part of the "Desktop Elements" data.

Any ideas where to find that out?

Thanks!

You're welcome to use whatever method you wish, but there are good reasons not to leap straight into using other tools as a workaround for something. For one, other forum users will come by this post one day and it'll be benefit most people not only to see how the problems we encountered were solved, but also that it was solved and allows them to utilise the solution without third-party tools that they might not have nor might not wish to install. Secondly, debugging a solution that uses a number of different technologies is harder. Lastly, having a solution that is self-contained from beginning-to-end will usually (but not always) be the most performant.

My original contribution when I came across this post was simply to highlight a C API that enumerates windows regardless of which desktop (space) they are on. I later expanded this into a JXA function as the API call returns an excess of information (most of which is useless to most people), and of a data type a lot of people wouldn't know what to do with.

It wasn't intended to solve the exact problem you put forward, but I also knew (or was very confident in my belief) that there was no other means (that doesn't require installation of third-party tools) of obtaining window data for windows that weren't on the currently-active desktop. As far as I know, this remains to be true, and if anything, more so today than it was as Apple continues to tighten its security protocols.

I knew there was something off last time I replied. I couldn't put my finger on it, or must have been half asleep, but the fact that you're getting a data dump at all was all we needed. When I started to explain about data types and casting, etc., I was sort of doing this on autopilot, partly because what I was writing holds true when thinking about the problem in the context of my own system. On Monterey, this:

ObjC.import('CoreGraphics');
$.CGWindowListCopyWindowInfo(16, 0);

returns an opaque [Object Ref] that subsequently needs to passed to ObjC.castRefToObject() in order to get the "data dump" that you were already getting without this. This is obviously what changed between Mojave/Catalina and later version of macOS.

Therefore, the solution is simply to remove the call to ObjC.castRefToObject() so that this line:

return  ObjC.deepUnwrap( ObjC.castRefToObject(
        $.CGWindowListCopyWindowInfo(16, 0) )).filter(e =>

is now simply this:

return  ObjC.deepUnwrap($.CGWindowListCopyWindowInfo(16, 0)).filter(e =>

The rest is just regular JavaScript, so can be left alone.

I'm such an idiot.

Exploring this further, it doesn't look like information about which desktop a window is on is returned by the API. To double check, here's a list of all possible keys that can appear for any individual window (𝑡he first part of the list are Required Window List Keys that will always be included, and the latter part of the list are Optional Window List Keys that may or may not be included):

$.kCGWindowNumber
$.kCGWindowStoreType
$.kCGWindowLayer
$.kCGWindowBounds
$.kCGWindowSharingState
$.kCGWindowAlpha
$.kCGWindowOwnerPID
$.kCGWindowMemoryUsage

// $.kCGWindowWorkspace — deprecated in macOS 10.8
$.kCGWindowOwnerName
$.kCGWindowName
$.kCGWindowIsOnscreen
$.kCGWindowBackingLocationVideoMemory

I wonder whether $.kCGWindowWorkspace used to hold the specific information about which desktop a given window was on, but it no longer works. However, what will provide a little more help for you is the key $.kCGWindowIsOnscreen, which does at least inform you whether a given window is on the currently active desktop or not.

Here it is implemented in the previous function declaration:

Application.prototype.getWindowList = function() {
        ObjC.import('CoreGraphics');
        let pids = Application('com.apple.systemevents').processes
                   .whose({'bundleIdentifier':this.id()}).unixId();
        return  ObjC.deepUnwrap($.CGWindowListCopyWindowInfo(16,0))
                .filter(e => pids.includes(e.kCGWindowOwnerPID)  &&
                                           e.kCGWindowLayer==0   &&
                                           e.kCGWindowAlpha==1   &&
                                          !e.kCGWindowIsOnscreen &&
                                           e.kCGWindowStoreType==1)
                .flatMap(x => [ { id     : x.kCGWindowNumber,
                                  name   : x.kCGWindowName,
                                  bounds : [ 'X', 'Y',
                                             'Width',
                                             'Height' ].map(z => 
                                           x.kCGWindowBounds[z]) } ]);
}

Application('Script Editor').getWindowList();

That will return the list of windows that are on desktops other than the currently active one. To return only those windows on the currently active desktop, remove the ! from the beginning of !e.kCGWindowIsOnscreen.

Um, for not getting it perfect the first time? I don’t think so.

Thanks for ALL your help. I’ll explore this new context.

The approach I was talking about was using a shell script and built-in tools, not third-party tools. I often find it easier to follow the UNIX philosophy of building a one step at a time pipeline instead of trying to do the entire reformatting, filtering, etc. process inside something like JavaScript.

Yes, the Apple doc says that’s exactly what it was for— and it only worked from 10.5 to 10.8. That’s probably what HyperSpace (which did a lot of what I’m trying to do) was dependent upon.

You're most welcome. I've been away from the KBM forum for a long while, so it's a nice way to get back into it.

That's not unusual. It's the nature of the imperative programming style, in which one operation is performed coded in a line, followed by a second operation coded on a second line, etc. JavaScript is perfectly capable of being written this way. The code you've seen above leverages JS's multi-faceted nature, in being a traditionally-imperative language, but having incorporated a lot of functional syntax and attributes that you're seeing a little of in those code snippets. I was trained in my formative years in functional languages before moving onto imperative ones (bar some toying around with BASIC and C back in school), and it's a lot easier transitioning in that direction. But the majority of professional developers (which I haven't been for a long while, hence I'm quite rusty) all started (and largely remained) with imperative languages, so the slight uphill climb to get one's head around a more functional approach one worth doing and one you won't be doing alone.

If you would prefer the JavaScript function to output a result as a string instead of a JavaScript array of objects, that is perfectly doable. It sounds like the JS object array, whilst containing most of the pertinent information you might need/want, might not be something you're comfortable handling and extracting specific bits of information out of...? Would something like a CSV-style output be more useful for you ?

Most of my personal scripting stuff nowadays is done on the command line. I don't know what it is about shell scripting, but I do find it very addictive. Most of my heavier coding is done in Swift now, but Swift lends itself quite reasonably as a scripting language as well, so I actually do most of this from the command line too.

What shell do you use ? zsh, presumably ?

When I run the following script from @ccstone

by putting it inside a KBM Execute Applescript action, I get a list delimited with the combination comma-space. I need to change that comma-space delimiter.

Unfortunately, I know I have filenames that have that comma-space character combination in them, so I can't break the resulting list at that character combination without risking breaking a filename sometimes, and also I'd have to remember to never, ever use that in filenames, which I can't count on either. It is, after all, both legal and useful.

How can I change the list delimiter in AppleScript to be something other than comma-space?

I did some digging and tried the following:

image

tell application "TextEdit"
	set AppleScript's text item delimiters to "|"
	set winNameList to name of every window
	set AppleScript's text item delimiters to "" -- ALWAYS SET THEM BACK
end tell
return winNameList

to attempt to set the delimiter to the pipe symbol (|) and it had no effect, the KBM window still showed comma-space between filenames.

Ideally, I'd use a newline, but I thought I'd do testing with a visible character first.

I then tried to see if I could change the delimiter as the list was built, using:

tell application "TextEdit"
	set winNameList to ""
	repeat with theItem in name of every window
		set winNameList to winNameList & "||" & theItem
	end repeat
end tell
return winNameList

But Script Editor complained, "Can’t make item 1 of name of every window of application "TextEdit" into type Unicode text." while highlighting the variable theItem at the end of the concatenation.

That tells me that it appears to be interpreting my repeat with theItem in name of every window properly, and the problem is in building the list. But what else is that item if it isn't Unicode text?

Any ideas?