Discovered this by happenstance and tested further because this can't be by design, right? It doesn't happen every time, but it's consistently reproducible for me. Can anyone else reproduce this behaviour?
I have a system defined keyboard shortcut ⌘F1 to open it and usually close it with ESC. This has been the most reliable for reproducibility; however, the behaviour has also presented when manually opening the window from the menu bar with the cursor and closing the window by clicking its X-button.
System Information
Macbook Air, M2 2022
macOS 15.4.1
Keyboard Maestro 11.0.4