FYI (11 months later), this approach will work most of the time, but if any other Raycast window is open, it will return a false positive. Specifically, it will falsely return true when the search window is NOT open if either
I spent a lot of time trying to find a reliable way to verify that it's the SEARCH window that's open, and so far, the best solution I've found is to look for a Raycast window that has the same height as the search bar. I've attached it here in case other people come looking for answers in the future.
Here's the latest version I use. Unlike the previous version, this one will also work if you use the default window mode instead the compact mode (or if you use the compact mode but have expanded the window to the full size because you're using an extension or have started searching for something). I use it as a subroutine, so that's what I'm uploading here. The dimensions for all the relevant windows are also included in a comment so you can tweak the script if/as needed.