I have seen cases where the screenshot just has black areas on the screen. You can turn on a debug mode with:
defaults write com.stairways.keyboardmaestro.engine FindImageWriteImages -bool YES
And restart the Keyboard Maestro Engine and then and after the failed macro you can look in the /tmp folder for the /tmp/target.tiff and /tmp/screen.tiff files and we can see if something weird is happening with them.