I don't have that app installed so can't say for sure, but you may try using one of the "Set Radio Button" actions. If that doesn't work, maybe the "Press a Button" action would work (just depends on if/how Keyboard Maestro recognizes the UI element).
Another idea would be to see if the UI of Finale will allow you to TAB through the options for mouse-free selection. While a bit clunky / not elegant, if that works, you could have Keyboard Maestro tab over the needed number of times until X Notehead is selected in the right column and then hit ENTER.
If none of those ideas work, you might try the "Click at Found Image" action. 
Some of these ideas were pulled from this old post here in case that's helpful:
This thread may also prove helpful in your quest: