Thanks for response @peternlewis - it's probably got to do something w/ the fact that it isn't signed properly. When I run codesign -v on it I get this:
mdbraber:~ $ codesign -v ~/Applications/Fastmail.app/
/Users/mdbraber/Applications/Fastmail.app/: invalid Info.plist (plist or signature have been modified)
Unfortunately it also gives the same error when trying to run the script through osascript from the commandline. I'll have to see if there are any other alternatives available...