There's your problem. You're testing for a button that doesn't exist.
Try targeting the checkbox with this macro. You can add a condition within the resultant script.
Here's an example using the Preview app:
tell application "System Events"
tell process "Preview"
-- Navigate to the checkbox
set theCheckBox to checkbox "Auto Rotate" of group 1 of group 2 of scroll area 2 of splitter group 1 of sheet 1 of window 1
-- Click the checkbox only if it's checked
if value of theCheckBox is 1 then
click theCheckBox
end if
end tell
end tell
Obviously you'll need to replace "Auto Rotate" of group 1 of group 2 of scroll area 2 of splitter group 1 of sheet 1 of window 1
with the relevant element reference for your macro.