Give this a go, @noisneil. Long-form and arrow-headed for readability.
It seems the trick is to select the row containing the checkbox before you click the checkbox...
tell application "System Events"
tell application process "Installer"
tell window 1
tell group 1
tell splitter group 1
tell group 1
tell scroll area 1
tell outline 1
repeat with eachRow in (get every row)
set cbName to name of checkbox 1 of eachRow
if cbName contains "AAX" or cbName contains "VST" then
select eachRow
tell checkbox 1 of eachRow
click
end tell
end if
end repeat
end tell
end tell
end tell
end tell
end tell
end tell
end tell
end tell