This AppleScript will set the flag of the selected message to red:
tell application "Mail"
set msg to item 1 of (get selection)
tell msg
set flag index to 0
end tell
end tell
Change 0 to 1 or 2 or 3 or 4 or 5 or 6 for different colors.
Change 0 to -1 to remove the flag
Also check out this post by Chris.