Should leave what on the Clipboard unchanged?
IMO this is a Clipboard/KM interaction issue -- whether that's Clipboard only making plain text available to KM or KM only taking plain text from the Clipboard is moot. But it makes sense -- how would the encode action include the Clipboard metadata that marks the text as RTF and not just some plaintext with RTF formatting symbols in it?
You'll see the same if you copy some formatted text, run this in Terminal:
pbpaste -Prefer rtf | base64 | base64 -d | pbcopy
...then paste into a TextEdit rich text document.
If you want to base64 some rich text, the easiest way would be write it to a temporary text file then encode that. If you want to skip the file then take a look at @ComplexPoint's Clipboard Viewer macro for an idea on how to extract the RTF code from the Clipboard for encoding (and I have no idea how you'd decode without using a temp file...).