Using buttons in Prompt for User Input for things other than OK and Cancel

Correct. The first button (which is displayed right-most) is the default button (pressing Return will trigger it), and the second button is the cancel button (pressing Escape will trigger it).

Add "/." to the name. I don't know where you are seeing "/.Cancel", the example shows "Help/H"

That said, the right most button should be the default button. If you really want Cancel, Top and Bottom, then you should have buttons:

Top/ (default) or Top/T (T)
Bottom/B (B)
Cancel/. (Escape or Command-.)

Note that if the dialog has text fields, then it would be Command-T and Command-B, but if the dialog has no text fields, it is just the letters by themselves.

It's not a bug, the default keys are based on position not whether the Cancel option is on.

And the right0-most button should never be Cancel on a Mac, it should always be the default button.