KM web server uses a self-signed certificate. This is fine, and it's very easy to add the certificate to keychain and trust it.
The problem is that keyboard maestro has a subject name of "Keyboard Maestro", whereas it needs to use a URL. Is there any chance of adding an alternative subject name to it - perhaps keyboardmaestro.local or something?
That way, even though it's self-signed, it would be trivial to trust the certificate. At the moment, I don't think there's a good way to do that.
Definitely seems like a question that @peternlewis would have to answer.
It does not use the domain name for the subject name because it does not know the domain name, and any IP address would likely change.
However, you can trust the certificate, simply connect (which you can do by clicking the Connect button in the Keyboard Maestro Web preferences), then show the certificate, and in the Trust section, adjust it to Always Trust, and then OK, and then Visit Site, and you'll need to enter your (admin?) password.