The issue is with the server certificate being to strict/not matching the hostname used in the agent installer.
First check the server peer certificate which hostnames are allowed (listed in the column HOST).
If you want to allow additional hostnames/IPs you can create a new server certificate containing all allowed hostnames seperated by space, comma, or semicolon. Or you can leave the default (*) to allow all hostnames.
Afterwards you have to assign this new certificate in the server settings and restart the Virtual Appliance (or the eraserver-Service) for the change to take effect.
You can also perform a repair installation and change the hostname. Please be aware that the hostname has to match eaxctly.