Agent Upgrade and Migration: Got new Proxy but not Server?

[Staj 5]

We use SCCM for software deployment and I'm trying to come up with a reliable and consistent way to move our ERA6 clients to a brand new ESMC7 server with fresh ESET client software v7 installs etc. but the Agent is being a pain, we can't uninstall it cleanly no matter what we do (tried removing password, self defense etc. nothing works) but we noticed that just installing the new agent in place does actually working.

Our new ESMC 7 server (esmc.redacted.local in this example) uses a proxy (that's completely unknown to the ERA6 infrastructure) whilst our old ERA6 server (antivirus.redacted.local in this example) doesn't.

We utilise an install_config.ini with the MSI as part of our install, after we install the new Agent 7.2.1266.0 it disconnects from ERA6 but never connects to ESMC7, we notice in the trace.log the following:

2020-10-27 12:15:09 Error: CReplicationModule [Thread 12bc]: CAgentReplicationManager: Replication finished unsuccessfully with message: InitializeConnection: Initiating replication connection to 'host: "antivirus.redacted.local" port: 2222' failed with: GetAuthenticationSessionToken: Failed to fetch device session token in timeReplication details: [Task: CReplicationConsistencyTask, Scenario: Automatic replication (REGULAR), Connection: antivirus.redacted.local:2222, Connection established: false, Replication inconsistency detected: false, Server busy state detected: false, Realm change detected: false, Realm uuid: REDACTED-GUID-1, Sent logs: 0, Cached static objects: 0, Cached static object groups: 0, Static objects to save: 0, Static objects to delete: 0, Modified static objects: 0]
2020-10-27 12:15:15 Error: AuthenticationModule [Thread 16cc]: DeviceEnrollmentCommand execution failed with: Request: Era.Common.Services.Authentication.RPCEnrollmentRequest on connection: host: "antivirus.redacted.local" port: 2222 with proxy set as: Proxy: Connection: esmc.redacted.local:3128, Credentials: Name: eset, Password: ******, Enabled:1, EnabledFallback:1, failed with error code: 14, error message: Connect Failed, and error details: 
2020-10-27 12:15:15 Warning: CReplicationModule [Thread 12bc]: GetAuthenticationSessionToken: Received failure status response: TEMPORARILY_UNAVAILABLE (Error description: session token temporarily unavailable, device is not enrolled yet)

So it's still trying to contact the old ERA6 server but through the new ESMC7 proxy? So the new install_config.ini was partially applied? Why wasn't the new host name applied too?

Edited October 27, 2020 by Staj

[Staj 5]

The new Settings Protection Password was applied too.

[MartinK 378]

5 hours ago, Staj said:

So it's still trying to contact the old ERA6 server but through the new ESMC7 proxy? So the new install_config.ini was partially applied? Why wasn't the new host name applied too?

From what you described it seems that AGENT is not able to connect to new ESMC, and thus attempting to connect to ERA that was used previously - this is expected behavior, used to recover in case of accidental breaking changes. Once AGENT will successfully connect to new ESMC, it will stop doing this.

Unfortunately from provided section of logs, it is not clear why connection to new ESMC fails. Could you please check it also for connection to new server? It should be clearly visible, that in regular interval, AGENT will try to connect to new ESMC, and in case it fails, it will immediately try to use previously working connection configuration.

[Staj 5]

@MartinKI'm afraid not, there is no other mention of any other connection. Those messages are the first of it's kind in the log.

[Camilo Diaz 2]

Can your clients telnet to your new ESMC server on port 2222? If not then there's highly likely a firewall stopping the connection. (assuming the new server is working as expected). What about new clients? Can they connect to the new server?

Edited October 28, 2020 by Camilo Diaz

[Staj 5]

@Camilo DiazThese are all identically configured systems on the same subnet, if I manually uninstall the Agent 6 and install Agent 7 it works fine, I've done about 12 tests now on separate systems, it's very consistent. Network based IDS/IPS show nothing interesting. There is no host-based firewall. If it really was being blocked, why isn't there a log entry of the failure in trace.log like there is now for every attempt to the old one?

[Staj 5]

@MartinK I've left the latest test computer on for many hours in this state now and not once has it tried to connect to new ESMC instance nor did it seem to ever try (according to trace.log).

So how do I fix this problem? Is there a way to forcefully apply configuration (again)? ecmd or something? What do I do from here?

[MartinK 378]

12 hours ago, Staj said:

@MartinK I've left the latest test computer on for many hours in this state now and not once has it tried to connect to new ESMC instance nor did it seem to ever try (according to trace.log).

So how do I fix this problem? Is there a way to forcefully apply configuration (again)? ecmd or something? What do I do from here?

Unfortunately I have no more idea - in previous (older) version, we had an issue, that during upgrade, configuration changes were not applied, and thus after upgrade, second execution of installer was required in order to perform upgrade with reconfiguration.

As you mentioned that policy was used to reconfigure AGENT to connect to new server, how was this policy actually applied? Asking as in case AGENT is not connecting, there is a chance it has not even received new policy.

[Staj 5]

Quote

We utilise an install_config.ini with the MSI as part of our install,

@MartinK

We installed it right over the top, install_config.msi along with Agent v7 MSI was used.

[Staj 5]

Quote

Unfortunately I have no more idea - in previous (older) version, we had an issue, that during upgrade, configuration changes were not applied, and thus after upgrade, second execution of installer was required in order to perform upgrade with reconfiguration.

I'm noticing this too, repairing the MSI Agent v7 install restores functionality. I think I'm just going to have to utilise a SCCM Task Sequence and PowerShell to perform this migration...