Some v7 endpoints won't activate

[GreenEnvy22 6]

We have a handful of computers that refuse to activate, not sure why. The vast majority of our machines are activated fine, but we haven't found any pattern to the ones that won't. These were all machines activated on EES 6.5-6.6, and we upgraded them to v7. 

After upgrading, they report they are not activated on the client end. In ESMC, they do not show up in "non-activated security product" filter in computers. If I open the details of a computer that is affected, ESMC shows green checkmark and "everything is fine", however there is no license key attached to the client, see screenshot 1.

If we create an activation job, pick our license, and target the machine, the job tried to run next time the client checks in, but it fails, see screenshot 2.

I've setup some brand new computers and they all activate fine, so it's not a general activation issue, just affecting a handful of machines.  I can't find any more detailed logs on the ESMC end to see why it's failing. We have plenty of seats available on our key.

Found these lines in clients trace logs:

2018-09-17 11:48:49 Error: CReplicationModule [Thread 1ab4]: InitializeConnection: Initiating replication connection to 'host: "eset.xxxxxxx.xxx" port: 2222' failed with: Request: Era.Common.Services.Replication.CheckReplicationConsistencyRequest on connection: host: "eset.xxxxxxx.xxx" port: 2222 with proxy set as: Proxy: Connection: :3128, Credentials: Name: , Password: ******, Enabled:0, EnabledFallback:1, failed with error code: 14, error message: OS Error, and error details: 
2018-09-17 11:48:49 Warning: CReplicationModule [Thread 1ab4]: InitializeConnection: Not possible to establish any connection (Attempts: 1)
2018-09-17 11:48:49 Error: CReplicationModule [Thread 1ab4]: InitializeFailOverScenario: Skipping fail-over scenario (stored replication link is the same as current)
2018-09-17 11:48:49 Error: CReplicationModule [Thread 1ab4]: CAgentReplicationManager: Replication finished unsuccessfully with message: InitializeConnection: Initiating replication connection to 'host: "eset.xxxxxxx.xxx" port: 2222' failed with: Request: Era.Common.Services.Replication.CheckReplicationConsistencyRequest on connection: host: "eset.xxxxxxx.xxx" port: 2222 with proxy set as: Proxy: Connection: :3128, Credentials: Name: , Password: ******, Enabled:0, EnabledFallback:1, failed with error code: 14, error message:  OS Error, and error details: Replication details: [Task: CReplicationConsistencyTask, Scenario: Automatic replication (REGULAR), Connection: eset.xxxxxxx.xxx:2222, Connection established: false, Replication inconsistency detected: false, Server busy state detected: false, Realm change detected: false, Realm uuid: 356af7a2-24c8-42d7-ac8e-061bb6fe9e5c, Sent logs: 0, Cached static objects: 71, Cached static object groups: 10, Static objects to save: 0, Static objects to delete: 0, Modified static objects: 0]
2018-09-17 14:58:49 Error: CReplicationModule [Thread 1ab4]: SendRequestAndHandleResponse: Rpc message response AUTHENTICATION_FAILURE (Token status: TOKEN_INVALID) -> Request new session token and resend replication request
2018-09-17 14:58:50 Warning: CReplicationModule [Thread 1ab4]: GetAuthenticationSessionToken: Received failure status response: TEMPORARILY_UNAVAILABLE (Error description: session token temporarily unavailable, device is not enrolled yet)

Thoughts?

[MartinK 378]

Unfortunately more detailed activation failure is not traced into ERA/ESMC logs -> error you posted indicates there were problems in AGENT-to-ESMC connection, caused by hostname resolution failure (maybe some network outage) during that time, but in case activation task is executed and results are reported, this issue resolved itself automatically.

Regarding activation: my recommendation is to attempt to activate product manually, i.e. use endpoint UI to do so. It will provide more details and will possibly offer solution, especially in case failure is caused by wrong configuration (for example there might be used HTTP proxy that is not accessible).

[GreenEnvy22 6]

It's not a name resolution issue causing activation, the agents are reporting into ESMC fine. We're having issues both with clients internal to our office (on internal DNS) and remote clients (on public DNS).

I just tried activating one of our problem machines again through ESMC, this one is a server running file security. Activation failed again. I see the attached error in the 'events' log on the client.

If I manually activate on client using the key, it works.

I also found a workaround, I found the ermm utility, and enabled that by policy. I then used it to push out our key to all the affected users, and the majority of them have now activated. There are a bunch of remaining ones but they are computers that haven't checked in for several hours so are offline. I expect almost all of them to be fixed up by next week.

If anyone else runs into this, once you enable ERMM, the command line is: eRmm.exe start activation --key abc-123-def-456-ghi 

Replace with your key. I liked this option as it didn't require the user do anything, and didn't require us giving out our key. Remember to disable ermm again afterwards if you don't use it for 3rd party integration to prevent a security risk.

 

[Marcos 5,070]

On a troublesome computer, please enable the following in the setup -> tools -> diagnostics:

1, Advanced network protection logging
2, Advanced licensing logging

Then try to activate Endpoint manually by entering the license key in the activation window. Then disable logging, gather logs with ESET Log Collector and provide me with the generated archive.

If manual activation works, do the same on another machine but try to activate it via an activation task sent from ESMC.

[MartinK 378]

19 hours ago, GreenEnvy22 said:

If anyone else runs into this, once you enable ERMM, the command line is: eRmm.exe start activation --key abc-123-def-456-ghi

In case activation using key or manual activation using EBA/ELA account works, problem might be with so called deployment tokens. They are part of activation task configuration and are extracted from your license on ESMC -> have you tried to create new activation task? Or you are using task that exists for longer time? Asking because there has been major upgrade of licensing backend recently (~10 days ago) that might have impacted your license. It is possible that wrong/obsolete deployment token is delivered to endpoint and thus failing to activate.

[GreenEnvy22 6]

Interesting Martin.

I can't test this as the command I used above got all our computers fixed up.

It was something odd as ESMC didn't see these clients as being unactivated (that dynamic group didn't show all these clients), but it also didn't show an assigned license for them.