GreenEnvy22

Followup #2. That broke again at 3am today. Not sure why, there were no windows updates, no server reboots. Agents just stopped reporting in several hours after restoring it. The server has tons of space, but I still saw an error message in the mysql logs about not being able to write a SERERNAME-bin.00001 file because there was no space, but the drive has 25GB free. Something seems messed up with the database. I backed it up and restored to a new server, but the new server complains the DB isn't configured correctly. I tried restoring it to a new appliance, but it says that version number (blank) isn't compatible with 9.1.xxxx. I did a repair install of server, which completed, but did not fix the issue. The mySQL section though gave me no issues. Next I tried uninstalling, leaving the DB in place, and reinstalling. Now I got the message about database not being configured properly. I'm next going to restore the VM from backups again, and try backing up that DB before it breaks, and restore to new machine, to see if it works.

As a followup, we restored the VM from a backup made the day before the issue began, and agents all started reporting in again. I didn't figure out for sure what caused it, but it may have been a database corruption due to low space. The windows logs showed some windows updates failing due to space issues. There was still about 6GB free when I looked at it today, but perhaps during an update that had got critically low. I gave the restored VM more space to avoid the issue going forward.

We just noticed an issue with our Eset Protect server. None of our 500+ agents have reported in since Nov 1st. As far as we're aware, no changes have been made to the server in weeks, other than it maybe auto-updating itself. Some agents are inside the firewall, others outside, but none of them are reporting in. I checked the agent logs and see messages like: 2022-11-07 14:32:24 Error: CReplicationModule [Thread 22c0]: CAgentReplicationManager: Replication finished unsuccessfully with message: Deadline Exceeded (code: 4) for request Era.Common.Services.Replication.GetStaticObjectsRequest (id: 2cef4920-d643-4a11-ae85-ba4bf536c88f) on connection to 'host: "eset.DOMAIN.org" port: 2222', Task: CStaticObjectMetadataTask, Scenario: Automatic replication (REGULAR), Connection: eset.DOMAIN.org:2222, Connection established: true, Replication inconsistency detected: true, Server busy state detected: false, Realm change detected: false, Realm uuid: 356af7a2-24c8-42d7-ac8e-061bb6fe9e5c, Sent logs: 0, Cached static objects: 162, Cached static object groups: 10, Static objects to save: 0, Static objects to delete: 0, Modified static objects: 0 2022-11-07 14:32:24 Error: CSystemConnectorModule [Thread 22d4]: CDeviceSnapshotWinLoaderUtils::ReadDiskDriveIDs: calling funtion=[CreateFile] for volume=[\\?\Volume{2c7bdea2-5dad-11ed-9f17-047bcb29463b}] failed, error=[5] 2022-11-07 14:35:54 Error: CReplicationModule [Thread 22c0]: InitializeConnection: Replication connection problem: Deadline Exceeded (code: 4) for request Era.Common.Services.Replication.CheckReplicationConsistencyRequest (id: c00aabdc-43e7-4ebd-8d85-9f34f9871cd0) on connection to 'host: "eset.DOMAIN.org" port: 2222' 2022-11-07 14:35:54 Warning: CReplicationModule [Thread 22c0]: InitializeConnection: Not possible to establish any connection (Attempts: 1) [RequestId: c00aabdc-43e7-4ebd-8d85-9f34f9871cd0] 2022-11-07 14:35:54 Error: CReplicationModule [Thread 22c0]: InitializeFailOverScenario: Skipping fail-over scenario (stored replication link is the same as current) [RequestId: c00aabdc-43e7-4ebd-8d85-9f34f9871cd0] 2022-11-07 14:35:54 Error: CReplicationModule [Thread 22c0]: CAgentReplicationManager: Replication finished unsuccessfully with message: Replication connection problem: Deadline Exceeded (code: 4) for request Era.Common.Services.Replication.CheckReplicationConsistencyRequest (id: c00aabdc-43e7-4ebd-8d85-9f34f9871cd0) on connection to 'host: "eset.DOMAIN.org" port: 2222', Task: CReplicationConsistencyTask, Scenario: Automatic replication (REGULAR), Connection: eset.DOMAIN.org:2222, Connection established: false, Replication inconsistency detected: false, Server busy state detected: false, Realm change detected: false, Realm uuid: 356af7a2-24c8-42d7-ac8e-061bb6fe9e5c, Sent logs: 0, Cached static objects: 162, Cached static object groups: 10, Static objects to save: 0, Static objects to delete: 0, Modified static objects: 0 2022-11-07 14:35:54 Error: CSystemConnectorModule [Thread 22d4]: CDeviceSnapshotWinLoaderUtils::ReadDiskDriveIDs: calling funtion=[CreateFile] for volume=[\\?\Volume{2c7bdea2-5dad-11ed-9f17-047bcb29463b}] failed, error=[5] And Status.htm: Error: Replication connection problem: Deadline Exceeded (code: 4) for request Era.Common.Services.Replication.CheckReplicationConsistencyRequest (id: c00aabdc-43e7-4ebd-8d85-9f34f9871cd0) on connection to 'host: "eset.DOMAIN.org" port: 2222' Task: CReplicationConsistencyTask Scenario: Automatic replication (REGULAR) Connection: eset.DOMAIN.org:2222 Connection established: false Replication inconsistency detected: false Server busy state detected: false Realm change detected: false Realm uuid: 356af7a2-24c8-42d7-ac8e-061bb6fe9e5c Sent logs: 0 Cached static objects: 162 Cached static object groups: 10 Static objects to save: 0 Static objects to delete: 0 Modified static objects: 0 All replication attempts: 285 On the Protect server (Windows VM, running Protect 9.1.1295, and web console 9.1.292.0): 2022-11-07 17:46:14 Error: NetworkModule [Thread 14d8]: Protocol failure for session id 9589, error:Receive: NodSslWriteEncryptedData: Internal error in the underlying implementations. 2022-11-07 17:47:02 Error: CReplicationModule [Thread 14a4]: RpcGetStaticObjectsHandler: Failed to load static objects list for peer c4d5ec23-dbe2-4563-bea8-81eeed0cffd2 with error: Object b5a1a0b4-1e4e-4a4d-99f5-dd5f0178cf28 was not found (LoadUser: User state deserialization failed) 2022-11-07 17:47:16 Error: CReplicationModule [Thread 19ec]: RpcGetStaticObjectsHandler: Failed to load static objects list for peer 09bfa9a4-c73c-408f-89d1-5836d8803d7e with error: Object b5a1a0b4-1e4e-4a4d-99f5-dd5f0178cf28 was not found (LoadUser: User state deserialization failed) 2022-11-07 17:47:17 Error: NetworkModule [Thread 1ba8]: Verify user failed for all computers: grd-tvcart1.DOMAIN.lan: NodVerifyCertificateChain failed: NodVerifyTrustResult: 42, NVT_NotTrusted, X509ChainStatus: 0x1, X509CSF_NotTimeValid, certificate: [Subject='CN=Agent certificate for host *, OU=IT, O=DOMAIN, L=MY_CITY, S=MI, C=US', Issuer='CN=Certificate Authority for DOMAIN ESET, OU=IT, O=DOMAIN, L=MY_CITY, S=GR, C=US', NotBefore=2015-Jul-23 04:00:00, NotAfter:2020-Jul-24 03:59:59, Serial=REMOVED, SHA256=REMOVED, SubjectKeyIdentifier=51fee92de54bc02db2e2805ddefade8259e20adb, AuthorityKeyIdentifier=REMOVED],10.2.12.61: NodVerifyCertificateChain failed: NodVerifyTrustResult: 42, NVT_NotTrusted, X509ChainStatus: 0x1, X509CSF_NotTimeValid, certificate: [Subject='CN=Agent certificate for host *, OU=IT, O=DOMAIN, L=MY_CITY, S=MI, C=US', Issuer='CN=Certificate Authority for DOMAIN ESET, OU=IT, O=DOMAIN, L=MY_CITY, S=GR, C=US', NotBefore=2015-Jul-23 04:00:00, NotAfter:2020-Jul-24 03:59:59, Serial=REMOVED, SHA256=REMOVED SubjectKeyIdentifier=REMOVED, AuthorityKeyIdentifier=REMOVED] 2022-11-07 17:47:17 Error: NetworkModule [Thread 1ba8]: Receive: NodSslWriteEncryptedData: Internal error in the underlying implementations., ResolvedIpAddress:10.2.12.61, ResolvedHostname:grd-tvcart1.DOMAIN.lan, ResolvedPort:51383 2022-11-07 17:47:17 Error: NetworkModule [Thread 1ba8]: Protocol failure for session id 9753, error:Receive: NodSslWriteEncryptedData: Internal error in the underlying implementations. 2022-11-07 17:47:30 Error: CReplicationModule [Thread 19dc]: RpcGetStaticObjectsHandler: Failed to load static objects list for peer a28b25cd-9ef8-4248-82a4-8296824b0409 with error: Object b5a1a0b4-1e4e-4a4d-99f5-dd5f0178cf28 was not found (LoadUser: User state deserialization failed) 2022-11-07 17:47:33 Error: NetworkModule [Thread 14d8]: Error code:121;The semaphore timeout period has expired; SessionId:9582 2022-11-07 17:47:43 Error: CReplicationModule [Thread 15f0]: RpcGetStaticObjectsHandler: Failed to load static objects list for peer 10d2c848-ecca-41f3-98ea-7476baeb7495 with error: Object b5a1a0b4-1e4e-4a4d-99f5-dd5f0178cf28 was not found (LoadUser: User state deserialization failed) 2022-11-07 17:47:57 Error: CReplicationModule [Thread df0]: RpcGetStaticObjectsHandler: Failed to load static objects list for peer 6d5de554-9d37-4e1c-a9bf-1c35cf24c887 with error: Object b5a1a0b4-1e4e-4a4d-99f5-dd5f0178cf28 was not found (LoadUser: User state deserialization failed) 2022-11-07 17:48:06 Error: NetworkModule [Thread 14d8]: Error code:10053;An established connection was aborted by the software in your host machine; SessionId:9729 2022-11-07 17:48:06 Error: NetworkModule [Thread 1bac]: Error in SendCallback: An established connection was aborted by the software in your host machine;Error code: 10053; SessionId:9729 2022-11-07 17:48:10 Error: CReplicationModule [Thread 1cf4]: RpcGetStaticObjectsHandler: Failed to load static objects list for peer 67156841-d3c4-4fbb-ae50-17943ffc0eaf with error: Object b5a1a0b4-1e4e-4a4d-99f5-dd5f0178cf28 was not found (LoadUser: User state deserialization failed) 2022-11-07 17:48:25 Error: CReplicationModule [Thread 1a4c]: RpcGetStaticObjectsHandler: Failed to load static objects list for peer 02a35abd-da18-4be2-a974-60dd0dd1c882 with error: Object b5a1a0b4-1e4e-4a4d-99f5-dd5f0178cf28 was not found (LoadUser: User state deserialization failed) 2022-11-07 17:48:38 Error: CReplicationModule [Thread 1508]: RpcGetStaticObjectsHandler: Failed to load static objects list for peer fd6bd6f9-88e1-4e46-8d31-932dadde7720 with error: Object b5a1a0b4-1e4e-4a4d-99f5-dd5f0178cf28 was not found (LoadUser: User state deserialization failed) 2022-11-07 17:49:00 Error: NetworkModule [Thread 14d8]: Error code:121;The semaphore timeout period has expired; SessionId:9802 2022-11-07 17:49:18 Error: NetworkModule [Thread 1bac]: Error code:121;The semaphore timeout period has expired; SessionId:9674 2022-11-07 17:50:12 Error: NetworkModule [Thread 1bb0]: Error code:121;The semaphore timeout period has expired; SessionId:9992 I see some old certificates mentioned there that expired in 2020. None of those are in use anywhere in the protect iterface, all ours show valid until 2025 for server and agents. The servers status.html shows all green. It does note the certificate is using an obsolete signature algorithm, but is still listed as OK. Scope Time (UTC) Text Server performance 2022-Nov-07 18:01:30 Overall performance status is: OK Synchronization status 2022-Nov-07 16:45:53 Idle - OK Last synchronization of Licenses was successful at 2022-Nov-07 16:45:53 (1 seat pools synchronized) Peer certificate 2022-Nov-07 16:45:45 OK Server peer certificate with subject 'CN=Server certificate for host eset.DOMAIN.org, OU=IT, O=DOMAIN, L=MY_CITY, S=Michigan, C=US' issued by 'CN=Certificate Authority for MYDOMAIN ESET, OU=IT, O=MYDOMAIN, L=MY_CITY, S=GR, C=US' with serial number 'REMOVED' is signed with obsolete signature algorithm and it is still valid for next 30 days Replication Throttling 2022-Nov-07 16:45:44 Current throttling state is: OK Max logs count is : 14000 Max logs KB is : 1048576 KB Max agents is : 280 NetworkModule statistics 2022-Nov-07 18:01:45 NetworkModule per minute statistics Number of SocketContainers: 167 Log duration statistics 2022-Nov-07 18:01:32 Logs Replication statistics for last 60 minutes, updated at least every 15 seconds PERFORMANCE_SERVER_EVENT: Total time: 601 miliseconds, count: 121, one process time: 4 QOS_DATABASE_EVENT: Total time: 277 miliseconds, count: 60, one process time: 4 QOS_NETWORK_EVENT: Total time: 208 miliseconds, count: 60, one process time: 3 TASK_CLIENT_EVENT: Total time: 86 miliseconds, count: 106, one process time: 0 FUNCTIONALITY_COMPUTER_STATUS: Total time: 35 miliseconds, count: 1, one process time: 35 LIVEGRID_STATUS: Total time: 35 miliseconds, count: 8, one process time: 4 APPLIEDPOLICYPRODUCTS_STATUS: Total time: 29 miliseconds, count: 1, one process time: 29 FUNCTIONALITY_PRODUCTS_STATUS: Total time: 28 miliseconds, count: 2, one process time: 14 FUNCTIONALITY_PROBLEMSDETAILS_STATUS: Total time: 25 miliseconds, count: 2, one process time: 12 EPNS_STATUS: Total time: 25 miliseconds, count: 1, one process time: 25 Total time: 1seconds, Total count: 362 Performance details 2022-Nov-07 18:01:30 Detailed performance statistics: I/O reads: 0 KB/s I/O writes: 1 KB/s I/O others: 1027 KB/s Logs latency: 0 s Pending logs: 0 Database size: 335 MB Received logs 2022-Nov-07 17:37:11 Received logs statistics: Received in last minute: 125 (2.08 /s) Received in last hour: 125 (0.03 /s) History of received logs in last hour (group by 10 seconds): Replication Statistics 2022-Nov-07 17:03:39 Logs Replication statistics per minute Number of throttled logs replications: 0 Number of succesful logs replications: 0 NetworkModule 2022-Nov-07 16:45:45 Network module configuration: Socket timeout is set to 21600 seconds Reverse DNS resolving is enabled Dns servers update interval is set to 1200 seconds Product 2022-Nov-07 16:45:29 Product install configuration: Product type: Server Product version: 9.1.1295.0 Product locale: en_US Performance Indicator Value Up time 01:16:22 Memory private usage 1184 MB Available physical memory 1842 MB Any thoughts here before I submit a ticket to support?

We had this issue too, after updating our on prem Protect, then replacing Oracle JDK with Amazon Corretto v17. Option 1, adding those 2 extra lines, worked for us as well. Edit: I should note the instructions are talking about Apache9, but we're on Apache7 and it was identical steps.

That didn't work with the cert that specified the server name, but it did work with the wildcard cert (just *) I've since gone back and edited the policy to use the wildcard, and it seems to be working now. Is there a known issue with using the hostname in the cert?

We've been getting notices that our peer certificates were going to expire soon (next week), so today I created a new server cert, and a new agent cert, in ESMC. The server cert I assigned in server settings, rebooted the VM (windows PC), and that looks like it's working fine. Both used the built in ESET Cert authority, which is still valid for 5 years. The certs are setup for hostname eset.mydomain.com, I also tried just leaving them as *, but neither worked. For agent cert, duplicated our existing agent policy, and setup the change of certificate there. The existing agent policy did not have a certificate specified at all, as clients got this info from the config.ini during installation, or it was just pushed to them from ESMC. I then assigned this new policy to a couple of test machines. Each of them reports in one more time, and I can see they now are assigned the new cert in ESMC, however they are no longer reporting into ESMC. I also tried creating an agent live installer, and installed agent from the batch file, but the agent is never able to connect. In their agent logs, I see messages like: 2020-07-15 14:33:12 Error: AuthenticationModule [Thread 2ee8]: DeviceEnrollmentCommand execution failed with: Request: Era.Common.Services.Authentication.RPCEnrollmentRequest on connection: host: "eset.mydomain.com" port: 2222 with proxy set as: Proxy: Connection: :3128, Credentials: Name: , Password: ******, Enabled:0, EnabledFallback:1, failed with error code: 14, error message: Connect Failed, and error details: 2020-07-15 14:33:12 Warning: CReplicationModule [Thread 23bc]: GetAuthenticationSessionToken: Received failure status response: TEMPORARILY_UNAVAILABLE (Error description: session token temporarily unavailable, device is not enrolled yet) 2020-07-15 14:33:12 Error: CReplicationModule [Thread 23bc]: InitializeConnection: Initiating replication connection to 'host: "eset.mydomain.com" port: 2222' failed with: GetAuthenticationSessionToken: Failed to fetch device session token in time 2020-07-15 14:33:12 Warning: CReplicationModule [Thread 23bc]: InitializeConnection: Not possible to establish any connection (Attempts: 1) 2020-07-15 14:33:12 Error: CReplicationModule [Thread 23bc]: InitializeFailOverScenario: Skipping fail-over scenario (stored replication link is the same as current) 2020-07-15 14:33:12 Error: CReplicationModule [Thread 23bc]: CAgentReplicationManager: Replication finished unsuccessfully with message: InitializeConnection: Initiating replication connection to 'host: "eset.mydomain.com" port: 2222' failed with: GetAuthenticationSessionToken: Failed to fetch device session token in timeReplication details: [Task: CReplicationConsistencyTask, Scenario: Automatic replication (OUT_OF_ORDER), Connection: eset.mydomain.com:2222, Connection established: false, Replication inconsistency detected: false, Server busy state detected: false, Realm change detected: false, Realm uuid: 356af7a2-24c8-42d7-ac8e-061bb6fe9e5c, Sent logs: 0, Cached static objects: 0, Cached static object groups: 0, Static objects to save: 0, Static objects to delete: 0, Modified static objects: 0] I've tried rebooting the client machines, but it didn't help. Any thoughts on what is going wrong?

Description: Support LDAP or RADIUS login for ESMC Administrators Detail: We'd love to see ESMC support the ability to login via LDAP or RADIUS, instead of just active directory and local users. We want to enable 2FA/MFA to protect ESMC, but trying to avoid the sprawl of apps needed on our phones, with every vendor pushing their own app for MFA. If LDAP or Radius were supported for logging into ESMC, it would open up the option for lots of other MFA services to work, like DUO.

We recently renewed out ESET subscription for 2 more years (Endpoint Security 7). We pushed out the new license via ESMC. Of our roughly 600 computers, we have about 30 who are having an issue. On the clients, they get warnings their license is expired. Investigating further on the server, I see the same license ID listed twice, once with the old expiry date, once with the new one, see screenshot. Other clients who don't have the expiry warning don't show the old license, just the current one. How can I fix this? I tried just making a new task to assign the current license, but while it runs to completion, it doesn't affect the computers. I believe I somehow need to remove that old one.

Interesting Martin. I can't test this as the command I used above got all our computers fixed up. It was something odd as ESMC didn't see these clients as being unactivated (that dynamic group didn't show all these clients), but it also didn't show an assigned license for them.

It's not a name resolution issue causing activation, the agents are reporting into ESMC fine. We're having issues both with clients internal to our office (on internal DNS) and remote clients (on public DNS). I just tried activating one of our problem machines again through ESMC, this one is a server running file security. Activation failed again. I see the attached error in the 'events' log on the client. If I manually activate on client using the key, it works. I also found a workaround, I found the ermm utility, and enabled that by policy. I then used it to push out our key to all the affected users, and the majority of them have now activated. There are a bunch of remaining ones but they are computers that haven't checked in for several hours so are offline. I expect almost all of them to be fixed up by next week. If anyone else runs into this, once you enable ERMM, the command line is: eRmm.exe start activation --key abc-123-def-456-ghi Replace with your key. I liked this option as it didn't require the user do anything, and didn't require us giving out our key. Remember to disable ermm again afterwards if you don't use it for 3rd party integration to prevent a security risk.

We have a handful of computers that refuse to activate, not sure why. The vast majority of our machines are activated fine, but we haven't found any pattern to the ones that won't. These were all machines activated on EES 6.5-6.6, and we upgraded them to v7. After upgrading, they report they are not activated on the client end. In ESMC, they do not show up in "non-activated security product" filter in computers. If I open the details of a computer that is affected, ESMC shows green checkmark and "everything is fine", however there is no license key attached to the client, see screenshot 1. If we create an activation job, pick our license, and target the machine, the job tried to run next time the client checks in, but it fails, see screenshot 2. I've setup some brand new computers and they all activate fine, so it's not a general activation issue, just affecting a handful of machines. I can't find any more detailed logs on the ESMC end to see why it's failing. We have plenty of seats available on our key. Found these lines in clients trace logs: 2018-09-17 11:48:49 Error: CReplicationModule [Thread 1ab4]: InitializeConnection: Initiating replication connection to 'host: "eset.xxxxxxx.xxx" port: 2222' failed with: Request: Era.Common.Services.Replication.CheckReplicationConsistencyRequest on connection: host: "eset.xxxxxxx.xxx" port: 2222 with proxy set as: Proxy: Connection: :3128, Credentials: Name: , Password: ******, Enabled:0, EnabledFallback:1, failed with error code: 14, error message: OS Error, and error details: 2018-09-17 11:48:49 Warning: CReplicationModule [Thread 1ab4]: InitializeConnection: Not possible to establish any connection (Attempts: 1) 2018-09-17 11:48:49 Error: CReplicationModule [Thread 1ab4]: InitializeFailOverScenario: Skipping fail-over scenario (stored replication link is the same as current) 2018-09-17 11:48:49 Error: CReplicationModule [Thread 1ab4]: CAgentReplicationManager: Replication finished unsuccessfully with message: InitializeConnection: Initiating replication connection to 'host: "eset.xxxxxxx.xxx" port: 2222' failed with: Request: Era.Common.Services.Replication.CheckReplicationConsistencyRequest on connection: host: "eset.xxxxxxx.xxx" port: 2222 with proxy set as: Proxy: Connection: :3128, Credentials: Name: , Password: ******, Enabled:0, EnabledFallback:1, failed with error code: 14, error message: OS Error, and error details: Replication details: [Task: CReplicationConsistencyTask, Scenario: Automatic replication (REGULAR), Connection: eset.xxxxxxx.xxx:2222, Connection established: false, Replication inconsistency detected: false, Server busy state detected: false, Realm change detected: false, Realm uuid: 356af7a2-24c8-42d7-ac8e-061bb6fe9e5c, Sent logs: 0, Cached static objects: 71, Cached static object groups: 10, Static objects to save: 0, Static objects to delete: 0, Modified static objects: 0] 2018-09-17 14:58:49 Error: CReplicationModule [Thread 1ab4]: SendRequestAndHandleResponse: Rpc message response AUTHENTICATION_FAILURE (Token status: TOKEN_INVALID) -> Request new session token and resend replication request 2018-09-17 14:58:50 Warning: CReplicationModule [Thread 1ab4]: GetAuthenticationSessionToken: Received failure status response: TEMPORARILY_UNAVAILABLE (Error description: session token temporarily unavailable, device is not enrolled yet) Thoughts?

I did a bit more testing on my end. Tried restarting apache, did not help. Restarted ESMC service, agents reporting back in again. Will talk to support on Monday, should be broken again by then.

Talked to ESET support today, they made a new agent check-in policy for every 20 minutes (we had it at 10), will see if it helps. Also earlier today when I generated the logs, I got the same "Deadline Exceeded", message jimmy reported for last synchronization.

This issue has re-occured for us, many of our machines not reporting in Since Aug 30, but a bunch are still reporting in. Tried installing on a brand new computer and it's not reported in so far. Will open a case with ESET.

I think I am having the same issue. About a week ago, I upgraded the ESET ERA 6 server to the new ESMC 7, that went fine, clients continued to check in. It was an in place upgrade on Windows server 2016. On Aug 24, I upgraded the agents on all my Windows servers and most of the workstations (both in office and remote), using the ESMC component upgrade tool. Didn't push any endpoint upgrades as of yet, just agent. Today I looked in ESMC, and I see only 1 of my servers is still reporting to ESMC, and less than half of my workstations reporting.. All the others stopped on Aug27, around 9:20pm local time. I looked at task execution histories and don't see anything happening on the 27th. No windows patches were pushed out that day, and this is in 2 different sites. I ran agent diagnostic logs on one of the windows servers, and no errors reported there. It shows a last authentication with todays date, but in ESMC it shows Aug27 for this machine. Last authentication 2018-Aug-30 12:13:14 Enrollment OK Peer certificate 2018-Aug-30 12:13:08 OK Agent peer certificate with subject 'CN=Agent certificate for host *,---removed---' issued by 'CN=Server Certification Authority, C=US' with serial number '---removed---' is and will be valid in 30 days Product 2018-Aug-30 12:13:03 Product install configuration: Product type: Agent Product version: 7.0.553.0 Product locale: en_US Replication security 2018-Aug-30 12:13:15 OK Remote host: ---eset.removed--- Remote product: Server I tried uninstalling agent on one of the windows servers, and it got stuck at stopping the service. Windows services console showed it as "stopping", and still was 10 minutes later. Killed it by PID, and it restarted but still didn't report in. At this point, I decided to reboot the ESMC server, and now it appears I have all my servers and clients starting to report in. So not sure what would cause ESMC to glitch in such a way that some clients still were recorded at reporting in, but others weren't. I'll monitor and see if the issue re-occurs.