ERA v7 Mac agents failing to check-in

[brandobot 2]

This has been an issue in our environment for the past year. We were told from our support rep that v7 agent will remedy this issue.

 

On the machine losing connection to the ESET Security management center, I ran nc -z *server redacted*2222
Connection to *server redacted* port 2222 [tcp/rockwell-csp2] succeeded!

 

This confirms that the client can reach the server; however, connection fails. The workaround fix has been to re-install the ESET agent to get it working again.

 

CDynamicGroupsModule	2017-Aug-28 16:19:04	Agent is matching 2 dynamic group templates from 20
CDynamicGroupsModule	2017-Aug-28 16:19:04	Agent is matching dynamic group template with uuid 00000000-0000-0000-7014-000000000003 and version 1
CDynamicGroupsModule	2017-Aug-28 16:19:04	Agent is matching dynamic group template with uuid eac7ca9f-d3ab-4200-9905-5a464f10a368 and version 2
CDynamicGroupsModule	2017-Aug-28 16:19:04	Evaluating 15 dynamic groups
AutomationModule	2017-Aug-28 16:19:09	Trigger: Tick ALLOWED [UUID=00000000-0000-0000-7006-000000000001, TYPE=REPLICATION].
AutomationModule	2017-Aug-28 16:19:09	Task: Executing task [UUID=00000000-0000-0000-7005-000000000001, TYPE=Replication, CONFIG=scenarioType: REGULAR linkData { dataLimit: 1024 isDisabled: false connections { host: "*SERVER REDACTED*" port: 2222 } }].
CReplicationModule	2017-Aug-28 16:19:09	CReplicationManager: Processing client replication task message
CReplicationModule	2017-Aug-28 16:19:09	CReplicationManager: Failed to start replication, connection for replication link '00000000-0000-0000-7007-000000000001' (Automatic replication (REGULAR)) is already pending
CReplicationModule	2017-Aug-28 16:19:09	CReplicationManager: Queuing replication task to be executed after current replication is finished
SchedulerModule	2017-Aug-28 16:19:09	Received message: GetRemainingTimeByUserDataRequest
CSystemConnectorModule	2017-Aug-28 16:19:19	Retrieving network configuration information
CSystemConnectorModule	2017-Aug-28 16:19:19	StatusLog_NETWORK_ADAPTERS_STATUS: "Rows":[]
CSystemConnectorModule	2017-Aug-28 16:19:19	StatusLog_NETWORK_IPADDRESSES_STATUS: "Rows":[]
CSystemConnectorModule	2017-Aug-28 16:19:19	StatusLog_NETWORK_IPGATEWAYS_STATUS: "Rows":[]
CSystemConnectorModule	2017-Aug-28 16:19:19	StatusLog_NETWORK_IPDNSSERVERS_STATUS: "Rows":[]
SchedulerModule	2017-Aug-28 16:19:19	Received message: RegisterSleepEvent
NetworkModule	2017-Aug-28 16:19:19	Forcibly closing sessionId:43, isClosing:0
NetworkModule	2017-Aug-28 16:19:19	Sending message: ConnectionFailure
NetworkModule	2017-Aug-28 16:19:19	Removing session 43
NetworkModule	2017-Aug-28 16:19:19	Closing connection , session id:43
NetworkModule	2017-Aug-28 16:19:19	The connection will be closed due to timeout. Resolved endpoint is NULL
CReplicationModule	2017-Aug-28 16:19:19	CReplicationManager: Replication (network) connection to 'host: "1XXXXX" port: 2222' failed with: Operation timed out

Edited September 24, 2018 by brandobot

[j-gray 40]

That's disappointing. We had the same issue with Mac agents, reportedly caused by 'unclean' shutdown. Only fix was to remove the agent via ARD and reinstall.

We were also told by support that this was fixed in the latest version. However, due to multiple and various issues noted here in the forums, I have not been in a hurry to "upgrade".

Do you have an existing support case for this?  If you don't mind providing it I can refer it to our support folks.

[brandobot 2]

On 9/25/2018 at 8:00 AM, j-gray said:

That's disappointing. We had the same issue with Mac agents, reportedly caused by 'unclean' shutdown. Only fix was to remove the agent via ARD and reinstall.

We were also told by support that this was fixed in the latest version. However, due to multiple and various issues noted here in the forums, I have not been in a hurry to "upgrade".

Do you have an existing support case for this?  If you don't mind providing it I can refer it to our support folks.

Case #103515. It's been open for nearly a year.  I've been told "our new version will fix this" at least 2-3 times. I've wasted countless hours installing their beta versions that they promised would fix the issue. 

Our current workaround is to uninstall and reinstall as well. Do you have a good way of determining which Macs are losing connection to the ESET console?

Are you using Jamf? How long have you been experiencing this issue? As far as I can tell, half of our Mac population (~350)  already lost its' connection in the past 1.5 weeks since we upgraded to v7 agent.

Edited October 4, 2018 by brandobot

[j-gray 40]

1 hour ago, brandobot said:

Case #103515. It's been open for nearly a year.  I've been told "our new version will fix this" at least 2-3 times. I've wasted countless hours installing their beta versions that they promised would fix the issue. 

Our current workaround is to uninstall and reinstall as well. Do you have a good way of determining which Macs are losing connection to the ESET console?

Are you using Jamf? How long have you been experiencing this issue? As far as I can tell, half of our Mac population (~350)  already lost its' connection in the past 1.5 weeks since we upgraded to v7 agent.

Thanks for the info.

We had a case (#39682) for this issue, opened 05/31/2017. I was also told it was resolved in this latest release. They didn't want to wait for me to upgrade and test, so they closed the case on 08/22/2018. I'll loop back with the support engineer and see where it goes. I spent countless hours testing beta agents, as well.

Unfortunately, we don't have a good way to find unmanaged Macs (or PCs for that matter). I occasionally run agent install tasks on unmanaged clients in the ERA console. If they return a successful install but still show unmanaged, it's a good indication the agent is broken again. We don't have JAMF. I use ARD to run the agent uninstall script, then re-run the agent install task via ERA. It's very tedious.

I'll let you know if I hear back from support. Good luck!

[MartinK 384]

8 hours ago, j-gray said:

New AGENTS should be automatically backuping their databases and restoring them once main database is detected as corrupted. Could you please verify that "broken" AGENT actually created backup:

/Library/Application Support/com.eset.remoteadministrator.agent/data.db
/Library/Application Support/com.eset.remoteadministrator.agent/data.db.bak

 

and possibly also verify that both data.db and data.db.bak are valid SQLite databases? Following commands should verify integrity:

sqlite3 /Library/Application Support/com.eset.remoteadministrator.agent/data.db "PRAGMA integrity_check"
sqlite3 /Library/Application Support/com.eset.remoteadministrator.agent/data.db.bak "PRAGMA integrity_check"

of both databases.

[j-gray 40]

9 hours ago, MartinK said:

New AGENTS should be automatically backuping their databases and restoring them once main database is detected as corrupted. Could you please verify that "broken" AGENT actually created backup:

Based on issues noted in these forums, I'm holding off upgrading to v7.5 at least until the first service release is available.

[brandobot 2]

12 hours ago, MartinK said:

and possibly also verify that both data.db and data.db.bak are valid SQLite databases? Following commands should verify integrity:

sqlite3 /Library/Application Support/com.eset.remoteadministrator.agent/data.db "PRAGMA integrity_check"
sqlite3 /Library/Application Support/com.eset.remoteadministrator.agent/data.db.bak "PRAGMA integrity_check"

of both databases.

On 9/25/2018 at 8:00 AM, j-gray said:

That's disappointing. We had the same issue with Mac agents, reportedly caused by 'unclean' shutdown. Only fix was to remove the agent via ARD and reinstall.

We were also told by support that this was fixed in the latest version. However, due to multiple and various issues noted here in the forums, I have not been in a hurry to "upgrade".

Do you have an existing support case for this?  If you don't mind providing it I can refer it to our support folks.

See below for screenshots. I ran your commands and the database integrity reports OK. I also attached the status.html where it shows the failure and last connection.

ESET console shows Sept 28, 2018 was the last connect day. I ssh'd into the machine and verified I can reach the eset server on port 2222 by doing "nc -z *IP* 2222" 

If I remove and re-install the Agent, it'll register just fine in the ESET Remote Administrator console. 

I wrote an extension attribute in Jamf to determine machines out of compliance. So far I've scanned 300 machines and 75 have the word "error" in the status.html, and haven't successfully replicated within the last 3 days.

 

[MartinK 384]

Thanks for details. AGENT is running, but it is not able to connect to ESMC with error "Failed to create subchannel"". This is new error for me as I don;t think it was reported since ESMC release. Does restarting AGENT service or whole machine resolves this issue, at least temporarily? In case not, would it be possible to capture network traffic using tcpdump/wireshark and provide it to me via PM?

[brandobot 2]

5 hours ago, MartinK said:

Thanks for details. AGENT is running, but it is not able to connect to ESMC with error "Failed to create subchannel"". This is new error for me as I don;t think it was reported since ESMC release. Does restarting AGENT service or whole machine resolves this issue, at least temporarily? In case not, would it be possible to capture network traffic using tcpdump/wireshark and provide it to me via PM?

This is what we've seen:

- Machine initially connects after installation of ERA.

- After an undetermined amount of time, usually anywhere from a couple days to a couple weeks, the machine will no longer check-in

- Removal and re-install of Agent and the machine will immediately show up again

I have not tried restarting the agent service yet. ESET support just showed me how to do this today. I will find another machine that is broken, try to restart the agent service to see if it gets it re-connected. If it does not, I can use wireshark to start capturing the log while restarting the service.

I don't want to rule out the network, but I don't think this is the cause because I've had test machines that never leave the office lose connection before and require a re-install of the agent to fix.