Agent install successful but client not communicating with console

[j-gray 35]

Agent installs successfully on a handful of Windows 7 workstations. I can see the services are running on each client. However, after hours/days, the client still shows in the console (via AD sync) as unmanaged/unknown.

 

I've had ESET running for close to a year now, and agent installs and policies are generally working properly, so these appear to be outliers.

 

How can I troubleshoot the issue remotely?  I can access directories, services, etc. remotely, but cannot log on directly to launch any local gui's, etc.

 

Any suggestions appreciated.

[MartinK 378]

Agent installs successfully on a handful of Windows 7 workstations. I can see the services are running on each client. However, after hours/days, the client still shows in the console (via AD sync) as unmanaged/unknown.

 

I've had ESET running for close to a year now, and agent installs and policies are generally working properly, so these appear to be outliers.

 

How can I troubleshoot the issue remotely?  I can access directories, services, etc. remotely, but cannot log on directly to launch any local gui's, etc.

 

Any suggestions appreciated.

 

I would suggest to get C:\ProgramData\ESET\RemoteAdministrator\Agent\EraAgentApplicationData\Logs\status.html from broken client computer. It should contain connection error.

[j-gray 35]

Thanks for the reply.

 

I looked at the status.html file, which gave me a chuckle;  everything is green and 'OK', server and certificate are correct, etc.  But all the info was generated in October of 2015.

 

The only other info I found was in the trace log in the same folder:

 

Error: Service [Thread 73c]: SQLiteException

: sqlite3_prepare_v2 failure

Error: Application UnregisterCECallback: failed with 7013

 

The other two workstations exhibiting similar behavior show the following error:

 

Error: CEssConnectorModule [Thread 984]: HIPS: Installed version of EES or HIPS is unsupported (Code: 14)

 

Any suggestions on best approach to resolve these, and what may have caused the issues in the first place?

 

Thanks again.

[MartinK 378]

Error: Service [Thread 73c]: SQLiteException

: sqlite3_prepare_v2 failureError: Application UnregisterCECallback: failed with 7013

 

The other two workstations exhibiting similar behavior show the following error:

 

Have not seen such error for long time. As error indicates, there is either something wrong with AGENT's database - it is either corrupted, or access to is is somewhat blocked. Please check whether there is enough space on disk where AGENT's database resides (default path is: C:\ProgramData\ESET\RemoteAdministrator\Agent\EraAgentApplicationData\Data\data.db). Access to this file may be also blocked by EES as part of self-defense. What version of AGENT and EES are installed on this machine? What version of HIPS module uses installed EES? I would also try to disable HIPS, restart system and check whether AGENT works.

 

In case database file is corrupted, it is very hard to guess what have happened. It may have been corrupted by power failure, application crash or maybe HW failure.

[j-gray 35]

On the system with the DB error, there is sufficient disk space. This one is running client 6.1.2227.0 and agent 6.1.444.0

 

The other two with the HIPS errors are both running client 6.1.2227.0 and agent 6.2.190.0

 

The problem in all three cases is that I cannot manipulate the systems remotely because of the agent; uninstall task will not run, upgrade task will not run. Deploy Agent task runs successfully, but does not correct the issue(s).

 

Given that I can't physically access the workstations, what's the best approach for remote repair?

[Gonzalo Alvarez 66]

Hi @j-gray,

 

Well most of people use TeamViewer for remote access or Log-me-in. Test and see.

 

I will start removing those old versions and install new ones 6.3 (also this can lead you

install ERAv6.3).

 

From other point of view, in my case:

• I check first the port the Agent use to communicate
• Add exception to the firewall in both sides (Terminal with Agent-Server with ERA)
  For firewall I mean "any" one (including ESET)

[MartinK 378]

On the system with the DB error, there is sufficient disk space. This one is running client 6.1.2227.0 and agent 6.1.444.0

 

The other two with the HIPS errors are both running client 6.1.2227.0 and agent 6.2.190.0

 

The problem in all three cases is that I cannot manipulate the systems remotely because of the agent; uninstall task will not run, upgrade task will not run. Deploy Agent task runs successfully, but does not correct the issue(s).

 

Given that I can't physically access the workstations, what's the best approach for remote repair?

 

Remote installation should run "repair" on AGENT installation -> it won't be successful in case DB was corrupted, so I guess problem is so called HIPS module of EES blocking AGENT's attempts to access its files. Unfortunately I am not sure where exactly could be problem. Is is possible to uninstall EES from target machine and attempt to repair AGENT (after system restart)? Or maybe manual upgrade of EES to never version?

[j-gray 35]

Only one system exhibits the corrupt database issue. The others have the HIPS error. Those with the HIPS error show they are replicating successfully with the server, but they simply won't show up as managed.

 

Is it safe to assume that the 6.2.190.0 agent is not fully compatible with the 6.1.2227.0 client?

 

And I'm still trying to figure out a way to repair these clients remotely via ERA console.  Any recommendations?

 

Thanks again for your help.

[Gonzalo Alvarez 66]

Hi @j-gray,

 

I cannot answer your question about compatibility.

 

To be honest I started full with 6.3, never face 6.1 and 6.2.

In your case I will start with 2 possible cases:

• Uninstall Endpoint, uninstall agent, reboot, install Agent 6.3, install endpoint 6.3.
   
• Install Agent 6.3 above of the installed one, (perhaps a Reboot just in case) install endpoint 6.3

That's will I do in your case, never tested because I never face it.

[MartinK 378]

Only one system exhibits the corrupt database issue. The others have the HIPS error. Those with the HIPS error show they are replicating successfully with the server, but they simply won't show up as managed.

 

Is it safe to assume that the 6.2.190.0 agent is not fully compatible with the 6.1.2227.0 client?

 

And I'm still trying to figure out a way to repair these clients remotely via ERA console.  Any recommendations?

 

Thanks again for your help.

 

Actually I was little wrong. They are fully compatible and this error means that this EES 6.1.2227.0 does not protects AGENT (= its. HIPS module is too old for it). You can safely ignore this error as it is expected for this versions combination.

If I understood you correctly, status.html of those AGENTs contains recent successful connection to your SERVER which indicates problem may be on SERVER. We have met with this problem, and most probable cause were:

• AGENT is connecting, but it is visible in Webconsole under different name than expected
• There are two installations of AGENT with the same unique identifier (cloning disk or VM machines) which causes two different installations to be represented with only one entry in Webconsole.

Could you please try to find this computer by last connection time when comparing status.html data and data visible in console? It may be possible for smaller networks.

[j-gray 35]

Thanks for the info. I think I figured out what is happening, just can't tell what's causing it. We're running AD Sync. Here's the scenario:

PC1 comes online and pulls IP 1.2.3.4.  PC1 goes offline, but stays in ERA console with IP 1.2.3.4

PC2 comes online and pulls IP 1.2.3.4 (because that IP is now available from DHCP).

 

ERA now thinks PC1 is online because of the IP address previously recorded for PC1.

So ERA shows PC1 online even though it is not, and shows PC2 as unmanaged even though it is managed.

 

I've verified DNS does not have duplicate hosts for the same IP.

 

We're not cloning/imaging workstations or using workstation VM's, so there should not be any duplicate uid's.

 

Any idea what's causing this?

[MartinK 378]

Name of computer is chosen in moment it is created - in your case it was named after IP because reverse resolving failed or DNS entry was not available. Even this computer changed IP address since first connection, it will be still visible as original name. I guess that is what caused confusion.

 

In order to rename computers, you have to either rename them manually (seems not practical in case computer are changing name very often) or you may use "Rename computers task" which can be scheduled to rename computer to FQDN they are reporting itself - I think this task has been added in 6.2 and improved for 6.3. For more details, please see documentation: hxxp://help.eset.com/era_admin/63/en-US/st_rename_computers.htm

 

Tip: you may manually name your computers as you wish. It is interpreted as IP/Hostname only when targeting it for remote installation (AGENT deploy) task.

[j-gray 35]

The computers are appearing in the console with their FQDN. But the computer that is offline is reporting an IP address that a new/online client has been assigned.

 

This conflict in the ERA console is my best guess as to why the online/healthy system shows up as unmanaged in the console.

 

 

As far as the client with the corrupt db, I was able to successfully remove the network agent via remote msiexec command. The challenge was finding the older agent installer left behind (C:\Windows\Temp) on an older client (ERA only contains latest agent, which wouldn't work for uninstall of older agent).

 

It would be awesome if there was a universal agent uninstaller that worked across multiple versions and could be run via command line.

[MartinK 378]

The computers are appearing in the console with their FQDN. But the computer that is offline is reporting an IP address that a new/online client has been assigned.

 

This conflict in the ERA console is my best guess as to why the online/healthy system shows up as unmanaged in the console.

 

In case you mean IP address shown next to computer name in main view, it is reported by AGENT -> in case AGENT is not connection, it will show IP address it used in moment it last connected and reported change. Regardless of this, computers in Webconsole are paired with AGENT installation using unique identifier, that is not associated with IP, FQDN or MAC address. Therefore it is not affected by HW or Network configuration.  In case computer is marked as unmanaged means, that this computer has never connected or last connection time has been erased. When you open client details for this particular computer, can you see any information that has been received from AGENT?

How was this unmanaged computer created? manually or using synchronization? How you installed AGENT on this system - using remote installation task?

 

 

It would be awesome if there was a universal agent uninstaller that worked across multiple versions and could be run via command line.

 

Actually something like this is being prepared, but I cannot give any ETA.

[j-gray 35]

I've attached a screen-cap that highlights the issue. Notice the hostname at the top, then the computer name at the bottom. Both of these systems have the agent and client installed but due to the hostname/IP conflict, they reflect various 'unknown' statuses. Yet both show green/current replications on their status.html pages.

 

All workstations are populated in ERA console via AD sync. None are manually added. Agents are all installed using the remote installation task. No GPO or other methods are used. We don't put the agent on images, so no cloning at all. One system shows some information in the console (see screen-cap). The other system named at the bottom shows up as unmanaged with no info.

 

And again, I've verified that DNS (both forward and reverse) are accurate.

 

Very puzzling.

 

agent-confusion.tiff

[MartinK 378]

Hostname we see on bottom is reported by client, and in case client is connecting (according to last connection time)  we can be almost sure this computer is WHSWIN703704B. I know it may be confusing, but name shown on top is static (it does not change automatically) and for some reason it is different than expected. Most probable reason is that DHCP server and DNS server were out of sync (or maybe only DNS caching caused this if enabled, see: https://support.microsoft.com/en-us/kb/245437).My guess is that AGENT was actually deployed to different computer than it was supposed to be during mentioned out-of-sync interval. In case my hypothesis is correct, you may be able to verify it from DHCP logs as those two computer should have shared the same IP address during time of deployment.

 

Regardless of reason, I would suggest you to regularly execute task Rename computers which will rename specified computer to current value - value from bottom of your screenshot. In case computers are using stable hostname and only IP addresses are changing, one-time execution should be sufficient. In case computers are changing also hostname, I would consider scheduling this task, to get current computer names.

[j-gray 35]

The 'Rename Computers' task runs every hour. In addition, because one of the conflicting clients was thought to be offline by ERA, it would delete it via the 'Delete not connecting computer' task. Then AD sync would pull it back into ERA when that task ran.

 

To work around the problem I:

1. deleted both systems from ERA, then ran AD sync again -no luck
2. deleted both systems from ERA, re-installed agent, then ran sync again -no luck

Only solution was to delete both clients from ERA, uninstall agent on both systems, then reinstall agent.

 

The hostname should not be changing at all on the network, except in a very rare case if a computer falls off the domain.

 

Unfortunately, this isn't an isolated incident, so I'm hoping to find the root cause...

[MartinK 378]

The 'Rename Computers' task runs every hour. In addition, because one of the conflicting clients was thought to be offline by ERA, it would delete it via the 'Delete not connecting computer' task. Then AD sync would pull it back into ERA when that task ran.

 

Is renaming task attached to proper group (If I recall correctly it renames computers only in Lost&Found group with default configuration)? Provided screenshot indicates it was either not renaming selected computer, or it does not work.

AD synchronization is merging computers by very simplistic algorithm -> it compares computers by name. In case no exact match is found, it assumes computer is missing and is re-created. This behavior seems to be direct outcome of previously mentioned issue with wrongly named computers.

 

Imagine you have only 1 computer in Webconsole, named X - but is is actually computer Y. Now when you run AD synchronization, it will create computer Y because it seems to be missing. Once you rename computer X to its proper name Y (so that you will have two computer with the same), another execution of AD synchronization task, if properly configured, will remove "dummy" computer Y created during previous synchronization and only correct computer managed by installed AGENT will stay.

 

Could you please check number of managed computer in Webconsole and compare it with number of AGENT you installed? My guess is that all clients will be in Webconsole + there will be "duplicates" that will be marked as unmanaged, because they never had AGENT installed.

 

Example how remote installation works. Lets have unmanaged computer called W. SERVER will indirectly resolve it's name W to IP address 1.1.1.1. After that, it will connect to 1.1.1.1 and install AGENT, that will be tied to W by unique identifier. That means no HW or network change can break this relation. In you case was AGENT installed either to different computer (i.e. wrong IP address) or computer was renamed afterwards.

 

Hope this brings more light into naming computers because I am out of ideas, as described problems are very hard to reproduce and debug.

Edited May 11, 2016 by MartinK