Upgrading PROTECT v8.0 to v9.0 : ERA server communication processing error: Connection refused

[ewong 8]

Hi,

With some time on my hand, I took the plunge and downloaded the necessary components to upgrade the PROTECT v8.0 (on a CentOS 7 system) to v9.0. 

I followed :https://support.eset.com/en/kb8150-manual-component-based-upgrade-from-eset-security-management-center-7x-for-linux-to-the-latest-version-of-eset-protect

After running the server install, the rdsensor install and the era.war deployment to tomcat, I started the services.

I tried logging on but get an error "Login failed: Connection has failed with state "Not connected"

Having experienced that before, it has something to do with Tomcat.

I did a "sudo systemctl start eraserver"  (just in case),  then "sudo systemctl start tomcat",  and then I tried logging on, it still reported Not connected.

sudo systemctl status tomcat -l

Quote

Feb 09 17:12:46 esmc.company.local server[4144]: Feb 09, 2022 5:12:46 PM sk.eset.era.g2webconsole.server.modules.logger.LogItem logInto
Feb 09 17:12:46 esmc.company.local server[4144]: SEVERE: [communication_error_run] Connection refused
Feb 09 17:12:46 esmc.company.local server[4144]: Feb 09, 2022 5:12:46 PM sk.eset.era.g2webconsole.server.modules.logger.LogItem logInto
Feb 09 17:12:46 esmc.company.local server[4144]: WARNING: [Administrator] Login (session creation) failed (code 3) from address remoteAddress: 192.168.0.27 [user-agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:68.0) Gecko/20100101 Firefox/68.0 SeaMonkey/2.53.10.1; accept-language: en-US,en;q=0.5].
Feb 09 17:12:46 esmc.company.local server[4144]: Feb 09, 2022 5:12:46 PM sk.eset.era.g2webconsole.server.modules.logger.LogItem logInto
Feb 09 17:12:46 esmc.company.local server[4144]: INFO: [] Closing connection
Feb 09 17:12:46 esmc.company.local server[4144]: Feb 09, 2022 5:12:46 PM sk.eset.era.g2webconsole.server.modules.logger.LogItem logInto
Feb 09 17:12:46 esmc.company.local server[4144]: SEVERE: [] Connection closing because of ERA server communication processing error: Connection refused
Feb 09 17:12:46 esmc.company.local server[4144]: Feb 09, 2022 5:12:46 PM sk.eset.era.g2webconsole.server.modules.logger.LogItem logInto
Feb 09 17:12:46 esmc.company.local server[4144]: SEVERE: [communication_error_run] Connection refused

 

Seeing the connection refused,  I'm guessing it's an Era Server issue. 

But when I do a "sudo systemctl status eraserver"  I get:

Quote

● eraserver.service - ESET PROTECT Server
   Loaded: loaded (/etc/systemd/system/eraserver.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2022-02-09 17:20:47 HKT; 1min 30s ago
  Process: 7014 ExecStart=/opt/eset/RemoteAdministrator/Server/ERAServer --daemon --pidfile /var/run/eraserver.pid (code=exited, status=0/SUCCESS)
 Main PID: 7015 (ERAServer)
   CGroup: /system.slice/eraserver.service
           └─7015 /opt/eset/RemoteAdministrator/Server/ERAServer --daemon --pidfile /var/run/eraserver.pid

Feb 09 17:20:47 esmc.company.local systemd[1]: eraserver.service holdoff time over, scheduling restart.
Feb 09 17:20:47 esmc.company.local systemd[1]: Stopped ESET PROTECT Server.
Feb 09 17:20:47 esmc.company.local systemd[1]: Starting ESET PROTECT Server...
Feb 09 17:20:47 esmc.company.local systemd[1]: Started ESET PROTECT Server.

 

So I try a "sudo systemctl restart tomcat"

But when I try to log on, I still get the "Not connected" error. 

I do see a /var/run/eraserver.pid, so I'm guessing that it is running as well as seeing it in the process list.

I took a look at the /var/log/eset/RemoteAdministrator/Server/trace.log and noticed that the last line was:

 

Quote

2022-02-09 09:35:14 Information: CDatabaseModule [Thread 7f30ac206740]: CDBSetupperBase::PerformUpgradeIfNecessary: Old routines and views are deleted.
2022-02-09 09:35:14 Information: CDatabaseModule [Thread 7f30ac206740]: CDBSetupperBase::PerformUpgradeIfNecessary: Going to create tables: fact_fe_threat_event tbl_computers_aggr tbld_activethreats_status_engineversion tbld_application_identifier tbld_application_parameters tbld_apps_installed_status_name tbld_apps_installed_status_vendor tbld_cause tbld_computers_comment tbld_computers_name tbld_devicecontrol_device_event_device tbld_deviceinformation_device_status_manufacturer tbld_deviceinformation_device_status_model tbld_devicelocation_gps_status_provider tbld_diagnostics_diagnosticzip_event_data tbld_eesvirusdb_status_versiondate tbld_enterpriseinspectoralert_event_rulename tbld_exportedconfiguration_event_configuration tbld_firewallagregated_event_protocol tbld_hwinventory_chassis_status_description tbld_hwinventory_chassis_status_manufacturer tbld_hwinventory_display_status_description tbld_hwinventory_display_status_manufacturer tbld_hwinventory_displayadapter_status_description tbld_hwinventory_displayadapter_status_manufacturer tbld_hwinventory_inputdevice_status_description tbld_hwinventory_inputdevice_status_manufacturer tbld_hwinventory_massstorage_status_description tbld_hwinventory_massstorage_status_manufacturer tbld_hwinventory_networkadapter_status_description tbld_hwinventory_networkadapter_status_manufacturer tbld_hwinventory_printer_status_description tbld_hwinventory_processor_status_description tbld_hwinventory_processor_status_manufacturer tbld_hwinventory_ram_status_description tbld_hwinventory_ram_status_manufacturer tbld_hwinventory_sounddevice_status_description tbld_hwinventory_sounddevice_status_manufacturer tbld_identifiers_list_status_value tbld_ip_mask tbld_loggedusers_list_status_domain tbld_loggedusers_list_status_fullname tbld_loggedusers_list_status_name tbld_osinformation_edition_status_os_info_edition tbld_osinformation_locale_status_os_locale_language tbld_osinformation_timezone_status_time_zone_name_offset tbld_processname tbld_quarantine_uploadedfile_event_password tbld_quarantine_uploadedfile_event_path tbld_rdsensor_newcomputers_status_computer_identifier tbld_rdsensor_newcomputers_status_ipv4ipv6 tbld_rdsensor_newcomputers_status_mergedidentifier tbld_rdsensor_newcomputers_status_netcardvendor tbld_rdsensor_newcomputers_status_osname tbld_remote_host tbld_rulename tbld_scantargetsid tbld_security_product_status_nameversion tbld_srvsecproduct_scantargets_status_data tbld_static_groups_comment tbld_static_groups_name tbld_submittedfiles_event_username tbld_sysinspector_sysinspector_event_logdata tbld_threat_event_engineversion tbld_threatname tbld_used_license_status_licenseid_licenseproductname tbld_usergroup tbld_username tblf_activethreats_status tblf_applicationactivationmatrix_status tblf_appliedpoliciescount_status tblf_appliedpolicieslist_status tblf_appliedpolicyproducts_status tblf_apps_currentversion_status tblf_apps_installed_status tblf_apps_securitystatus_status tblf_apps_versioncheck_status tblf_audit_event tblf_blockedfiles_event tblf_certificates_peercertificate_status tblf_cloudalerts_event tblf_computer_connected_event tblf_computer_lost_event tblf_computercloningtickets_status tblf_computeridentityrecovered_event tblf_computerlocationmembership_status tblf_devicecontrol_device_event tblf_deviceinformation_device_status tblf_devicelocation_gps_status tblf_diagnostics_devicecontrol_device_event tblf_diagnostics_diagnosticzip_event tblf_diagnostics_firewall_event tblf_diagnostics_hips_event tblf_diagnostics_spam_event tblf_diagnostics_webcontrol_link_event tblf_dynamicgroups_content_status tblf_dynamicgroupsmembership_status tblf_dynamicthreatdetectionanalyses_status tblf_dynamicthreatdetectionglobalcustomersstatistics_status tblf_dynamicthreatdetectionglobaldetectionstatistics_status tblf_eesevent__event tblf_eesvirusdb_status tblf_encryption_storage_status tblf_enterpriseinspectoralert_event tblf_enterpriseinspectoroverview_status tblf_enterpriseinspectoroverviewtotals_status tblf_epns_status tblf_exclusionhitsagregated__event tblf_exclusionhitssummary_status[root@esmc Server]#

 

 

Is this an indication that it is recreating all those tables and that this is normal?  It was sometime since I installed v8 so I don't recall having this issue.   

Any clarifications appreciated,

 

Thanks!

Edmund

PS: I just realized that I hadn't backed up the database... oh well.  I guess if this goes fubar, I'll need to recreate this whole set up again.  Lesson learnt.

 

 

[ewong 8]

While watching the trace.log, something became very apparent.  It was repeating the process of deleting tables/creating tables..  as evident by the following log:

I've attached a log to this message.

It basically deletes all the existing tables...  then it recreates them...  does some stuff in the middle.  Loads modules.. then detects something  "Checking if ETL DB required" and then it finds that is required. "ETL upgrade required"...

I'm a little stumped as to why it's doing that.

Any help appreciated

 

Edmund

test.log

Edited February 9, 2022 by ewong

[ewong 8]

Complete PEBCAK..  I totally messed up the database.   So I'm now in the process of installing PROTECT v9. 

Oh well..  live and learn.

 

[Adrastos 0]

I just wanted to pass on my experience in updating the OS of my ESET 9 ERA console.  After upgrading I received the dreaded "Login Failed, Connection has failed with the state of 'Not connected'" in ESET Remote Administrator".  After the upgrade of the OS, it appears that the ERA server didn't start.  I checked by running "systemctl status eraserver".  The error I received was that it was not running.  So I started it with "systemctl start eraserver".  I wasn't able to log in yet. 

I restarted the database with "systemctl restart mysqld" and then I was able to log in.  Hope this helps someone.  I also restarted the tomcat server "systemctl restart tomcat" during my troubleshooting, but that didn't seem to help.  I think the overall problem was that the era server wasn't running.  Once it was running, I then needed to reconnect the database by restarting it.  

Edited February 21, 2022 by Adrastos

[MartinK 378]

On 2/21/2022 at 3:46 AM, Adrastos said:

I just wanted to pass on my experience in updating the OS of my ESET 9 ERA console.  After upgrading I received the dreaded "Login Failed, Connection has failed with the state of 'Not connected'" in ESET Remote Administrator".  After the upgrade of the OS, it appears that the ERA server didn't start.  I checked by running "systemctl status eraserver".  The error I received was that it was not running.  So I started it with "systemctl start eraserver".  I wasn't able to log in yet. 

I restarted the database with "systemctl restart mysqld" and then I was able to log in.  Hope this helps someone.  I also restarted the tomcat server "systemctl restart tomcat" during my troubleshooting, but that didn't seem to help.  I think the overall problem was that the era server wasn't running.  Once it was running, I then needed to reconnect the database by restarting it.  

Any chance you checked logs for more details? During upgrade, it is fairly normal that user is disconnected from the console in a moment when ERA/ESMC/PROTECT server is stopped due to upgrade, but it should definitely start afterwards. What might be confusing is that resource-consuming part of the upgrade process is actually executed during first startup of newly upgrade server, but during this phase, connection from the console are not possible which might lead to a confusion and often even to user-enforce interruption of this process.

Also you mentioned "after upgrade of the OS" .. does it mean you have made some upgrade of your linux environment? If so, could you please provide more details of this environment - I can imagine that significant jump between major releases of linux distributions might cause such issues, but it should not happen with regular system updates...

[MartinK 378]

On 2/9/2022 at 10:43 AM, ewong said:

While watching the trace.log, something became very apparent.  It was repeating the process of deleting tables/creating tables..  as evident by the following log:

I've attached a log to this message.

It basically deletes all the existing tables...  then it recreates them...  does some stuff in the middle.  Loads modules.. then detects something  "Checking if ETL DB required" and then it finds that is required. "ETL upgrade required"...

I'm a little stumped as to why it's doing that.

Any help appreciated

 

Edmund

test.log 115.87 kB · 0 downloads

In this case, what is suspicious in logs you provides is this section:

2022-02-09 09:50:46 Information: CDatabaseModule [Thread 7fba899c0740]: CDBSetupperBase::PerformUpgradeIfNecessary: Processing log in ETL:QOS_NETWORK_EVENT
2022-02-09 09:50:47 Information: CDatabaseModule [Thread 7fba899c0740]: ETL CLogsETLMapper: Starting plan for logtype: QOS_NETWORK_EVENT - number of logs:44178

2022-02-09 09:52:02 Information: [Thread 7f1ca4d56740]: Loading ESET modules from /var/opt/eset/RemoteAdministrator/Server/Modules/
2022-02-09 09:52:02 Information: Kernel [Thread 7f1ca4d56740]: Local time is 2022-02-09 17:52:02
2022-02-09 09:52:02 Information: Kernel [Thread 7f1ca4d56740]: InstallConfiguration: ProductLine: era
2022-02-09 09:52:02 Information: Kernel [Thread 7f1ca4d56740]: InstallConfiguration: ProductVersion: 9.0.2144.0

which indicates that PROTECT service was interrupted during process of database upgrade. In this case it was fairly soon after it actually started. In case it was not made intentionally by user of operating system, most common issue resulting to this state we are aware of is possible insufficiency of system resources. Especially database server might consume more resources than expected during this phase. Is it possible this might be the case? It is hard to estimate as we do not know the details and from logs it dies not seem to be large environment.

Any chance OS logs contain more details (for example OOM might indicate process was killed due to extensive use of resources) or possibly crash reports of our "ERAServer" process during this phase? It seems that process was failing repeatedly but there are no crash reports in our logs.

[ewong 8]

Hi,  Sorry for the delay in response.

What I ended up doing (iirc) was to junk the whole install by removing the whole installation and installing it over again.   I think there was a hiccup during the installation process. 

Edmund