ERA 6.4 to 6.5 Upgrade Error - "ETL upgrade required"

[wind-e 1]

Hello,

We've upgraded ERA from 6.4.29.0 to 6.5.522.0 using the ERA Component Upgrade task. Everything went fine during the upgrade and seemed to go smoothly, however, after trying to log into the Web Console afterwards we received "Login failed: Connection has failed with state 'Not connected'" after providing our credentials.

• We gave it 30 minutes to connect to the database (which is hosted on SQL Server 2008 R2 instance on a different VM) and still received the error.
• We then tried restarting the ESET Remote Administrator Server service, but it just hung on "Stopping" indefinitely.
• We then rebooted the server and followed the instructions on this ESET KB article: hxxp://support.eset.com/kb5852/?viewlocale=en_US, which did nothing.
• The ESET Remote Administrator Server service will now start and stop 3 seconds later.
• Upon viewing the trace.log (found at C:\ProgramData\ESET\RemoteAdministrator\Server\EraServerApplicationData\Logs), we noticed the last entries are the following:

2017-03-14 21:27:04 Information: Kernel [Thread 6e4]: Used memory before modules load and initialization is 28548 KB
2017-03-14 21:27:04 Information: Kernel [Thread 6e4]: Initializing module CTranslatorModule
2017-03-14 21:27:04 Information: CTranslatorModule [Thread 6e4]: Registering translator resource
2017-03-14 21:27:04 Information: CTranslatorModule [Thread 6e4]: Module successfully initialized
2017-03-14 21:27:04 Information: Kernel [Thread 6e4]: Initialized module CTranslatorModule (used 8 KB)
2017-03-14 21:27:04 Information: Kernel [Thread 6e4]: Initializing module CUpdatesModule
2017-03-14 21:27:04 Information: Kernel [Thread 6e4]: Initialized module CUpdatesModule (used 48 KB)
2017-03-14 21:27:04 Information: Kernel [Thread 6e4]: Initializing module CDatabaseModule
2017-03-14 21:27:05 Information: CDatabaseModule [Thread 6e4]: Startup configuration builder builds successfully ...
2017-03-14 21:27:06 Information: CDatabaseModule [Thread 6e4]: Startup connection pool created ...
2017-03-14 21:27:06 Information: CDatabaseModule [Thread 6e4]: Builder rebuilds successfully ...
2017-03-14 21:27:06 Information: CDatabaseModule [Thread 6e4]: Connection pool recreated ...
2017-03-14 21:27:06 Information: CDatabaseModule [Thread 6e4]: CDBSetupperBase::PerformUpgradeIfNecessary: Checking if ETL DB is required
2017-03-14 21:27:06 Information: CDatabaseModule [Thread 6e4]: CDBSetupperBase::PerformUpgradeIfNecessary: ETL upgrade required.
2017-03-14 21:27:07 Information: CDatabaseModule [Thread 6e4]: CDBSetupperBase::PerformUpgradeIfNecessary: Going to back up old logs.
2017-03-14 21:58:03 Information: Kernel [Thread f80]: Unloading module: CUpdatesModule
2017-03-14 21:58:03 Information: Kernel [Thread f80]: Unloading module: CTranslatorModule
2017-03-14 21:58:03 Error: Service [Thread f80]: [Microsoft][ODBC SQL Server Driver][SQL Server]Invalid object name 'tbl_log_apps_installed_status'.

 

Does this mean the database is corrupt or didn't finish updating??? Not sure what went wrong with the upgrade... Would really like to avoid a database restore!

Edited March 14, 2017 by wind-e

[MartinK 378]

Upgrade to 6.5 may take more time than 30 minutes in case of slower HW configuration or larger amount of stored data (logs). This error was most probably caused by interrupted upgrade procedure you mentioned. Unfortunately we are not able to provide workaround until we know what exactly got corrupted.

Please restore DB backup and try to upgrade again. If it won't work, contact ESET support for that more details could be provided for analysis.

[wind-e 1]

Thanks @MartinK. The database was large (this was due to a configuration mistake in which there were 2 Agent tasks to report installed applications). It seems we did not wait long enough for the database to complete it's upgrade and we corrupted it.

Fortunately, we were able to restore both the 6.4 ERA server and the DB (hosted separately) successfully. We will try upgrading again and have more patience this time (we now see not waiting long enough is a known issue )

[CraigB 1]

@wind-e Can you let me know if your upgrade is successful? I had the same error and had to roll our server back to a snapshot taken before the upgrade to get it back online. I left my upgrade running for over 2 hours, and it appeared to have hung, but maybe it was still running... Only have about 300 client computers connecting, but they are scattered across many networks.

Haven't had a chance to re-run the update since.

[wind-e 1]

I plan on trying again tomorrow or Friday and will share results.

Do you know if ERA was actively running a task at the time of your upgrade? I believe mine was (whether or not that matters, I'm not sure), and that's really the only thing I'm going to do differently this time (besides waiting a day for it to complete).

[CraigB 1]

Thanks

I'd stopped both the ERAagent and ERAserver services, and the tomcat webserver before starting the upgrade, so I don't think so. Possibly did not wait long enough for those services to stop before running the upgrade though.

[MichalJ 434]

Hello,

You should not stop the service before doing the upgrade. The recommended approach to upgrade is to run the "Remote Administrator components upgrade task" targeted to the ERA server machine from ERA webconsole. 

[rekun 43]

I did not stop any services, but one of my upgrades took more than an hour, with the Sql service using a slot of disk io. 

 

So just give it some time, it will finish eventually. There is properly some big database design changes in 6.5 so the old data needs to be converted. 

[wind-e 1]

@CraigB Just finished trying to upgrade to 6.5 again. This time it was a success! Took about 1 hour.

[MichalJ 434]

There have been major database changes done, to make the console more responsive and improve the UX & performance. We are glad to hear that updates were smooth. 

[CraigB 1]

22 hours ago, MichalJ said:

Hello,

You should not stop the service before doing the upgrade. The recommended approach to upgrade is to run the "Remote Administrator components upgrade task" targeted to the ERA server machine from ERA webconsole. 

Thanks Michael

I was following the instructions here: hxxp://support.eset.com/kb3670/?intcmp="KBCJ_3668_Linuxupgrade"
which states to stop the tomcat7 service. The first time I tried the upgrade, it is possible I did not wait long enough (perhaps put a warning in the instructions that a long wait is required for the database upgrade - when I could not access the system after the normal 10 minutes or so it takes from a restart, I assumed it had failed). I tried a second time after stopping all the services. I see now this is also incorrect.

I am unclear on how I can add the server agent to the webconsole so I can target the server itself. Are there some instructions for this (for linux)?

[wind-e 1]

Quote

perhaps put a warning in the instructions that a long wait is required for the database upgrade - when I could not access the system after the normal 10 minutes or so it takes from a restart, I assumed it had failed

hxxp://support.eset.com/kb5949/#ERA

Technically not in upgrade instructions, but it's still documented.

....I didn't read this either beforehand. 

[foneil 342]

FYI, we posted an Alert related to this issue 

Upgrade to ERA 6.5 may take some time to complete

[E-Concept Applications 0]

Hello everybody,

I had the same problem, and this topic allowed me to solve it.
Indeed, the update can take very long!
For exemple, my ERA database is between 7 and 8GB (224 computers in ERA), and it took several hours to make the upgrade.

It is therefore necessary to grin and bear it, and to anticipate the upgrade to make it in a period where it does not interfere with your activity.

And to the attention of Eset: would it be possible to make more visible the KB that you published, because if we don't go on this topic, it's very difficult to find it. (And a french translation would be even better!)

Thanks to all!

PS : Sorry for my aproximative English...

Edited March 29, 2017 by E-Concept Applications