Server problem after reboot - ELAM

[DavidC 0]

Hi,

We have a VM windows server 2019 with ESET Server Security for Microsoft Windows Server installed. Since 2 "windows update tuesday's", on reboot, server doesn't boot (spinning wheel).

The first time, we done a restore of previous backup (2 days before), it's has booted well, done all updates, no more problems.

The 2on time, same problem, same solution (with 2 days before backup). This time, we have more time to test what appends and after restore backup from some previous hours, we can reproduce the problem. The only way to start the server was starting in safe mode or disabling ELAM. Removing eset product with uninstaller in safe mode "repairs" it definitively.

 

Do you have a feedback of this problem ? Some ideas to avoid  new problems ?

[Marcos 5,070]

Please configure Windows to generate complete memory dumps as per https://support.eset.com/en/kb380. Then rename the eelam.sys driver back and reproduce the system lockup. Then manually trigger a crash according to the above KB. Next rename eelam.sys driver in safe mode (ekrn will not load and protect the server). Compress the memory dump and supply it to me in a compressed form along with logs collected with ESET Log Collector.

[DavidC 0]

On 4/17/2023 at 1:00 PM, DavidC said:

efsw logs

 

efsw_logs.zip

[DavidC 0]

memory dump

MEMORY.rar

[Marcos 5,070]

Is the dump from the same computer as the ELC logs? I have some doubts because:

1, The dump doesn't contain any ESET services or drivers, however, it contains data about Symantec drivers (e.g.  symefasi64.sys, SymELAM.sys, SYMEVENT64x86.SYS).

2, The dump was taken after the system had been up and running for 14 days, I'd expect it to be generated shortly after a reboot if ekrn did not start.

3, The ELC logs do not show any Symantec files. They show ekrn as not running and were taken 3 minutes after a system start so they appear to be from the right machine. There's no record of ekrn not being started due to timeout in the system logs. The system log is full of errors like this:

The ADSync service was unable to log on as CAXXXXXXXXYA\AADXXXXXXXXXXX44 with the currently configured password due to the following error:
The user name or password is incorrect.
To ensure that the service is configured properly, use the Services snap-in in Microsoft Management Console (MMC)., 19/04/2023 14:56:45
The application log is full of errors like
svchost (2620,D,23) SRUJet: Database C:\WINDOWS\system32\SRU\SRUDB.dat: Index UserIdTimeStamp of table {D10CA2FE-6FCF-4F6D-848E-B2E99266FA89} is corrupted (0)., 11/04/2023 19:53:00

Not sure if any of the above issues this could somehow affect the start of ekrn.

You could try the following to fix the error with srudb.dat:

1. Open the Start menu and type "Indexing Options" and select it.
2. Click on the "Advanced" button.
3. In the "Advanced Options" dialog box, click on the "Rebuild" button.
4. Click "OK" to confirm the rebuild process.
5. Wait for the rebuild process to complete. This may take some time, depending on the size of your index and the speed of your computer.

Once the index has been rebuilt, the error message should disappear, and your Windows Search should function normally again.

[DavidC 0]

Well... Let me verify with my collegue what he has done. He had to do all actions with the backup just before the reboot and crash (4h before).... Maybe he have done something wrong.

 

With backup just before crash, becaise production was restored from another backup (30hours before)

[DavidC 0]

ok,
He had mixed 2 machines (that was causing error, and production "restored" machine)
 

New try:
- restored machine with backup from just before crash (3-4 hours before)
- verified that the problem exists : normal boot stays with spinning wheel, but forcing elam-disabled boot mode starts the machine

- disabled adsync and windows search services, problem persist after reboot (no boot on normal mode)

- in elam-disabled mode, I enable crash ctrl keys, complete dump option, and did the recollection of logs. I confirm that there's no Eset icon in the task bar.

- trying to boot in normal mode : never-end boot, generate a memory dump but no sufficient disk space (only 10Gb)
- reboot in elam-disabled mode, to retrieve disc space

- trying to boot in normal mode : never-end boot, generate a memory dump successfully
 

We are looking for what was the problem in the backup VM because production is already restored but the problem has appeared 2 times in 2 months.

MEMORY-2.rar

[DavidC 0]

efsw_logs-2.zipAnd the efsw logs

[Marcos 5,070]

Here are our findings:

• An older version of ESET Server Security 8.0.12011.0 is installed
• As a result, in newer program versions we had to replace some Windows dlls in the ESET install folder with dlls signed also by ESET to prevent verification using the Windows catalog
• The catalog got probably corrupted and was restored:

CatalogDB: 9:59:27 AM 4/20/2023: Init:: Database previously shutdown dirty  C:\WINDOWS\system32\CatRoot2\{127D0A1D-4EF2-11D1-8608-00C04FC295EE}\catdb
CatalogDB: 9:59:27 AM 4/20/2023: Init:: Database previously shutdown dirty  C:\WINDOWS\system32\CatRoot2\{F750E6C3-38EE-11D1-85E5-00C04FC295EE}\catdb

• As a result, Windows started to verify the Windows dlls signature which is known to take minutes. However, there is a default timeout 30s for starting services so starting ekrn failed. After the next restart, verification of dlls signature takes less as the results are cached by Windows and ekrn starts alright.

We recommend to use the latest version of ESET Server Security v10 (or at least the latest v9 which will be fully supported until March 2026 according to https://support-eol.eset.com/en/policy_business/product_tables.html).

Keep an eye on C:\Windows\System32\catroot2\dberr.txt  and make sure that catalog corruption doesn't occur.

[DavidC 0]

Hi Marcos and thanks for you response.

So, we have to upgrade ESET version. Maybe a conflict that appears when WU and Update Catalog.

Another question appears to me : ESET protect cloud console says there's no automatically upgradable product for these computer.
The clients computer are already upgraded to Endpoint Antivirus 10 but none of the servers are. All servers are in Server Security 8 and i don't find how to do it (manual or automatically)

How do i do this ?

[Marcos 5,070]

You can download the latest version from our website https://www.eset.com/int/business/download/file-security-windows/.

If I remember correctly, automatic program updates were first supported by v9. The so-called uPCU is installed after the next server reboot after being downloaded to prevent disruption.

[DavidC 0]

ok, thanks. I'll do that.

[DavidC 0]

Hi,

In order to add information, note that trying to upgrade to V10, this error appears:

We cannot find the installation of ESET Server Security in windows installer database. After clicking OK, ESET install helper will run to restore Server Security Sentries.

 

If i try to continue, it says that the folder already exist.

 

i manage it with eset uninstaller and after that, an install.