Real-time file system protection is non-functional on Server 2016 / ESET Server Security 9.0.12013.0

[st3fan 6]

Hello, we noticed that real-time file system protection is non-functional on about 15 servers. This all started after the last reboot. 

The first server started reporting this on Friday after a reboot.

 

We have rebooted all the affected servers but the error did not clear. The ESET Service (ekrn) is up and running.

Detection Engine;28381;12/11/2023
Rapid Response module;23441;12/11/2023
Update module;1039;10/23/2023
Antivirus and antispyware scanner module;1605.2;11/15/2023
Advanced heuristics module;1227;10/30/2023
Archive support module;1345;11/27/2023
Cleaner module;1245.1;12/5/2023
Anti-Stealth support module;1187;8/28/2023
Firewall module;1439.2;11/15/2023
Translation support module;1990;11/10/2023
HIPS support module;1466;9/19/2023
Internet protection module;1457;9/4/2023
Database module;1124;9/12/2023
Configuration module;2099.6;11/23/2023
Direct Cloud communication module;1134;10/5/2023
Rootkit detection and cleaning module;1033;9/16/2022
Network protection module;1692;7/15/2022
Script scanner module;1170;11/8/2023
Cryptographic protocol support module;1082;11/6/2023
Advanced Machine Learning module;1146;11/22/2023
Security Center integration module;1038;7/28/2022

 

Is anyone else seeing this?

[Marcos 5,158]

You are using an old version of ESET File Security v9.0.12013. Please update to the latest v9 (9.0.12018) or to v10.

[st3fan 6]

We are in the process of upgrading to 10.0.12014.0 but we still have a few servers that are on version 9.0.12013.0. Even though this is not the most recent version,  9.0.12013.0 is still supported if I am not mistaken (even 8.0.12015.1 is still in limited support afaik). Please correct me if I am wrong.

I am unable to upgrade ESET on any of the affected servers. Auto-update fails. Software deployment task fails. And manually running the MSI file fails too.

 

Event Log shows this:

Product: ESET Server Security -- Error 1922. Service 'ESET Service' (ekrn) could not be deleted.  Verify that you have sufficient privileges to remove system services.

 

FYI, this problem only affects 15 servers. We still run 9.0.12013.0 on about 50 servers (same OS etc.) where we don't see any problems yet. I have opened a ticket with our local Support.

 

[Marcos 5,158]

The Entrust certificate expired on December 7. Only ESET File Security for Windows v9.0.12018 and newer have drivers that can verify the new certificate.

[st3fan 6]

But why would this cause problems with HIPS drivers, firewall services etc?

For testing purposes I rebooted another server where 9.0.12013.0 is installed - and now we see the same issue there too.

So this essentially means that ESET will break on all servers that are rebooted now, if 9.0.12013.0 is installed there? I am assuming this will affect other ESET customers too, i.e. anyone who uses 9.0.12013.0 or lower (and I am sure there are plenty)?

FYI, upgrading ESET does not seem possible after the reboot anymore. All upgrade attempts fail, showing the error I mentioned above.

[st3fan 6]

A few more updates on this:

I have prepared the auto-update for version 10.0.12014.0 for all 9.0.12013.0 servers that have not been rebooted yet. Afterwards I rebooted a few servers for testing purposes. All good.

The 9.0.12013.0 servers that have been rebooted before the auto-update was prepared are broken. I had to remove ESET in Safe Mode and reinstall it. All other upgrade attempts were unsuccessful.

 

So to summarise for any other ESET admin who might run into this:

Do not reboot your Windows servers if you are still running ESET Server Security 9.0.12013.0.

 

@Marcos, surely we can't be the only ones who still use ESET v9. I am curious what the plan is for all other customers who might run into this problem. And I am assuming there will be a lot of problems this week, when Windows servers are rebooted after Patch Tuesday.

[Marcos 5,158]

Many of our users have probably already updated to the latest v9. It is crucial to keep the product updated, in this case at least to the latest version v9 if not to v10. We had been notifying about the expiring Entrust certificate and the need to upgrade the product and the OS to an ACS-compatible version throughout this year.

[Marcos 5,158]

This worked for me:
- set a system date from before Dec 7
- rebooted the server to ensure that drivers are loaded
- installed EFSW 9.0.12018.0 over 9.0.12013.0
- rebooted the server

[st3fan 6]

40 minutes ago, Marcos said:

We had been notifying about the expiring Entrust certificate and the need to upgrade the product and the OS to an ACS-compatible version throughout this year.

Are you referring to this @Marcos? https://support-eol.eset.com/en/trending_weol2023_10_2022.html

 

Edited December 11, 2023 by st3fan

[st3fan 6]

Assuming the above article was the one that you were referring to @Marcos, I am not sure if this applies in our case but please correct me in case I misunderstood.

• What is described in this article should only impact new installations or upgrades (“Existing installations of ESET products will not be affected and will keep receiving module updates.”). In our case, only existing installations were affected.
• Our issues did not start after any upgrades etc. The issues started after a simple server reboot, without any other changes.
• Our Windows servers are up-to-date and they should support ACS (still checking). If this was not the case, I assume we would not be able to do a fresh install of version 10.0.12014.0 (includes an installation block if the OS does not support ACS).

[StotheR 0]

Hello,

we have the exact same problem with many servers. First I guessed it had something to do with Windows updates, but according tho this thread here, it looks like it had somethin to do with a system restart (which happend after Windows update installation last night). Our Windows Server systems are all up to date, so they shouldn't missing the relevent ACS KB.

Is there a easy way to fix this? 

[StotheR 0]

All affected servers have File Security v9.0.12013.0 installed. And this was just the first wave of reboots due to Windows Update instllation. This night will come a lot more

[Marcos 5,158]

We are preparing a workaround via an automatic module update, however, it will be necessary to upgrade to the latest v9 or v10 afterwards.

You can fix it yourself immediately by:
1, Uninstalling ESET in safe mode using the ESET Uninstall tool and installing the latest v9 or v10.
2, Changing the system date to a day from before December 7, rebooting the server and upgrading to the latest version v9 or v10.

[st3fan 6]

We had to uninstall ESET on all affected servers that were rebooted. I can confirm that this is definitely not related to the December Windows updates. In most cases we had to remove ESET using the uninstaller in safe mode as the ESET service could not be stopped/removed properly anymore.

Before you reboot any other servers, make sure that you install a more recent ESET version.

I doubt this problem is caused by this (which I believe is what Marcos meant). This should not affect existing installations according to the article.

[Marcos 5,158]

1 minute ago, st3fan said:

I doubt this problem is caused by this (which I believe is what Marcos meant). This should not affect existing installations according to the article.

It's partly related. ACS support is required after the expiration of the SHA256 Entrust certificate on Dec 7 and this issue is caused by an already expired Entrust certificate which prevents drivers from being loaded when ekrn.exe starts.

[SteD 2]

Hi!

As we also have this issue, I would like to know, if only Server Security is affected, or if Endpoint Antivirus will also stop working (and if not, why only server)?

Regarding the planned fix: will it work to repair servers with the issue or will it only prevent this issue for future reboots and the affected servers need to be updated with the latest ESET Version?

Is there also a list of version, which have the issue and which don't have the issue?

[Marcos 5,158]

7 hours ago, SteD said:

As we also have this issue, I would like to know, if only Server Security is affected, or if Endpoint Antivirus will also stop working (and if not, why only server)?

We haven't been reported issues with Endpoint.

 

7 hours ago, SteD said:

Regarding the planned fix: will it work to repair servers with the issue or will it only prevent this issue for future reboots and the affected servers need to be updated with the latest ESET Version?

The "fix" will allow drivers to load, however, we strongly advise to upgrade to the latest v8/v9 or v10 as soon as possible.

The latest EFSW v8 and v9 are not affected. ESET server products v10 were not affected at all since these were already signed with an ACS certificate.

[baran 0]

How to restore 1400 servers to the date before December 7th and upgrade to version 10? Is it possible? Fix this problem so that it can easily upgrade to version 10. Please

[Kirb 0]

We're also having this issue on a server hosted in Azure. I'm not even sure it's possible to change the date back to before Dec 7th on these. There's got to be another way to fix this.

[StotheR 0]

 

[mikeewa 0]

On 12/13/2023 at 5:00 AM, Marcos said:

We are preparing a workaround via an automatic module update, however, it will be necessary to upgrade to the latest v9 or v10 afterwards.

You can fix it yourself immediately by:
1, Uninstalling ESET in safe mode using the ESET Uninstall tool and installing the latest v9 or v10.
2, Changing the system date to a day from before December 7, rebooting the server and upgrading to the latest version v9 or v10.

Any idea when the workaround will be available?

[Marcos 5,158]

2 hours ago, mikeewa said:

Any idea when the workaround will be available?

HIPS module 1467.3 is on the pre-release update channel and has been downloaded by a quite big group of users on the regular update channel already.

[mikeewa 0]

The hips module will fix the other modules not working as well? How do I force the update on the regular update channel?

 

[Marcos 5,158]

2 minutes ago, mikeewa said:

The hips module will fix the other modules not working as well? How do I force the update on the regular update channel?

 

It is not possible to force it. It's only possible to switch to the pre-release update channel or wait a bit until the module is downloaded from the regular update channel.

[mikeewa 0]

And this will allow me to upgrade without having to uninstall through safe mode, or the hips trick?

 

thanks!

 

Mike