The ekrn service failed to start / Patch Tuesday Windows Updates

[heh 0]

Hi guys, thank you for notification guide. It works! Tested on  M$ january 2023 patches - ekrn service failed to start too.

 

[Peter Randziak 1,163]

1 hour ago, heh said:

Hi guys, thank you for notification guide. It works! Tested on  M$ january 2023 patches - ekrn service failed to start too.

Thank you for keeping us posted and for a report of the issue with the latest patches too.

[srvdberg 1]

HI guys, i could reproduce it with latest windows updates:

After installation and reboot, the view from PROTECT:

Remediation through dynamic group and run command works! But as stated before, a reboot is preffered.

 

[jon1234 0]

We are experiencing the same issue with the Jan updates as well.  I was hoping it was the Dec ones only, but that does not appear to be the case.

 

Jon

[thae 12]

So this only happens after an installed update? Once you reboot or restart the service it's gone until the next updates?

[jon1234 0]

1 hour ago, thae said:

So this only happens after an installed update? Once you reboot or restart the service it's gone until the next updates?

That has been my experience.

 

Jon

[JimChev3 9]

I also experienced this issue first in December and only on Server 2019 systems (mix of virtual and physical servers) and again now in January, although it appears to have spread to 2016 servers as well. Server 2022 doesn't appear to be affected so far.

As with others, it's a timeout on ekrn during the reboot following application of the monthly updates. Either manually starting the "ESET Service" or a second reboot will re-enable functionality.

I would also add that I think it's rather disingenuous to simply point at Microsoft and wipe your hands of the issue. Even if it is something Microsoft did to cause it, at the very least, you should be working with them to figure out why and what can be done to resolve this, whether it's on their end or yours. All it takes is one moment of inattention on a customer's part and there's a public-facing server with no protection for who knows how long until the problem is noticed. I personally feel like if you have no intention of taking this matter up with them and getting it resolved, I'm going to soon be forced to find another vendor. This is too big an issue to just shrug your shoulders and live with indefinitely.

 

[Peter Randziak 1,163]

Hello @JimChev3,

we are in contact with the Microsoft regarding this, I hope we will be able to address it together.
During the December we were under an impression that it is a one time issue, sadly it is not.
I agree it is an serious issue, especially for the machines which are not managed i.e. the error is not reported to a central management console...

Peter

[JimChev3 9]

1 hour ago, Peter Randziak said:

Hello @JimChev3,

we are in contact with the Microsoft regarding this, I hope we will be able to address it together.
During the December we were under an impression that it is a one time issue, sadly it is not.
I agree it is an serious issue, especially for the machines which are not managed i.e. the error is not reported to a central management console...

Peter

Good to hear! That's all I needed to know. Looking forward to further news as it comes in. Thank you Peter!

[winstonsmith84 3]

I think maybe I see what could be a part of the problem. Our server environment has a mix of either 2 cores or 4 cores. What it seems like I'm seeing is that quad core servers aren't having the problem but dual core ones are. Is anyone else seeing this? Are these latest updates so CPU intensive that a dual core setup can't start Eset in time?

Marcos mentioned low hardware specs probably being a culprit and it seems like dual core CPUs can't get Eset started because of these updates.

[FRiC 10]

5 hours ago, winstonsmith84 said:

Marcos mentioned low hardware specs probably being a culprit and it seems like dual core CPUs can't get Eset started because of these updates.

For me it was old servers (Xeon E3-1225V5 4C with hard drives) vs. new servers (Xeon Silver 4215R 8C16T SSD). Fails on the old servers, and fine on the new servers. Doesn't matter how many cores or memory are assigned to the VM's. If speed is involved it could be the hard drives.

It only takes a few seconds to restart the service or reboot, but it'll be nice if this is resolved.

[David Parmentier 0]

I also confirm the same problem since the updates of December 2022 and January 2023
The service does not restart after Windows updates.
Restarting the server after the update fixes the problem...

This is not a "top" behavior...
It's a good thing the console reported the problem, otherwise we would have hundreds of servers without antivirus...

[Trooper 67]

Happened to me again this month as well.  Hope MS and ESET can get this fixed up for February updates.

[Sameer 2]

create a run command with the following :

net start "ESET Service" 

and apply the run task to the effected machine seems to be working in my case.

Edited January 19, 2023 by Sameer

[Peter Randziak 1,163]

5 hours ago, Trooper said:

Happened to me again this month as well.  Hope MS and ESET can get this fixed up for February updates.

Next released versions of ESET products should contain a workaround for this issue.

[Marcos 5,259]

Just now, Peter Randziak said:

Next released versions of ESET products should contain a workaround for this issue.

In particular, Microsoft dlls used by ESET that are located in the ESET install folder will be cross-signed to establish trust without verification using catalogues to prevent excessive reads from catalogues causing significant delays when starting the service until the threshold for killing the service is reached.

[FTL 2]

On 1/12/2023 at 11:16 AM, Peter Randziak said:

Hello @JimChev3,

we are in contact with the Microsoft regarding this, I hope we will be able to address it together.
During the December we were under an impression that it is a one time issue, sadly it is not.
I agree it is an serious issue, especially for the machines which are not managed i.e. the error is not reported to a central management console...

Peter

A mega serious issue for those of us with web servers too that this past Sunday morning were left in a completely unprotected state from 3am when my RMM applied the updates until i logged on at 9am to just health check things and found all my 2016 and 2019 servers glowing red in EPC.

Ive ended up getting PRTG to monitor the service to see if its running and alert me if it isnt as a fail safe.

Luckily nothing untoward happened during this unprotected time, its alarming though!

[Marcos 5,259]

You can work around the MS issue by increasing the timeout for starting services by creating a registry DWORD value named ServicesPipeTimeout with the value 60000 under HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet.

https://learn.microsoft.com/en-us/troubleshoot/windows-client/system-management-components/windows-trace-session-manager-service-not-start-event-id-7000

[Trooper 67]

On 1/19/2023 at 2:35 AM, Peter Randziak said:

Next released versions of ESET products should contain a workaround for this issue.

Thanks Peter this is good to know.

[FRiC 10]

I see that 9.0.12017.0 is released and states "IMPROVED: Protected antimalware service will not time out any longer during boot when Windows updates keep the file-system busy".

Thanks for getting this out so quickly, but should I install it and reboot the servers before next Patch Tuesday comes around, or can I do both at the same time to save one reboot?

[Marcos 5,259]

Yes, it needs to be installed and the machine rebooted prior to the next Windows update which could trigger verification of loaded dlls. As of Endpoint v9.1, Microsoft dlls embedded in ESET installers have been counter-signed so verification after Windows update by catalogs would not occur.

[karsayor 8]

18 hours ago, Marcos said:

Yes, it needs to be installed and the machine rebooted prior to the next Windows update which could trigger verification of loaded dlls. As of Endpoint v9.1, Microsoft dlls embedded in ESET installers have been counter-signed so verification after Windows update by catalogs would not occur.

Thanks. When is this version 9.0.12017.0 scheduled to be released via auto-update ? It should be done before the next patch tuesday IMO

Edited January 31, 2023 by karsayor

[Peter Randziak 1,163]

Hello @karsayor,

The release to web/repository  was on January 30 and will be followed by first phase of Auto-update/uPCU release (50%) on 13 February, while the last phase (100%) is scheduled for 27 February.
In order to speed up the process and receive update right after the web/repository release administrators can use "Check for updates" button available in the application GUI.

It is a new version and we speak of servers so for sure we need to proceed with caution...

Peter

Edited January 31, 2023 by Peter Randziak

[karsayor 8]

Hi @Peter Randziak

I agree that release schedule should be done with care if there are significants changes in the new version. I tought it was only a "bugfix".

I tried to click on the check for updates in GUI but it doesn't seem to find anything. 

Also, is there a way to trigger the Auto-Update/uPCU  with a task from management console ?

[st3fan 6]

I also just tried this on a test server. I don't receive the update either if I run a manual "check for updates". Auto-update settings are enabled/enforced.

 

What I also noticed and what I can't understand:

If I enforce the auto-update policies on all endpoints in a certain branch, some notebooks receive the update on the same day. If they don't receive it, then I usually do a manual "check for updates". However, there are always a few devices were none of this works. They simply don't receive the update, even though the same policies apply. I ran into this issue two weeks ago. No matter what I did, the auto-update did not arrive. And then yesterday morning, all affected devices suddenly received the update (I noticed the device restart required/recommended notification on the management console). There were no policy changes on my side, so I am not sure what causes this.