The ekrn service failed to start / Patch Tuesday Windows Updates

[st3fan 8]

Hi everyone

I am busy testing/installing the latest Windows Updates on Windows Server 2016. After the reboot I noticed that ESET (Server Security 9.0.12013.0) did not start. I am setting this in the Event Viewer.

The ekrn service failed to start due to the following error: 
The service did not respond to the start or control request in a timely fashion.

 

Is anyone else seeing this?

 

[MarcFL 28]

Yes, it's prevalent - see Reddit below:
ESET Server Security for Microsoft Windows Server does not start with after December Windows Updates released yesterday (12-13-22)

 

Edited December 14, 2022 by MarcFL

[MarcFL 28]

You can try starting the Eset service in Windows services and if not, rebooting the server which helped a user on Reddit.

See: https://www.reddit.com/r/sysadmin/comments/zkmwww/patch_tuesday_megathread_20221213/j06ofmq/

 

Edited December 14, 2022 by MarcFL

[st3fan 8]

I was unable to start the ESET Service manually. Rebooting the server a second time helped.

Yesterday this affected every single server we updated (more than 10). Today I updated three more servers (same image, same OS, same ESET version) and I have not run into this problem so far. What changed? 

[Peter Randziak 1,170]

Hello guys,

thank you for your reports.

Can you please provide us with 1. Process monitor log from the boot time with advanced output enabled capturing events in the system during a boot, when the issue manifests and 2. output from the ESET Log Collector took afterwards?

Once you have the logs, please send a private message to me and @TomasP with the download details to check.

Thank you, Peter

[Marcos 5,277]

It appears that on system with low HW configuration the system is so busy installing the Windows update that the start of other services may time out. However, once the Windows update has been installed the issue should resolve automatically and services should not timeout any more.

[Peter Randziak 1,170]

Hello guys,

so the issue won't happen after the reboot, the logs can be obtained only if you enable the logging for the reboot, which will apply the Windows updates.

The issue is related to performance and as Marcos mentioned, the restart the system after few minutes as the patch will be processed so there would be no performance issues on the following reboot.

Peter

[Sec-C 6]

We also have ~40 win servers VMs with the same error message. And this is just the first batch. Is there a way to prevent this from happening? We have hundreds of VMs ready for automated win update deployment and we really can not handle them all individually afterwards.

[Marcos 5,277]

5 minutes ago, Sec-C said:

We also have ~40 win servers VMs with the same error message. And this is just the first batch. Is there a way to prevent this from happening? We have hundreds of VMs ready for automated win update deployment and we really can not handle them all individually afterwards.

Not without not installing the updates. The problem has been allegedly reported from other AV vendors too and Microsoft confirmed that lower hw configuration or other reasons causing services to load slower account for said issues.

Do you continue to experience the issue after installing the updates and rebooting the machines?

[Peter Randziak 1,170]

So I recommend to monitor the application of the Windows patches more closely than usually.

As mentioned above, the performance seems to be the key so if possible, make sure that the servers have enough processing power when the restart to apply the patches is to come.

The status of ESET security solution is being reported to the management console, so if it fails to start on some of the servers the restart after few minutes of run will resolve it.

Peter

[FRiC 10]

Well, the four servers I tested on (from Reddit thread above) are indeed older servers (E3-1225 V5 with 16 GB) so yeah, no way to speed them up. But since this is the first time this has happened so I hope it's only for this time. On my newer servers the updates worked fine.

[Trooper 67]

This happened to me on Server 2019 and Server 2022.  I have almost 60 VM's so not going to install Windows Updates until something other than yet another reboot fixes it.

[Sec-C 6]

2 hours ago, Marcos said:

Do you continue to experience the issue after installing the updates and rebooting the machines?

We tried starting the eset service manually on ~10 machines.  There where no more obvious errors afterwards. We have not tried additional reboots, since we need the machines online. Is there a way to make the agent try starting the security product?

[Mitchell 13]

3 minutes ago, Sec-C said:

We tried starting the eset service manually on ~10 machines.  There where no more obvious errors afterwards. We have not tried additional reboots, since we need the machines online. Is there a way to make the agent try starting the security product?

Managed to get the service started by run-command task from ESET PROTECT with command:

net start "ESET Service"

 this was only on a test VM though, uncertain if there are any unwanted side effects.  

Edited December 15, 2022 by Mitchell

[Steve_P 1]

I've done 10 servers so far, mix of 2012/2016/2019/2022 (lol) most with 8.0 but also some on 9.0. All rebooted with ESET starting correctly and update installed.

[heh 0]

We also have same issues - Win2016 and above. There ase no problems to start service later.

Guy, how to set some notification for this special event in Eset Protect to send mail message?

[MarcFL 28]

Question for Eset:  Why THIS month of Win updates?   Something must be different.

[Marcos 5,277]

13 hours ago, MarcFL said:

Question for Eset:  Why THIS month of Win updates?   Something must be different.

It's a question for Microsoft to determine what was different since ESET was not the only vendor affected.

The update generates a lot of disk activity inside C:\Windows\WinSxS\Catalogs by Cl.dll checking file hashes (generates more than 20,000 file opens).

[winstonsmith84 4]

On 12/16/2022 at 2:30 AM, heh said:

how to set some notification for this special event in Eset Protect to send mail message?

This. How do I get a notification set up to alert me about this? I don't see any built in notifications for this specific scenario.

[mcmcmc 0]

Create a ESET client task to autostart the eset service

net start "ESET Service"

Trigger daily at 6:00 or sth

[winstonsmith84 4]

Anyone? We have people updating servers and not checking for Eset afterwards and then servers are left to run unprotected until someone just happens to notice days later. We need a notification for when this happens. How do I set this up?

[Mitchell 13]

You can create a dynamic group with the following condition:

 

and then create a "dynamic group changes" notification for that: 

 

You could also trigger the previously mentioned "run command" task using a joined dynamic group trigger or scheduled trigger on that group to "auto heal" affected systems. (but as previously mentioned, A reboot is probably preferred) 

[kingoftheworld 10]

On 12/16/2022 at 12:22 PM, Marcos said:

It's a question for Microsoft to determine what was different since ESET was not the only vendor affected.

The update generates a lot of disk activity inside C:\Windows\WinSxS\Catalogs by Cl.dll checking file hashes (generates more than 20,000 file opens).

I don't know that it is a question for Microsoft to determine.  From the linked thread, only one other AV vendor was mentioned.  A bigger question is why is ESET not attempting to restart itself given the settings for the service is to always attempt to restart itself?

[Marcos 5,277]

It's Windows itself that restarts services in case of a crash provided they are configured so.

[kingoftheworld 10]

On 12/29/2022 at 11:37 AM, Marcos said:

It's Windows itself that restarts services in case of a crash provided they are configured so.

Fair enough on the restarting of the service. Based on your statement of Microsoft needing to look at this would indicate that ESET is not investigating this on their end?