Eset file security v7 on RDS 2016 high cpu termsrv

[Skynet 0]

Hi,

haven't found anyone else with these problems, but maybe eset can confirm?

We are using a Windows 2016 Server DC v1607 (all updates installed) with remote desktop services  aka terminalserver and Eset V7.

 

ESET Security Management Center (Server), Version 7.0 (7.0.577.0)
ESET Security Management Center (Web-Konsole), Version 7.0 (7.0.429.0)
ESET File Security  7.0.12014.0  / 7.0.12016.0 (I think I've tested both)

 

The "Remotedesktopservices" / termsrv generates a massive amount of cpu usage, I saw it using over 40% on a VM with 10x 2,6Ghz! It is impossible to work on the Server anymore. Once the Server is restarted everything works fine for some hours until "the bug" starts again. 

 

I've still uninstalled, reinstalled, used different policies and followed esets guidance for policies on terminalserver, nothing helped.

I've done some tests over the last 2 weeks without eset and everyting was fine, last week I decided to install eset file security again and give it a try and today we've got the same problem again. I have to leave it uninstalled, so don't ask me for any traces! Maybe you can confirm a known bug or test it on your own.

 

 

 

[Marcos 5,070]

Approximately how many users log into the server per minute? Does temporarily disabling the startup file check task run after user logon in Scheduler make a difference?

[Skynet 0]

Hi Marcos,

the amount of logins is no problem, we only have a maximum of 22 remote users, approx. 10-15 logged in during the working hours. If you ask for logins per minute it would be something under 1 (I still have checked the event log for suspicious logins).

Sorry, can't say someting about the "startup file check task", I had to uninstall eset. I might give the old V6 file security a try and see how it performs.

[Peter Randziak 1,130]

Hello @Skynet,

can you please share any news with us i.e how it went?

I'm willing to help you with the troubleshooting and even to check the logs with devs with a priority, if you install the latest version and the issue will persist.

Regards, P.R.

[Skynet 0]

Hi Peter Randziak,

At the weekend 9th/10th Feb. I've installed the latest V7, directly on Monday 11th the termsrv went up in massive cpu usage so I had to uninstall it again.

Last weekwend 16th/17th I've installed the latest File Security V6.5, until now it performs well.

I'm pretty sure it is an issue with V7, cause nothing else changed.

I would like to help troubleshooting but if I install V7 again, we will have a lot of employee that can't work for hours until all data is collected and of course I can't say when it happen so waiting for technical assistance in that case is very time critical.
So any idea?

 

[Peter Randziak 1,130]

Hello @Skynet,

1. so I would advise to install the v.7, with the default settings before the working hours, run manual modules update, do not apply any additional policies via ECMS/ERA.

I checked one very similar case you your's and they had deeper scanning set, thus consuming much more resources.

2. Disable "StartUp scan task after user logon" in scheduler

3. Have Process monitor prepared

Once the high load starts please record about a minute with the Process monitor (with advanced output enabled) and save it as native .pml log

4. Dump ekrn via the build-in option - Diagnostics - Create diagnostic dump

5. Collect the logs via ESET log Collector

6. You may restore the server's functionality, I assume it shouldn't take more than 3 minutes to collect the 3 logs.

In the ticket mentioned the load dropped after about 3 minutes after disabling the real-time file system protection.

 

In case it won't happen with the "StartUp scan task after user logon" in scheduler disabled, you may try to enable it and try to get logs with it.

 

Once you have the logs / any results please pack them, upload to a safe location and send me the download details via private message to check.

 

Please let us know any news regarding this.

Thank you in advance, P.R.

[Marcos 5,070]

Please carry on as follows:
- enable advanced operating system logging as follows:

- reproduce the issue with high CPU utlization by ekrn
- stop logging after a few seconds (10-20)
- gather logs with ESET Log Collector

[Skynet 0]

On 2/20/2019 at 1:17 PM, Peter Randziak said:

Hi, just a quick reply for the moment:

I've installed V7 again, but it was no fun without any policy, so I had to activate the terminalserver policy again. Also I've disabled the "StartUp scan task after user logon" in scheduler and excluded the termsrv from realtime-scan. For the moment V7 still works fine, so no logs for you.

Anyway your programmers should take a look for the difference between V6 and V7 that may cause the issue.

 

[Peter Randziak 1,130]

Hello @Skynet, 

good, please keep us posted.

There are quite many changes between v.6 and v.7 and moreover this issue might be caused by an updated module, currently we have about 50 of them,...

Once the issue reappears we should be able to find quite quickly what's wrong from the logs,... 

Thank you, P.R.

[Marcos 5,070]

Also it could be that the issue started to occur after some changes in the system itself (e.g. Windows updates). Without troubleshooting the issue and narrowing it down, it's impossible to tell what the culprit is. It could be that installing V6 with default settings wouldn't make any difference either.

In the mean time we've found out that ESMC may apply a weird configuration of the startup scan task run after user logon which causes a lot of more objects to be scanned than it's supposed. Unfortunately it is not clear yet how the particular user made such policy; by default correct flags and settings should be applied. That also means that installing Endpoint from scratch without applying a policy with Scheduler settings should prevent the issue from occurring.

[Skynet 0]

20 hours ago, Peter Randziak said:

Hello @Skynet, 

good, please keep us posted.

There are quite many changes between v.6 and v.7 and moreover this issue might be caused by an updated module, currently we have about 50 of them,...

Once the issue reappears we should be able to find quite quickly what's wrong from the logs,... 

Thank you, P.R.

Okay, today it starts again... you got a message with a download link from our Onedrive.

[Peter Randziak 1,130]

Thank you, I will check them and pass to the dev team, if needed.

Peter