EFS for Linux issue - eset_rtp(ertp_wait_for_reply) - Ubuntu 21.04

[ph4ckvv3r 0]

Hi,

The company I work for is implementing ESET for Business to manage the assets. My machine is quite a robust machine (Ryzen 9 5900HX, 64 GB Ram, 3TB SSDs) and not slow at all but when I installed ESET (eea), a simple sudo su takes about 10 seconds to finally ASK for the password and another 20 to confirm. Gedit took incredible 30s. And grub-update close to 1m30s. Completely messed up performance on things that take miliseconds normally.

So, after editing grub to add eset_rpt=0, and waiting forever to have grub updated, when I rebooted I've got a huge list of timed out scanner responses from eset_rpt. So, I thought this was going to be fast, after the first minute I took my cellphone and started recording, when it was reaching 4 minutes of recording I've gave up and shutdown the camera and the notebook manually. Booted now with the real time protection off and machine is reusable again.

This error has been reported before but answer does not help as it mentions version EFS 7.2 upgrade and I'm in EEA 8.1.3 (which does not even include the efs binaries, but has scand)

Tried to run scand manually hoping I could see what could be causing the crash. No luck there.

 

ph4ckvv3r@ph4ckvv3r ~> /opt/eset/eea/lib/scand --version
/opt/eset/eea/lib/scand (eea) 8.1.3.0
ph4ckvv3r@ph4ckvv3r ~> /opt/eset/eea/lib/scand 
fish: “/opt/eset/eea/lib/scand” terminated by signal SIGABRT (Abort)
ph4ckvv3r@ph4ckvv3r ~ [SIGABRT]> 

 

Any ideas on how to solve this problem? Is there any dependency missing? How can I get a proper error from scand?

 

Cheers,

 

Pablo

[Marcos 5,074]

The problem seems to be that you are using an unsupported version of Ubuntu:

https://help.eset.com/efs/8.1/en-US/?system_requirements.html

Supported operating systems

ESET Server Security for Linux (ESSL) has been tested on the listed operating systems' latest minor releases. Update your operating system before installing ESSL.

64-bit Operating System	Secure Boot supported	SELinux Support	Note
RedHat Enterprise Linux (RHEL) 7	✔	✔	ESSL SELinux module policy installation requires an installed selinux-policy-devel package. To start the OS without ESSL SELinux module, use the eset_selinux=0 kernel parameter during OS boot.
RedHat Enterprise Linux (RHEL) 8	✔	✔
CentOS 7	✔	✔
CentOS 8	✔	✔
Ubuntu Server 16.04 LTS	✔
Ubuntu Server 18.04 LTS	✔
Ubuntu Server 20.04 LTS	✔
Debian 9
Debian 10	✔
SUSE Linux Enterprise Server (SLES) 12	✔
SUSE Linux Enterprise Server (SLES) 15	✔
Oracle Linux 8	✔ (stock kernel only)		If the Unbreakable Enterprise Kernel is used, the kernel-uek-devel package must be installed manually. In this case, Secure Boot is not supported.
Amazon Linux 2

[kurco 19]

Hi ph4ckvv3r,

as mentioned in previous comment, there is no official support for Ubuntu 21.04. But still it could be caused by something else, not necessary by unsupported distribution. Are you using some specialized software on your machine? something that generates huge amount of events?

Linked issue you are mentioning is about Eset File Security, both products are sharing the same scanning core. efs is not a binary, but product name. Scand can't be executed manually, because it only reacts on requests from other authorized services.

Correct link for supported distributions for EEA: https://help.eset.com/eeau/8/en-US/?system_requirements.html

Is it possible to collect logs and attach them?

thanks,

Peter

 

Edited July 27, 2021 by kurco

[ph4ckvv3r 0]

Hi Marcos and Kurko

If you tell me how to collect error logs, yes, I can collect them and share, although I may need to filter some of it depending on the info on them. The logs I could find are only .dat with some binary info on it with also some clear text. I couldn't find one with my specific error.

As I said, only option I had so far was to disable the kernel module. 

About software I run, well, this machine has to run multiple projects and we run a lot of containers for different projects. So, really heavy load on files.

About the version of Ubuntu: 20.04 (20.11 too) don't support my hardware,so, I can't downgrade.

I do, however, have other softwares running that are targeted to 20.04 and I could manage to get it running on 21.04 just by adding the proper libraries versions to it. If I could spot why the scanner process times out to send response to eset_rpt (report?) than it would be easier to fix, but I don't even have errors on the logs I've found, probably because I don't know where to look at (/var/log/eset had an events.dat but no error.dat).

Please, tell me what to do to generate those logs and I will gladly share them.

Cheers,

Pablo

Edited July 27, 2021 by ph4ckvv3r

[ph4ckvv3r 0]

Hi Peter/Kurko

Being a bit off-topic now: there is an odd information in the link you sent for requirements. That page says:

Secure Boot Secure Boot is not supported.

And at the same time on the side menu you have an item named Secure Boot that shows to add the kernel signature to UEFI.

https://help.eset.com/eeau/8/en-US/system_requirements.html?secure-boot.html

[ph4ckvv3r 0]

Hi,

Found interesting information here

Seems that problem is with kernel 5.11, even if you use Ubuntu 20.04, and, for my luck 21.04 uses 5.11...

 

ph4ckvv3r@ph4ckvv3r ~/Downloads> uname -a
Linux ph4ckvv3r 5.11.0-25-generic #27-Ubuntu SMP Fri Jul 9 23:06:29 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

 

I guess I will need to downgrade to Ubuntu LTS, my problem is that when I tried before, it couldn't boot the note as hardware was too new for 20.04. 😞 Maybe I will try one of the other distros, as we run containers for everything, the hosting version of Linux does not matter much (except for new hardwares).

Cheers, if I get luck I'll let you know.

[ph4ckvv3r 0]

Hi again, found the link for how to get the logs. But the machine gets so slow it's really hard to use. I will try it tomorrow.

Here is the link in case anybody asks for it

https://help.eset.com/eeau/7/en-US/realtime-protection-cannot-start.html?collect-logs.html

[kurco 19]

Hi ph4ckvv3r,

Firstly, thanks for noticing documentation inconsistency about secure boot, we will fix this right away.

I understand, that it's really hard to generate logs, when machine is constantly freezing. Probably the best way is to disable RTP, this can be done during boot in Grub, you don't need to edit it and update, when OS is already running. Check this, it's documentation for EFS, but it's the same on EEA: https://help.eset.com/efs/8/en-US/disable-realtime-protection-at-boot.html?zoom_highlightsub=eset_rtp

What kind of container are you using? Docker? Maybe they could be the source of extensive events sending into our scanning core. Perhaps you could try to add some "Performance exclusion" on paths, where containers file-systems are located and events generated there will be ignored by our scanning services.

I have deployed an Ubuntu 21.04 virtual machine and installed EEA. No timeouts occurred, but I will try to install docker there and also some other software, to see if I can replicate somehow yours issue. 

Regards,

Peter

 

[ph4ckvv3r 0]

Hi Peter,

Last night I've gave it another go and had the following situation (which also explains why I didn't notice that when we setup on Friday, but only on Monday when I booted the machine):

1 - I've installed. No issues. Reboot

2 - First reboot: machine worked just fine. Can't remember for sure now if I added the license on step 1 or 2, but I'm pretty sure it was on 1

3 - Second reboot: slowness started and I badly could login using the terminal to redefine grub boot, plus the fact that grup update took nearly 2 minutes to run.

I can't get much sense of why this happens on this order, but this would probably be the reason why you didn't notice slowness on the VM: did you reboot the VM? How many times?

Other than that, machine at the booting stage has very little being run, and the boot itself is quite slow. One important detail that may (or may not?) make any difference: I'm using an LVM encrypted volume. A colleague also reported having issues on his setup, and in his case he uses only Debian, uname -a output is Linux arlen 5.10.0-8-amd64 #1 SMP Debian 5.10.46-2 (2021-07-20) x86_64 GNU/Linux, but he says that it lasts for like 5 minutes on every boot and then it works "normally". My problem didn't go away after an hour trying to get what was the source of it. 

 It's being quite hard to spot the problem, so, I will try later to setup Ubuntu 20.4,2. That version may have support for the hardware.

I'll keep you posted. That is not the solution but it would solve my problem.

Cheers,

Pablo

 

[kurco 19]

Hi Pablo,

thanks for investigation. I will try to setup new VM with encrypted LVM and follow your steps. One more question, are you using EDTD feature? is it enabled in your product?

Regards,

Peter

[ph4ckvv3r 0]

Hi Peter,

I cant' be sure if I have EDTD, it's not listed as installed but I'm not sure this is the right place to check this.

Cheers,

Pablo

 

[Marcos 5,074]

You probably cannot check it on Linux directly but you'd see EDTD in client detains in the ESET PROTECT console:

[ph4ckvv3r 0]

Marcos and Peter,

Seems I've got something missing here then. I've installed the EEA Agent, then I've installed the Antivirus. Only "Eset" gui I have is the one that provides the info on the previous screenshot (which reads on its title as "Eset Endpoint Antivirus").

What am I missing? Would a missing part of the solution be the reason for this odd "blocking" behaviour from the eset fs kernel module? Why does that only happens to me after the second reboot? (please, remember I've install it, then it requires a reboot, everything works, but on the next reboot it starts to fail oddly for no apparent reason).

Also, when this happens I've got an infinite number of messages trying to shutdown the machine (got it after I added the eset_rtp=0 to grub, so I could work that day). Check the screenshot (I have the video, but it's almost 4 minutes of me wondering how long it would take till it ends - I don't know, I forced the shutdown after 4 minutes so I could boot with eset_rpt=0 to get the machine back to work).

Cheers,

Pablo 

Edited August 1, 2021 by ph4ckvv3r

[kurco 19]

Hi Pablo,

Could you please clarify what do you mean by "EEA Agent"? do you mean Management Agent? But Eset Endpoint Antivirus does not need anything else to by functional. Also I'm not aware, that Eset Endpoint Antivirus installation needs reboot.

From that screenshot you have attached, only thing that is clear, is that our scanning service is overloaded by events from eset_rtp kernel module. But most probably paths seen on that screenshot are only a fraction of events.

Also what about container you have mentioned before? all of them are starting automatically after boot? have you tried to add performance exclusions on path used by containers from management console? I have suggested it some time ago in one comment.

Thanks,

Peter

[ph4ckvv3r 0]

Hi Peter,

The problem is that this machine compiles images that go to client's environments, so there is no "exclusion path" that I could allow without any compromise to the security. This machine is not supposed to be slow, all the disks are quite fast ssds, and there is 64GB of RAM which should be more than enough for the scans not to get "overloaded".

About what I call "EEA", well I call it the only thing I can find in the /opt/eset folder. 🙂

How can I query ubuntu to get the proper names of what I have installed here?

eea/now 8.1.3 seems to be the only matching package.

If you tell me where to set exclusion paths, I can give it a try.

Cheers,

Pablo

[SUPERHOT 0]

I'm having the same issue. Just installed ESET Endpoint on Ubuntu 21.04 and my machine has slowed to a crawl. Even simply opening the terminal can take 15 or 20 seconds. Looking in the process monitor, nothing is using CPU or RAM. But the whole computer is unusable. Please help.

[SUPERHOT 0]

Okay, I found something. I had Secure Boot enabled in BIOS/UEFI but it didn't seem to be configured. I went ahead and disabled it, and the problems went away. Now my computer is at full speed. This also might explain why the issue didn't manifest in a virtual machine. That said, I would like secure boot to be working, but I can at least use my computer for now while we figure this out.

[ph4ckvv3r 0]

Thanks for the info. Never had the secure boot enabled here for quite a simple reason: RTX 3080. Every time there is an driver update or a kernel release, it's a pain to get it "on" properly as you need to disable secure boot, and then resign the whole thing: not worth the effort. However, there was an update today. I will try again. But I'm betting this is a problem with LUKS and the EFS. 

[ph4ckvv3r 0]

Hi Peter,

Answering a previous question: dynamic threat is on

Trying to get rid of it now. 😄 

Cheers

[ph4ckvv3r 0]

Hi all!!!

SOLVED!!! After removing the policy I still had problems to get the scan working. I've triggered it remotely and it was working, so RTP was the only problem. Well, after so many attempts and even resetting the BIOS I've forgot to set secure boot as off.

It is working now flawlessly. Peter, you were right, last week while we were trying to get it working here, someone saw my "red strip" and trying to help clicked the "ENABLE" on EDTD. Thank god I could remove that.

Cheers,

Pablo

[kurco 19]

Hi Pablo,

thanks for investigation and info sharing. Still I want to investigate this further, could you please clarify one thing for me? our support of secure boot is only manual sign with imported certificate, but no changes in kernel module, so I'm not sure how disabled secure boot could help with performance. With enabled secure boot, did you signed manually our module according to this page https://help.eset.com/eeau/8/en-US/installation.html?secure-boot.html.

Disabled EDTD could be reason of better performance, EDTD is an additional layer of security, therefore it could impact performance.

Thanks,

Peter

Edited August 5, 2021 by kurco

[ph4ckvv3r 0]

Hi Peter,

Seems I was wrong. Today the computer started to be slow again. The secure boot was just because it was not starting. Then it started, was fine for a week but now again, on Monday, super slow. Last time was the same: after the weekend, machine become very slow. Maybe it partied to strong over the weekend? 🙂

Anyway, today, starting week, really slow. I will give it a go to let scan runs as my colleague reported. "After 5-10 minutes it comes back to normal". I'll report again tomorrow morning. This weekend I will try not shutting it down and see if Monday is slow again.

I've also have added a restart to eea every 15 minutes. And seems that it just worked and made the machine "fast" again. Again, a tip I've received from my friend: restart the service and morning goes back to "normal" for a while. Oddly, it seemed to work just now.

Cheers,

Pablo

[SUPERHOT 0]

The slowness came back for me too. I'm not sure if secure boot was related or just a red herring.

Sometimes if I restart the computer will be fine, other times it is slow as dirt. I even uninstalled ESET and it was still slow until I reset bios and changed a few things.

I've been running without ESET for a few days and my machine is stable and fast. Not sure what the real problem is but the app is unusable in this state. I will wait to hear when this is fixed before trying again.

[ph4ckvv3r 0]

SUPERHOT,

 

so far my solution has been to have a cronjob to reestart eea service every 15 minutes. Sometimes the machine boots slow, I go for a cigarrete, coffe, bath, and when I'm back it's normal. It's really random problem, can't get a track of what is slowing down the machine. 

Amazing enough, running HTOP shows all my cores using less than 2%. But I can badly run something in terminal. All of the sudden (every 15 minutes) service is restarted, cores go really high and back to normal in seconds. 

Add this to /etc/crontab and that's the "fix" you can do at the moment.

*/15 * * * *     root    systemctl restart eea
 

[LesRMed 23]

15 hours ago, SUPERHOT said:

I even uninstalled ESET and it was still slow

This tells me that your issue is not related to ESET.