EFS causes drbd/pacemaker setup to crash

[mike.miyw 0]

Recently i migrated from esets 4.5.15 to EFS 7.1.247 on one of my 2 HA servers that are doing HA via drbd and corosync / pacemaker.

After the migration i got rung up at night as the EFS box claimed to see timeouts on the internal network connection and was not able to handle the HA anymore (so it tried to shift the services over to the other node).

Trying to login via SSH i also revceived a timeout. Only thing that helped was a hard reboot of the box.

The OS is a debian 10 (latest).

 

Going into the forensics, i got the following loglines:

 

This is what i got as first indicator of a problem

Dec  2 02:33:11 miyw02-18 pacemaker-execd[12819]:  warning: mysql-daemon_status_15000 process (PID 31328) timed out
Dec  2 02:33:11 miyw02-18 pacemaker-execd[12819]:  warning: mysql-daemon_status_15000:31328 - timed out after 15000ms

 

This is the corresponding kernel log 

Dec  2 02:33:06 miyw02-18 kernel: [149346.294602] eset_rtp: wait for scanner reply timeout, path: /tmp/#sql_7d43_0.MAD, event: CLOSE, pid: 32067
Dec  2 02:33:06 miyw02-18 kernel: [149346.550679] eset_rtp: wait for scanner reply timeout, path: /var/log/pacemaker/pacemaker.log, event: CLOSE, pid: 31328
Dec  2 02:33:08 miyw02-18 kernel: [149348.598543] eset_rtp: wait for scanner reply timeout, path: /opt/splunk/_deployments/splunk_8.0.0.0/bin/splunk-optimize, event: OPEN, pid: 31329
Dec  2 02:33:09 miyw02-18 kernel: [149349.878748] eset_rtp: wait for scanner reply timeout, path: /var/log/pacemaker/pacemaker.log, event: CLOSE, pid: 31336
Dec  2 02:33:09 miyw02-18 kernel: [149349.878758] eset_rtp: wait for scanner reply timeout, path: /var/log/pacemaker/pacemaker.log, event: CLOSE, pid: 31337
Dec  2 02:33:09 miyw02-18 kernel: [149349.878766] eset_rtp: wait for scanner reply timeout, path: /var/log/pacemaker/pacemaker.log, event: CLOSE, pid: 31334
Dec  2 02:33:09 miyw02-18 kernel: [149350.134795] eset_rtp: wait for scanner reply timeout, path: /var/log/pacemaker/pacemaker.log, event: CLOSE, pid: 31341
Dec  2 02:33:09 miyw02-18 kernel: [149350.134805] eset_rtp: wait for scanner reply timeout, path: /var/log/pacemaker/pacemaker.log, event: CLOSE, pid: 31339
Dec  2 02:33:10 miyw02-18 kernel: [149351.158848] eset_rtp: wait for scanner reply timeout, path: /var/log/pacemaker/pacemaker.log, event: CLOSE, pid: 31343
Dec  2 02:33:10 miyw02-18 kernel: [149351.159260] eset_rtp: wait for scanner reply timeout, path: /var/log/pacemaker/pacemaker.log, event: CLOSE, pid: 31344
Dec  2 02:33:11 miyw02-18 kernel: [149351.670688] eset_rtp: wait for scanner reply timeout, path: /run/icinga2/icinga2.pid, event: CLOSE, pid: 911
Dec  2 02:33:11 miyw02-18 kernel: [149351.670723] eset_rtp: wait for scanner reply timeout, path: /usr/lib/nagios/plugins/check_apt, event: OPEN, pid: 31348
Dec  2 02:33:15 miyw02-18 kernel: [149355.254788] eset_rtp: wait for scanner reply timeout, path: /opt/splunk/data/var/lib/splunk/kvstore/mongo/diagnostic.data/metrics.interim.temp, event: CLOSE, pid: 22511

 

anyone experiencing similar issues ? 

 

Edited December 2, 2019 by mike.miyw

[Peter Randziak 1,130]

Hello @mike.miyw,

please provide us with the logs collected by collect_logs.sh script available at /opt/eset/efs/sbin/  https://help.eset.com/efs/7/en-US/?collect-logs.html 

and output from strace so we can have it checked.

You can upload the logs to a safe location and send me a private message with download details.

Peter

[Acc-Eset 0]

Hi,

 

You found a solution ?

I had the same problem on a Debian 10. I removed the last version of EFS and reinstall the old one, it's ok for now.

Thx

[Peter Randziak 1,130]

Hello @Acc-Eset,

well running the ESET security products in Chroot is not supported, even if you do not notice any issues with it that does not mean it actually works properly.

 

Chroot is a feature that can be "enabled" for selected processes. According to chroot(2):

In particular, it is not intended to be used for any kind of security purpose, neither to fully sandbox a process nor to restrict filesystem system calls.

So it seems the way to resolve the issue is to configure the Chroot to exclude the ESETs processes.

 

Please share the results with us, 

Peter

[Acc-Eset 0]

Hello,

I'm not sure i understand your answer.

I installed Eset security products in Debian 10 normally, like a Debian 9 and all the others servers.

The problem appeared after a reboot. So i wasn't in chroot, or maybe i misunderstood ?

Thanks