Idle-stat scan causing high memory pages

[j-gray 37]

Is there any way to dial back the idle-state scanning? It frequently triggers excessive memory paging, specifically on our file servers.

We also find that when an infected file is present, idle-state will report it up to 5 or 6 different times in a 24-hour period, which seems to indicate that it is  scanning  the entirety of a 1TB drive multiple times per day. We'd like to dial this back, as well if at all possible.

TIA

[Marcos 5,259]

5 minutes ago, j-gray said:

It frequently triggers excessive memory paging, specifically on our file servers.

Please provide a screenshot for illustration.

5 minutes ago, j-gray said:

We also find that when an infected file is present, idle-state will report it up to 5 or 6 different times in a 24-hour period

Detected threats should be cleaned and not detected repeatedly during next scans. Please provide logs collected with ESET Log Collector.

[j-gray 37]

In the detection instances they were all malware within an archive; one was an email within a very old .mbx file and the other within an .iso file. My understanding (and according to support), because it's an object within an archive, it's unable to be remediated by ESET.  My bigger concern though is what seems to be frequent, repetitive idle-state scanning.

And unfortunately I don't have a screenshot of memory usage. I get alerted by our monitoring application and typically miss the occurances. Because it's random in nature I can't predict when it will occur, either. It does coincide with times that idle-state is scanning. Additionally, it doesn't occur when idle-state scanning is disabled.

[Marcos 5,259]

Malicious attachments should be cleanable in MBX files. Is the MBX file the email storage that Thunderbird uses? You could also consider excluding MBX from being scanned by the idle-state scanner.

As for ISO, if it contains only a malicious file, the whole archive should be deleted/cleaned automatically.

[j-gray 37]

On 7/29/2022 at 10:51 PM, Marcos said:

Malicious attachments should be cleanable in MBX files. Is the MBX file the email storage that Thunderbird uses? You could also consider excluding MBX from being scanned by the idle-state scanner.

As for ISO, if it contains only a malicious file, the whole archive should be deleted/cleaned automatically.

Interesting. I'm certainly not seeing that behavior after multiple detections over multiple days. In all cases we see, "Action error - 
Unable to clean". This particular mbx file appears to be a very old and disused Outlook Express file.

And in fact, ESET support (#00393240) advised that the only way to clean it successfully was to enter the inbox and delete the suspect message.

[j-gray 37]

Updated: I disabled idle-state scanning and I get no more high memory utilization errors.

It's clear to me that the idle state does frequent and repeated scanning of file shares and causes high memory utilization.

I'd open a support case, but I really don't have the patience to go through Level 1 support. If anyone at ESET cares to pursue this and can put me in touch with Level 2 support, I'd be happy to collect whatever information is needed to help resolve this.

[Marcos 5,259]

You could provide some screenshots to illustrate the issue and we'll try to reproduce it ourselves.

[j-gray 37]

55 minutes ago, Marcos said:

You could provide some screenshots to illustrate the issue and we'll try to reproduce it ourselves.

The challenge is that it happens at random times whenever idle-state runs. From the detection logs, we can see that when malware is present, it gets detected 5-6 times daily, which indicates that idle-state is scanning the entire 2TB drive frequently.

We use Microsoft System Center Operations Manager, which alerts to excessive memory consumption. We usually see the alerts after the issue has occurred and can't manually reproduce the issue. In addition, this alert/alarm state is generated in Vmare vCenter, where these virtual machines reside.

I believe the easiest way for you to reproduce it would be with a vm of a Windows 2102 R2 file server with 4GBs of RAM,  2TB's worth of small files and idle-state scanning enabled.

[itman 1,746]

Verify that "Smart Optimization" is enabled in ThreatSense settings for Idle-Scan settings section. Also verify that "Run background scans with low priority" is also enabled in the same section.

Edited August 22, 2022 by itman

[Marcos 5,259]

Scanning gigabytes of data several times a day is a bit overkill to say the least. I'd recommend disabling scanning of archives, sfx archives and email files in the Idle-state scanner setup, or at least set a reasonable size limit for scanned objects, e.g. to 10 MB.

[j-gray 37]

21 hours ago, itman said:

Verify that "Smart Optimization" is enabled in ThreatSense settings for Idle-Scan settings section. Also verify that "Run background scans with low priority" is also enabled in the same section.

Yes, those settings were selected

[itman 1,746]

On 8/22/2022 at 4:55 PM, j-gray said:

The challenge is that it happens at random times whenever idle-state runs. From the detection logs, we can see that when malware is present, it gets detected 5-6 times daily, which indicates that idle-state is scanning the entire 2TB drive frequently.

It appears that when Eset detects the presence of malware, it is reverting to full scan mode.

[j-gray 37]

2 minutes ago, itman said:

It appears that when Eset detects the presence of malware, it is reverting to full scan mode.

I don't believe that's the case, as the detection source always indicates 'idle-state scan'.

I have a feeling that idle-state just runs continuously, hence the repeated detections and high memory utilization.

[Marcos 5,259]

Most likely detected threats remain uncleaned because they were found in an archive or mailbox. As I have already mentioned, if you could disable archives, sfx archives and email files in the Idle scan setup so that these are not scanned, or at least set a reasonable size limit for scanned objects, e.g. 10 MB.

[j-gray 37]

10 hours ago, Marcos said:

Most likely detected threats remain uncleaned because they were found in an archive or mailbox. As I have already mentioned, if you could disable archives, sfx archives and email files in the Idle scan setup so that these are not scanned, or at least set a reasonable size limit for scanned objects, e.g. 10 MB.

@Marcos Yes, that is understood. My point here is regardless of memory utilization, scanning a large volume 5-6 times every 24-hours seems excessive. 

[itman 1,746]

I really believe the issue here is idle-time scan mode. By design, this mode performs continuous scanning. However, the Smart Optimization feature should exclude files previously scanned unless there was a change in the file; i.e. different file hash.

Now let's add the VM element into the picture. When the VM is initialized, files are recreated within it. In essence, these a new never seen files as far as Eset is concerned and its going to scan the files again. I would exclude VM files from idle-time scanning and instead run a scheduled scan on them.

[Marcos 5,259]

Executables, dll and sys files are not re-scanned unless modules have been updated in the mean time and the files are not whitelisted.

[j-gray 37]

1 hour ago, itman said:

I really believe the issue here is idle-time scan mode. By design, this mode performs continuous scanning. However, the Smart Optimization feature should exclude files previously scanned unless there was a change in the file; i.e. different file hash.

This would make sense, though with Smart Optimization enabled it was detecting malware in the same static .iso and .mbx file 5-6 times every 24-hours, so clearly not the case.

We can certainly add a bunch of exclusions, but that would seem to defeat the purpose of a scan. To me, the underlying issue is the frequency of the idle-state scan and the resources it consumes.

[Marcos 5,259]

4 minutes ago, j-gray said:

This would make sense, though with Smart Optimization enabled it was detecting malware in the same static .iso and .mbx file 5-6 times every 24-hours, so clearly not the case.

Archive scans are not cached. Basically only exe, dll and sys files are.

[itman 1,746]

1 hour ago, j-gray said:

This would make sense, though with Smart Optimization enabled it was detecting malware in the same static .iso and .mbx file 5-6 times every 24-hours, so clearly not the case.

By "static" I assume you mean this .iso file is one whose contents do not change?

It appears Eset for some reason can't remove the malware within the .iso file. How about copying the .iso file to external media. Then copy it to a stand alone device not connected to the corp. network. Mount the .iso file and use Eset to scan and remove any malware found. Dismount the .iso file, copy the cleaned .iso file to external media, and then reload it to its original location.

[j-gray 37]

On 8/24/2022 at 12:32 PM, itman said:

By "static" I assume you mean this .iso file is one whose contents do not change?

It appears Eset for some reason can't remove the malware within the .iso file. How about copying the .iso file to external media. Then copy it to a stand alone device not connected to the corp. network. Mount the .iso file and use Eset to scan and remove any malware found. Dismount the .iso file, copy the cleaned .iso file to external media, and then reload it to its original location.

Yes, correct. The files that were being flagged were very old, static, not changing, not in use.

I handled the malicious files some time ago. My point with the multiple detections per day of the same static files is that it indicates excessive scanning by idle-state. Which is also causing high memory utilization.

I've disabled idle-state scanning and no longer get the multiple daily memory alerts.