Scanning take too long to finish

[Microbe 5]

Hi ESET Team, 

 

The reason why i posted (copy and paste ) the concern of our client is because he was doing testing related with this concern, We hope that you will be able to help us give them the recommendation and answer that he need.

 

 

New information

 

Meanwhile, I’ll provide answers to their questions, but in doing some related tests, I have some new significant information as well. Please bear with me as I go into some details, and then please pass on what I’ve found to ESET HQ. It might be related to the known bug that ESET said they’re aware of in their email of 26/3 (see the message thread below).

 

Firstly, to answer ESET HQ’s questions, there’s a lot of inconsistency between scans, but the latest results of a full PC Smart Scan, with Archives supposedly disabled, and Smart Optimisation supposedly enabled, showed it scanned 4.8 million objects and took 15.5 hours. This was also with duplicates removed, which I’ll go into in more detail below in “Scanning Duplication” because scans are much longer when files are scanned multiple times, as you’d expect. It’s these sorts of numbers I haven’t been able to understand (especially when a second scan done soon after will look fairly similar, apparently not reduced by Smart Optimisation), and is making scans non-viable. From what I’ve seen just lately, the problem isn’t caused by any user files, and may not be related to any files at all.

 

This is what I’ve found… As ESET HQ suggested, I tried to look at some folders that I thought might have been contenders for taking an extra long time, and this showed something very interesting. When I scanned three selected file sets with Smart Scan settings, they were surprisingly quick, but what I noticed was that the reported exceptions that always come up in my weekly Smart Scan results for these particular file sets (like “Error reading archive” and “Archive damaged” and so on), didn’t get reported. That’s probably what should be expected from a Smart Scan, but it’s not what I actually see in my weekly Smart Scans. I was suspicious about this, so I experimented with a hunch … I did a re-scan of these same file sets with the In-Depth Scan setting. I’ve never used that in any of my scans, or in the weekly scans, but it behaved just like my weekly scans. It was much slower, and it DID report all the same exceptions that I see in the results for these same file sets in my supposed Smart Scan weekly scans.

 

It'll probably help if I give a specific example. This is a message that comes up in a scan result from an In-Depth Scan: “C:\Users\dw_001\Documents\Business\Insight Resources\Community Websites\templates\Hitech\_vti_cnf\Hitech.zip » ZIP » - archive damaged”. This doesn’t come up in a Smart Scan of a file set that includes that file, as expected because that scan type excludes archives, but it DOES come up in my weekly scans, which are Smart Scans and supposedly exclude archives. How can that be?

 

It makes it seem as though the weekly scheduled scan is behaving very much like an In-Depth Scan, NOT as a Smart Scan.

 

I’d previously reported on 10/4 there was a problem with Archives being scanned even when they’ve been disabled in the scan type (e.g. Smart Scan). Unfortunately ESET HQ didn’t respond to that, but it turns out the problem could be that the Smart Scan selection is strangely being ignored and something like an In-Depth Scan is being performed instead. That would explain why I’m not seeing any benefit from Smart Optimisation as well I guess, because In-Depth Scans don’t include Smart Optimisation.

 

Tests just with the three selected file sets show a massive difference between an In-Depth Scan and a Smart Scan in the speed of the scans, the numbers of objects scanned, and the results reported. The only trouble is, from my real-world experiences, it looks like I haven’t been getting the benefit of a Smart Scan in my weekly scans. It always seems to revert to something that looks very much like an In-Depth Scan instead, and takes forever.

 

Even more oddly though, while the weekly full PC Smart Scans are consistently behaving in this problematic way, unfortunately I haven’t been able to duplicate the problem with a smaller file set, so that’s extremely confusing. The documented results are very clear, but it’s totally illogical, and computer programs are supposed to be logical, right?

 

Can anyone shed any light on all of this? Why/how could a Smart Scan actually act more like an In-Depth Scan, but only in some circumstances and not others? Before you say that can’t happen, please remember I have scan result documentation that shows it has happened.

 

Scanning Duplication

 

Secondly, what I alluded to above regarding “duplicates removed”, is that there seems to be a software bug, or at least a programming oversight. I raised it in my email of 10/4, and attached proof that shows if there are folder shortcuts selected in the scan file set, those files are re-scanned in addition to the original file set, so the same files are scanned multiple times. ESET’s 11/4 reply to that follows, and my new response to that is in blue …

• The existence of file or folder shortcuts shouldn’t lead to ESET re-scanning all the same objects, should it Gil?
  ESET HQ: With Smart optimization it's likely that a file referenced by a shortcut would not be scanned again if it has already been scanned.
  
  ESET HQ seems to be answering this by saying what the program should do, but that’s ignoring what I’m saying it does do. I sent examples from scan results that readily show that the same files were scanned as many as four times, simply because there were shortcuts to large folders. (That file is attached again.) I agree with ESET HQ, that certainly shouldn’t happen, but it does. The problem should be easy to duplicate, so please do that and then hopefully it can be addressed. I get around the problem now by removing the folder shortcuts from scan file sets, but surely that shouldn’t be necessary as ESET HQ’s reply above would also suggest.

In conclusion

 

I expect I may have to abandon weekly scans unless someone is able to make some sense of what I’ve found, even though abandoning scans would be a concern because past scans have actually found malware that wasn’t picked up by Real-time File System Protection.

 

I’ve spent a lot of time trying to understand some apparent idiosyncrasies and unpredictability of ESET scans so I can use them productively, and I know there’s again a lot of information in this email. Hopefully with all this information, your own tests and analyses will now be able to identify causes and resolve the issues I’ve come across … for everyone’s benefit.

 

 

NOTE : I also attached the file that we received

 

Cheers, 

Gil 

scan examples.pdf

[Marcos 5,085]

Please provide logs collected with ESET Log Collector from the machine in question so that we can check ESET's configuration.

[Microbe 5]

Thank you ESET Team, 

I will send the require logs, once I get it.

Thank you very much!

 

Cheers,

Gil

[Microbe 5]

Hi ESET Team, 

 

See the below information  from our client, it seems that he do not need to send the logs for us , but he has additional question, check his response with blue color.

 

Thanks , however I can save ESET HQ time and effort looking into logs. I’ve re-checked scans since the upgrade to v17.1.9.0 and the main issue seems to have been fixed. On the device I’ve been using for tests, I don’t think I’m seeing an improvement from multi-threading, but significantly it does now seem to be doing the type of scan that’s actually been selected. That means a Smart Scan isn’t doing an In-Depth Scan any more, thankfully! Instead of scanning 4.8 million objects as a “Smart Scan”, taking 15.5 hours, including archives etc when it shouldn’t, now it scans 1.4 million objects and takes around 4 hours. I can live with that.

 

So … I’m happy to move on from that problem for now, and just keep monitoring. I’ll get back if something goes amiss.

 

A few things are still unresolved though …

1. I’m still confused about Smart Optimisation. On a small file set, a Smart Scan done after the software version update took 19 minutes, and then a second scan of the same files straight after that took just 3 seconds! so a clear benefit there, and that’s what I’d expected from Smart Optimisation … but two Smart Scans of the whole system done back-to-back using v17.1.9.0 showed just a 20% improvement between the scans in time taken, and almost no change in numbers of objects scanned. So, that inconsistency is very hard to understand if indeed Smart Scans “prevent files that have not been changed since last scan, from being scanned AGAIN”.
   Any clarification you can provide about this will be appreciated.
2. On the other issue, the problem of folder shortcuts leading to scanning duplication is still happening with v17.1.9.0, so this is still outstanding:
   
   The existence of file or folder shortcuts shouldn’t lead to ESET re-scanning all the same objects, should it Gil?
   ESET HQ: With Smart optimization it's likely that a file referenced by a shortcut would not be scanned again if it has already been scanned.
   
   ESET HQ seems to be answering this by saying what the program should do, but that’s ignoring what it does do. I sent examples from scan results that readily show the same files were scanned as many as four times, simply because there were shortcuts to large folders. (That file is attached again.) I agree with ESET HQ, that certainly shouldn’t happen, but it does. The problem should be easy to duplicate. I get around it now by removing large folder shortcuts from scan file sets, but surely that shouldn’t be necessary as ESET HQ’s reply above would also suggest.
3. And, also regarding file sets, one other recent observation is intriguing as well – I noticed a recent scan was spending quite a bit of time going through many objects in “C:\System Recovery\Repair\Backup\ …”. When I checked that hidden directory, File Explorer said access to it was restricted, but it was empty anyway, so …
   Why/how can many, many objects still be scanned in that directory if it’s empty?
   And can I just exclude that directory without any issue?

Thanks,

Darryl

updates _scan examples.pdf

[Marcos 5,085]

Please provide the logs as requested since we had a suspicion that the on-demand scan was not configured properly.

1, The difference is very likely caused by a little number of whitelisted files. Non-whitelisted (untrusted) files are re-scanned after each module update.

2, Files referenced by shortcuts are scanned. However, if they are trusted/whitelisted they should not be scanned each time. I've made a test by creating a shortcut to a 360 MB sfx file. While the first scan took about 7 minutes, after re-scanning the shortcut the scan ended immediately.

3, Perhaps the files inside the folder were hidden. If certain files were scanned, then they must have been there.

[Microbe 5]

Thanks Gil. I don’t think my experiences exactly match with this reply, but it’s okay to close this ticket now. I’ll monitor over time and see how it all goes now that the main problem thankfully seems to have been fixed by the new program version.

 

Thank you ESET Team