ESET Smart Caching Questions

[0xDEADBEEF 43]

The EES version is 7.1.2045.5

There is a malicious powershell script sample that is detected by ESET's database for some time. It originally has .txt extension. By default it doesn't seem that ESET realtime/on-demand scan will detect the sample when the extension name is .txt (same for javascript and CAD files, if the extension of .js is missing the scan engine won't detect it).

Now that since the scan engine shows clean, I rename the file extension from .txt to .ps1, and I rescan it, the result is still clean.

 

However, if I open the script, add a white space, and save it, it is immediately detected and quarantined by the scan engine.

Later I found that even I don't modify the file, upon the next signature update the file that was once marked as clean will be detected.

Therefore I assume this is due to ESET scan's caching mechanism, as changing the filename doesn't change the file hash, the clean result will be cached and the scan will be directly skipped the next time. Upon the next update, the cache will be invalidated so the file is redetected again.

 

Just wondering if this is expected. I can see this is effective as a performance enhancement trick, because even the right click scan may "trust" such malicious file, executing it will anyway trigger the inspection again and detect it.

 

A side question: why is ESET's context menu scan still single threaded? Any plan to extend it to multi-thread scan (I mean emulate multiple files at a time)

Edited July 2, 2019 by 0xDEADBEEF

[itman 1,659]

Had a similar experience a few months back. Except in my case , Eset originally didn't detect the PowerShell code.

Well over a year ago, I found this nifty .NET based PowerShell global keylogger. Nothing earth shattering code-wise since the code had been available since 2015 on the pen-tester web site I found it on. In fact, I used the code to create a .ps1 script to run the keylogger to verify that Eset's B&PP does actually scramble keystokes and actual data could not be intercepted with this keylogger. The thing to note here is the .ps1 file code was not subsequently in time detected by any Eset on-demand scanning. Most likely since the file had been previously marked as scanned and clean.

Two months ago, I opened the same .ps1 script using notepad to copy the code to later paste into another forum. No modification to the original script code or anything. The minute I closed the .ps1 file, Eset caught it and quarantined it. Of note was that the detection signature date was the very same date!

My conclusion was that Eset's protections had subsequently updated to detect global keylogger code characteristics. Eset's realtime heuristics detected these characteristics upon file close and suspended the file. It was submitted via LiveGrid which promptly created a sig. for this specific code. The file was then promptly quarantined showing this new signature in the Eset log file.

[0xDEADBEEF 43]

51 minutes ago, itman said:

Had a similar experience a few months back. Except in my case , Eset originally didn't detect the PowerShell code.

Well over a year ago, I found this nifty .NET based PowerShell global keylogger. Nothing earth shattering code-wise since the code had been available since 2015 on the pen-tester web site I found it on. In fact, I used the code to create a .ps1 script to run the keylogger to verify that Eset's B&PP does actually scramble keystokes and actual data could not be intercepted with this keylogger. The thing to note here is the .ps1 file code was not subsequently in time detected by any Eset on-demand scanning. Most likely since the file had been previously marked as scanned and clean.

Two months ago, I opened the same .ps1 script using notepad to copy the code to later paste into another forum. No modification to the original script code or anything. The minute I closed the .ps1 file, Eset caught it and quarantined it. Of note was that the detection signature date was the very same date!

My conclusion was that Eset's protections had subsequently updated to detect global keylogger code characteristics. Eset's realtime heuristics detected these characteristics upon file close and suspended the file. It was submitted via LiveGrid which promptly created a sig. for this specific code. The file was then promptly quarantined showing this new signature in the Eset log file.

Something I've noticed is that seems ESET is spending a huge effort since last year to optimize the real-time monitoring performance. A side indicator is that its performance score in AV-TEST has improved significantly (5.5/6.0 recently in business test, in the past it was usually 3~4 out of 6.0 for home products). I can feel the real-time monitoring has become less sensitive (not in a bad way, though)

Still I was wondering why the scanner is detecting some files solely by extension. Though detecting a text file and infer its language/content may not be straightforward, many other vendors do. I am not seeing a vulnerability in doing so, but sometimes it is a bit confusing 

[itman 1,659]

1 hour ago, 0xDEADBEEF said:

Still I was wondering why the scanner is detecting some files solely by extension. Though detecting a text file and infer its language/content may not be straightforward, many other vendors do. I am not seeing a vulnerability in doing so, but sometimes it is a bit confusing 

Is your sample PowerShell script packed, encrypted, or obfuscated?

[0xDEADBEEF 43]

2 hours ago, itman said:

Is your sample PowerShell script packed, encrypted, or obfuscated?

yes it is obfuscated

[itman 1,659]

10 hours ago, 0xDEADBEEF said:

yes it is obfuscated

Malware Research Group back in 2017 tested 12 security solutions for their ability to detect malicious code in obfuscated PowerShell scripts. Test was run on Win 10 as part of a review of AMSI capability.

The categories of obfuscation were low, medium, and high. Eset at that time scored well detecting low and medium obfuscated scripts. However, it and most of the other tested products failed the highly obfuscated scripts.

Don't know if Eset's performance in this regard has improved since this test was conducted. What I do know is it is extremely difficult to detect highly obfuscated PowerShell scripts. Since you stated you never attempted to execute your sample PowerShell script, appears Eset's heuristic sandbox scanning utilizes AMSI. So it does indeed show that Eset can detected obfuscated PowerShell scripts prior to execution; whether highly obfuscated ones can be detected requires further examination.

Ref.: https://www.mrg-effitas.com/research/current-state-of-malicious-powershell-script-blocking/

[Marcos 5,070]

Please provide me with the file for internal debugging. Do you mean that it was not detected neither by real-time protection nor on-demand scanner after renaming the extension?

[0xDEADBEEF 43]

53 minutes ago, Marcos said:

Please provide me with the file for internal debugging. Do you mean that it was not detected neither by real-time protection nor on-demand scanner after renaming the extension?

yes. So if the script is landed on disk as .txt file for the first time, the realtime protection won't detect it, neither does on-demand scan. If I later rename it to .ps1 the on-demand scan still does not detect it.

However if the script is landed on disk as .ps1 file for the first time, the realtime protection will immediately detect and quarantine it.

It is very likely that in the first case the file is cached as "benign" due to .txt extension initially, and later got exempted from being scanned again by the right-click scanner even with .ps1 extension.

The attached sample's password is "infected"

sample.zip

[itman 1,659]

3 hours ago, 0xDEADBEEF said:

So if the script is landed on disk as .txt file for the first time, the realtime protection won't detect it, neither does on-demand scan.

I can verify this and the issue is worse than I thought.

I restored my keylogger script from quarantine. I then promptly renamed it to .txt from .ps1. I knew from previous experience that if I left the file with the .ps1 extension, Eset in short order would re-detect it and place it in quarantine again.

I then did an Eset context-scan on the folder containing the .txt keylogger script - zip detection by Eset. I then opened and closed the .txt keylogger script - zip detection by Eset.

[0xDEADBEEF 43]

1 minute ago, Marcos said:

I'm unable to reproduce it since ps.txt is immediately detected by real-time protection. I thought it would be detected after renaming the extension to ps1 but that doesn't seem to be the case.

This is very weird. I am running on latest EES with all settings set to default (except for the exclusion list). This has been reproduced on more than one machines on my side (I've not tested EIS but itman confirmed it)

This is not the first time I've experienced this. Previously another javascript malware sample has the same issue (it lost its .js extension hence ESET on-demand didn't detect it, though it could if the extension was correct)

Other users are welcome to try and see if this is reproduce-able on their side, however I don't think I can provide a sample link for them to download here...

 

Another thing to note is that even though such case may "slip through" the realtime protection, when actually executing it, the engine will anyway inspect it again. So if this is indeed something unexpected, at least it is not something really bad so far. I will try some other types of scripts and see how it goes.

[itman 1,659]

Now for the discussion if Eset "should" be detecting malicious script code in a .txt file?

This really is a tough one for a definitive yes. One misuse I can think of is renaming the file to txt.1 and then using FTYPE from a command script:

ftype logger.txt.1="C:\WINDOWS\system32\windowspowershell\v1.0\powershell.exe" assoc .ps1=logger.txt.1

Edited July 3, 2019 by itman

[Marcos 5,070]

4 minutes ago, itman said:

Now for the discussion if Eset "should" be detecting malicious script code in a .txt file?

Not really. There would be always ways to bypass detection. Imagine an AV would have to parse, e.g. 100 MB txt file on access each time. The machine would be unusable. You can disable Smart optimization to make sure that files are scanned each time they are accessed, however, it will most likely have adverse effect on performance as well.

[itman 1,659]

I just clarified something.

I uploaded my keylogger.txt file to a file sharing site. When I downloaded the file, Eset caught it and quarantined it.

So for the record, Eset does scan all files at attempted file creation time.

Edited July 3, 2019 by itman

[itman 1,659]

On 7/2/2019 at 1:46 PM, 0xDEADBEEF said:

Now that since the scan engine shows clean, I rename the file extension from .txt to .ps1, and I rescan it, the result is still clean.

I couldn't duplicate this. After renaming my keylogger script to .ps1 and running a context scan on the folder where it is stored, Eset detected it:

I believe what might have occurred in your case is the following.

LiveGrid submitted the .txt file for analysis when the file was created. Note: you should enable the Eset option to be alerted when a file submission occurs.

When you renamed the file to .ps1 and rescanned it, LiveGrid analysis had not been completed yet.

Later when you opened script, modified it, and attempted to save it, LiveGrid had completed its analysis and pushed a detection update to your device. Hence, the Eset detection at this point.

-EDIT- It is still somewhat of a mystery how your malware samples end up on your testing device. When we discussed this via PM and I suggested uploading to a fileshare and subsequent downloading from it, Eset appeared to detect the malware each time at attempted file creation time.

Edited July 4, 2019 by itman

[0xDEADBEEF 43]

8 hours ago, itman said:

When we discussed this via PM and I suggested uploading to a fileshare and subsequent downloading from it, Eset appeared to detect the malware each time at attempted file creation time.

Sometimes I got batches of samples (in zip) for analysis. I agree that it is slightly different from web download distribution. 

Anyway, I've recorded the case I was mentioning here: https://imgur.com/a/hGcPdsl

Note the two zip files have exactly the same file, except that the one with _ps1 contains .ps1 extension

[itman 1,659]

2 hours ago, 0xDEADBEEF said:

Anyway, I've recorded the case I was mentioning here: https://imgur.com/a/hGcPdsl

I reviewed the video and here's my opinion what took place.

First and notable, we are dealing with password protected archives. I will get back to this later.

In regards to the creation of the malware .txt file after unzipping the archive, it appears at this point local creation of .txt files are not scanned by the realtime engine as thoroughly as file extensions associated with executable code. Or and more likely since the script code was contained in a .txt file, obfuscated, and the file created locally, Eset didn't employ anti-obfuscation methods it would have for a .ps1 file. In other words, Eset didn't recognize the PowerShell code.

Renaming the .txt extracted file to .ps1 would not have changed its hash value. Since Eset had prior scanned the file, it was ignored in the subsequent context scanning.

So in reality is there a vulnerability here? I say no since the script would have been scanned again at PowerShell execution time via the Win 8/10 AMSI interface which would have revealed the actual script code after removal of the obfuscation characters.

Now for password protected archives. I downright consider them dangerous and really should not be extracted unless their source can be 100% verified as legitimate.

Edited July 4, 2019 by itman

[0xDEADBEEF 43]

2 hours ago, itman said:

So in reality is there a vulnerability here? I say no since the script would have been scanned again at PowerShell execution time via the Win 8/10 AMSI interface which would have revealed the actual script code after removal of the obfuscation characters.

Nope, I think I've found some issues already. But will not post here for now...

[0xDEADBEEF 43]

@Marcos I've PM-ed you the PoC sample. Could you evaluate if it is indeed a vulnerability?

[Marcos 5,070]

5 hours ago, itman said:

So in reality is there a vulnerability here? I say no since the script would have been scanned again at PowerShell execution time via the Win 8/10 AMSI interface which would have revealed the actual script code after removal of the obfuscation characters. 

That is my understanding as well; with AMSI-enabled OS's the file should be detected on execution of the interpreter (haven't tested it though but in theory it should work).  Also if such file is included in an archive, it would be scanned by web protection and would be detected regardless of the extension.

There are actually 3 ways how to solve it: always scan txt files (could make certain system slow down when an application is logging to big txt files extensively), keep the optimization as is, utilize HIPS to scan such files at the right point (like AMSI on not AMSI-enabled systems). The last one would be probably best but I'm not sure if it would be 100% effective and could not be circumvented with some effort. At any rate, it would be a long-term solution that cannot be implemented immediately.

[0xDEADBEEF 43]

36 minutes ago, Marcos said:

That is my understanding as well; with AMSI-enabled OS's the file should be detected on execution of the interpreter (haven't tested it though but in theory it should work).  Also if such file is included in an archive, it would be scanned by web protection and would be detected regardless of the extension.

However in my test this is not the case. My test env is Win10 1903 so AMSI is enabled. I did no source code change to the payload. What's worth noting is that I cannot reproduce the case on EES. Only EIS has such issue.

I also uploaded the sample to a webhost to simulate the user download scenario with a different hash, still no luck...

[itman 1,659]

7 hours ago, 0xDEADBEEF said:

However in my test this is not the case. My test env is Win10 1903 so AMSI is enabled.

After renaming the .txt file to .ps1, did you actually try to run the script?

To do so, you can't just double click on the .ps1 file since Win 10 blocks direct execution of .ps1 files. You would have to do so using;

powershell -nop -w hidden -ep bypass C:\xxxxxx\scriptname.ps1

from another script, command prompt, etc..

Edited July 5, 2019 by itman

[itman 1,659]

7 hours ago, 0xDEADBEEF said:

I also uploaded the sample to a webhost to simulate the user download scenario with a different hash, still no luck...

As I stated in a previous post, this is due to the fact the .txt file contains obfuscated code. Additionally if the script was packed or encrypted, Eset would not detect it on download or subsequent scanning. This is because the script will only decloke when loaded into memory at execution time. This is what Microsoft created the AMSI interface for. It is a memory sandbox where the script code can be examined by AV solutions after it has been decloked, but prior to actual execution.

Edited July 5, 2019 by itman

[0xDEADBEEF 43]

6 minutes ago, itman said:

As I stated in a previous post, this is due to the fact the .txt file contains obfuscated code. Additionally, if the script was packed or encrypted, Eset would not detect it on download or subsequent scanning. This is because the script will only decloke when loaded into memory at execution time. This is what Microsoft created the AMSI interface for. It is a memory sandbox where the script code can be examined by AV solutions after it has been decloked, but prior to actual execution.

See my PM. I'm not going to discuss it here before they say it is expected and acceptable

Edited July 5, 2019 by 0xDEADBEEF

[itman 1,659]

17 minutes ago, 0xDEADBEEF said:

See my PM.

Please do as you theorized; rename in another script the .txt script to a .js script and execute it. Does Eset now detect it?

Edited July 5, 2019 by itman

[0xDEADBEEF 43]

2 minutes ago, itman said:

Please do as you theorized; rename in another script the .txt script to a .js script and execute it. Does Eset now detect it?

for powershell script, I have already given you the answer in the prev replies:

On 7/3/2019 at 3:57 PM, 0xDEADBEEF said:

Another thing to note is that even though such case may "slip through" the realtime protection, when actually executing it, the engine will anyway inspect it again. So if this is indeed something unexpected, at least it is not something really bad so far.

Of course I said this after doing the actual experiment. However, it is not always the case (I can enumerate at least 2 other possible ways but I've yet tested them)