Very poor test result

[Marcos 5,074]

2 minutes ago, novice said:

ESET did not perform well also in Av Comparatives for Sep 2018   (98.5%)  , so why everybody is so surprised now?????

If you really think that other AVs outperform ESET, then why you are still using it? Just a rhetorical question.

[Nightowl 202]

2 minutes ago, novice said:

ESET did not perform well also in Av Comparatives for Sep 2018   (98.5%)  , so why everybody is so surprised now?????
 

 

Still you shouldn't miss somewhere they also typed , that even though AVs do reach 100% that doesn't mean that it will always work/protect you in 100%

Still 98.5 is an excellent rating , which grants ESET 3 stars from AV Comparatives (which is the highest)

[Marcos 5,074]

If I was to perform a quick on-demand test of very fresh malware detection:

Fresh Filecoder.Crysis:
A        clean
S     clean
ESET Win32/GenKryptik.CTAZ
A        clean
M    clean
D        clean
B  clean
K    clean

Fresh Dridex trojan:
A        clean
ESET Win32/Dridex.U
S     clean
A        clean
M    clean
B  clean
K    clean
D        clean
M       clean

And I could continue like that with any fresh malware sample but there's no sense in it. The point is bypassing a single protection layer obviously skews results as we all know that at least some of the above AVs would detect the malware upon download or execution.

[Nightowl 202]

10 minutes ago, Marcos said:

If I was to perform a quick on-demand test of very fresh malware detection:

Fresh Filecoder.Crysis:
A        clean
S     clean
ESET Win32/GenKryptik.CTAZ
A        clean
M    clean
D        clean
B  clean
K    clean

Fresh Dridex trojan:
A        clean
ESET Win32/Dridex.U
S     clean
A        clean
M    clean
B  clean
K    clean
D        clean
M       clean

And I could continue like that with any fresh malware sample but there's no sense in it. The point is bypassing a single protection layer obviously skews results as we all know that at least some of the above AVs would detect the malware upon download or execution.

The question is what differences did other AVs do to get it right? , If I understood you correctly the other AVs did stop the execution or stop the download

So what protects the user from 0-day threats , like the Python one which is 0-day for ESET because ESET doesn't have it in it's database so it bypassed the AV.

[Marcos 5,074]

5 minutes ago, Rami said:

The question is what differences did other AVs do to get it right? , If I understood you correctly the other AVs did stop the execution or stop the download 

I'm sorry, I don't understand. Each vendor uses its own mechanisms to detect and get suspicious files. There are many ways how vendors get samples but the quickest way to learn about new malware is via feedback systems which is ESET LiveGrid in our products. What matters is how quickly a particular vendor can identify new malware and respond to it by adding recognition.

Quote

So what protects the user from 0-day threats , like the Python one which is 0-day for ESET because ESET doesn't have it in it's database so it bypassed the AV. 

The problem with script malware is that scripts can be modified easily even by people with little knowledge (e.g. by kids or students) until they become undetected by the vendor that they focus on. No matter what vendor it is, detection of scripts can be relatively easily circumvented. The only 100% protection against script malware is blocking the script interpreter from interpreting scripts, e.g. which are placed outside of a folder in which execution of (legitimate) scripts is allowed. And that is also why we recommend applying HIPS anti-ransomware policies to improve protection even more.

[Nightowl 202]

3 minutes ago, Marcos said:

I'm sorry, I don't understand. Each vendor uses its own mechanisms to detect and get suspicious files. There are many ways how vendors get samples but the quickest way to learn about new malware is via feedback systems which is ESET LiveGrid in our products. What matters is how quickly a particular vendor can identify new malware and respond to it by adding recognition.

The problem with script malware is that scripts can be modified easily even by people with little knowledge (e.g. by kids or students) until they become undetected by the vendor that they focus on. No matter what vendor it is, detection of scripts can be relatively easily circumvented. The only 100% protection against script malware is blocking the script interpreter from interpreting scripts, e.g. which are placed outside of a folder in which execution of (legitimate) scripts is allowed. And that is also why we recommend applying HIPS anti-ransomware policies to improve protection even more.

 I am sorry I didn't explain correctly , but I understand now.

Thanks for the explanation Marcos.

[Buzzle 4]

8 minutes ago, Marcos said:

The problem with script malware is that scripts can be modified easily even by people with little knowledge (e.g. by kids or students) until they become undetected by the vendor that they focus on.

I don't know if my clarification isn't enough or you ignored my post, but let me re-clarify this 

50 minutes ago, Buzzle said:

the Python script only plays the role of executing the malware in the ./Phase1 and ./Phase2 folders. It it NOT a ransomware. It's called malex for a reason (MALware EXecutor)

 

[NVMe 0]

1 hour ago, Marcos said:

If you really think that other AVs outperform ESET, then why you are still using it? Just a rhetorical question. 

Disappointing reply by ESET. I just bought into ESET coming from years of Bitdefender (extremely buggy but great protection).

While other AVs successfully pass this test method ESET currently doesn't - it also didn't stop Cerber Ransomware.

I hope ESET can convince us that they are offering great protection against malware and ransomware. Otherwise I'm about to write off my ESET investment and switch my AV solution again.

[Nightowl 202]

9 minutes ago, NVMe said:

Disappointing reply by ESET. I just bought into ESET coming from years of Bitdefender (extremely buggy but great protection).

While other AVs successfully pass this test method ESET currently doesn't - it also didn't stop Cerber Ransomware.

I hope ESET can convince us that they are offering great protection against malware and ransomware. Otherwise I'm about to write off my ESET investment and switch my AV solution again.

For several years of ESET usage I have never been disappointed by ESET or got myself into trouble because of a failure from ESET's side , mistakes happen here and there , and sometimes a threat get missed by the AV , it's okay it happens with every AV company

Kaspersky most of the times score 100% in their tests , but even though some tests they fail to reach the 100%.

[Marcos 5,074]

14 minutes ago, NVMe said:

While other AVs successfully pass this test method ESET currently doesn't - it also didn't stop Cerber Ransomware. 

I hope ESET can convince us that they are offering great protection against malware and ransomware. Otherwise I'm about to write off my ESET investment and switch my AV solution again. 

Please refer to tests performed by prestigious testing organizations that adhere to AMTSO principles, such as www.av-comparatives.org, SE Labs (selabs.uk), www.av-test.org, Virus Bulletin, etc. As I have proved above, removing even one protection layer may lead to incorrect and skewed results. I showed where ESET detected a fresh Filecoder Crysis ransomware which was missed by all other well-known AVs but in fact that didn't tell anything about if they would protect users from the ransomware upon download or execution.

[Nightowl 202]

5 minutes ago, Marcos said:

Please refer to tests performed by prestigious testing organizations that adhere to AMTSO principles, such as www.av-comparatives.org, SE Labs (selabs.uk), www.av-test.org, Virus Bulletin, etc. As I have proved above, removing even one protection layer may lead to incorrect and skewed results. I showed where ESET detected a fresh Filecoder Crysis ransomware which was missed by all other well-known AVs but in fact that didn't tell anything about if they would protect users from the ransomware upon download or execution.

By removing one protection layer you mean that the file could be stopped at the download moment by web protection? or excecution moment?

Or do you mean he did disable one of the protection features in ESET inorder for the test to start?

[NVMe 0]

@Rami: Thanks, thats reassuring and good to hear. In the end it's the longterm protection which matters. But still ESET should take this test result and make it's protection engine even better. I'm currently running Acronis Ransomware Protection in parallel but ESET should be able to protect against ransomware 100% reliably.

It would be great if ESET would reproduce this exact test and explain why the result isn't worrisome and ESET offers great protection even for ransomware. Ideally via Youtube video and explaining all steps in a way that the average user understands.

I really want to continue using ESET as it has many strengths in comparison with its competitors, especially the very low system impact which pays off on a daily basis.

 

[Marcos 5,074]

1 minute ago, Rami said:

By removing one protection layer you mean that the file could be stopped at the download moment by web protection? or excecution moment?

If it was a real malware found in the world then there was most likely a url from which it would have been downloaded. Web protection might have blocked it upon download for instance. However, since we didn't get missed samples for verification (see the AMTSO principles quoted above), we cannot tell if it would have been detected / blocked. Respected AV testing organizations provide AV vendors with missed samples or hashes at least and give time to dispute the results before they are published.

[Nightowl 202]

1 minute ago, NVMe said:

@Rami: Thanks, thats reassuring and good to hear. In the end it's the longterm protection which matters. But still ESET should take this test result and make it's protection engine even better. I'm currently running Acronis Ransomware Protection in parallel but ESET should be able to protect against ransomware 100% reliably.

It would be great if ESET would reproduce this exact test and explain why the result isn't worrisome and ESET offers great protection even for ransomware. Ideally via Youtube video and explaining all steps in a way that the average user understands.

I really want to continue using ESET as it has many strengths in comparison with its competitors, especially the very low system impact which pays off on a daily basis.

 

Against 0-day threats , mostly all of the AVs will rely on their Cloud to see if there are another infection or threat that looks like the 0-day , but it might bypass the AV depending on the situation if I am not mistaken , AVs do act differently because each one has a different engine and way of detection and default policies that are set for the test/user , for example Microsoft Defender tests have a strict policy that will flag too many false positives , in the same time if you kill the internet so you won't have any cloud , Windows Defender will fail to detect a lot of threats because it lacks the cloud's data

Even if you switch to another AV brand , you won't get the 100% protection because there isn't a 100% protection , the same thing goes for physical protections or for example a house door that looks like 100% safe , don't worry someone will know how to open it or brute force it , or kick it and get inside or anything in life , there is no 100%

But indeed you still need to rely on the software that you are using that it won't fail you , but still most of the infections that people get are caused by themselves not the AV , whether they surfed a malicious website , or opened a file or gave an admin permissions to untrusted file so it starts

But I do rely on ESET technology , It's light and the detection rate is high and I just got used to it , I don't feel like switching even though there are another 100% scores in the AV tests but I do prefer to stay with ESET , because I have been using it since the age of V2 , no I don't work for them.

[Marcos 5,074]

14 minutes ago, Rami said:

But indeed you still need to rely on the software that you are using that it won't fail you , but still most of the infections that people get are caused by themselves not the AV , whether they surfed a malicious website , or opened a file or gave an admin permissions to untrusted file so it starts

When we get reports about infections and especially about ransomware that encrypted files, in 99,9% of cases it's because of insecure RDP. Attackers gain access to user's machine by performing brute-force attacks and once they get in with administrator rights, they disable the AV and run ransomware. Therefore users should never rely on that merely installing antivirus will protect them from threats. Last but not least, it's also safe computing that can save you from getting infected.

For information about technology and protection layers that ESET has developed and has been continually improving, please read https://www.eset.com/int/about/technology/.

[Nightowl 202]

18 minutes ago, Marcos said:

If it was a real malware found in the world then there was most likely a url from which it would have been downloaded. Web protection might have blocked it upon download for instance. However, since we didn't get missed samples for verification (see the AMTSO principles quoted above), we cannot tell if it would have been detected / blocked. Respected AV testing organizations provide AV vendors with missed samples or hashes at least and give time to dispute the results before they are published.

8 minutes ago, Marcos said:

When we get reports about infections and especially about ransomware that encrypted files, in 99,9% of cases it's because of insecure RDP. Attackers gain access to user's machine by performing brute-force attacks and once they get in with administrator rights, they disable the AV and run ransomware. Therefore users should never rely on that merely installing antivirus will protect them from threats. Last but not least, it's also safe computing that can save you from getting infected. 

I appreciate your explanation , now I understand , thank you.

[itman 1,659]

4 hours ago, Buzzle said:

Second, the reason why Kaspersky only get a Pro-active detection ratio of ~80% but still pass Phase1 is due to the fact that BOTH HitmanPro and MalwareBytes detect nothing after Phase 1

Perhaps the non-detection is due to the fact both products have shown issues with malware detection in AV lab testing? Malware Research Group up until recently used to include both products in their quarterly 360 Full Spectrum tests: https://www.mrg-effitas.com/wp-content/uploads/2018/05/MRG-Effitas-2018Q1-360-Assessment.pdf

[itman 1,659]

15 hours ago, itman said:

Or is what we have here is a polished presentation using a pre-evaluated ransomware sample that my sponsors product detected but its major competitor did not?

To clarify, what I was trying to suggest is this and like activity is possible in ad hoc testing; not that it actually occurred. I have no direct proof that TPSC engaged in such activity.

[esset 2]

4 hours ago, Marcos said:

I'm sorry, I don't understand. Each vendor uses its own mechanisms to detect and get suspicious files. There are many ways how vendors get samples but the quickest way to learn about new malware is via feedback systems which is ESET LiveGrid in our products. What matters is how quickly a particular vendor can identify new malware and respond to it by adding recognition.

The problem with script malware is that scripts can be modified easily even by people with little knowledge (e.g. by kids or students) until they become undetected by the vendor that they focus on. No matter what vendor it is, detection of scripts can be relatively easily circumvented. The only 100% protection against script malware is blocking the script interpreter from interpreting scripts, e.g. which are placed outside of a folder in which execution of (legitimate) scripts is allowed. And that is also why we recommend applying HIPS anti-ransomware policies to improve protection even more.

'The only 100% protection against script malware is blocking the script interpreter from interpreting scripts, e.g. which are placed outside of a folder in which execution of (legitimate) scripts is allowed. And that is also why we recommend applying HIPS anti-ransomware policies to improve protection even more.' Does this recommendation also apply to regular consumers ? I guess this requires a somewhat convoluted approach or an additional product ?

I'm quite satisfied with Eset as such. Still, the not 100 % detection in AV-comparatives' tests suggests there is room for progress, One other major vendor in particular (nearly?) Always scores 100 %, but of course with its own drawbacks.

[razorfancy 9]

8 hours ago, Marcos said:

Below find my personal comments that may not represent an official response of the company on this test.

1, It's not a real world test and it appears that some protection layers were bypassed (e.g. web protection with more aggressive detection and url blocking), ie. the results might not reflect how ESET would protect users in real life. Also the question is if the missed sample was actual or synthetic threat. Since we didn't get missed samples for verification, we don't know how prevalent in the world they are.

2, A false positive test was not a part of the test. It's easy to detect 100% of malware if also clean files are detected.

3, The author works for Emsisoft. Despite the claims of being independent, it's hard to believe that this did not affect the test in any way. It's also interesting that Bitdefender got best results and Emsisoft uses its engine as well.
Employees of AV companies should not perform tests that they proclaim to be independent and unbiased. Only prestigious and respectful AV testing organizations should do that where independence is ensured. It would not be too difficult to make a test where an AV scoring 100% in other tests would get 0% if the "right" samples were picked in the test set.

4, "If a sample successfully makes it to memory and begins execution, it is considered a miss." This is a flawed methodology. A file has to be first unpacked in memory before it is executed. Advanced memory scanner triggers a scan only after a file has been executed and unpacked in memory.

 

I strongly recommend taking tests from youtube or performed by other than non-professional testers with a pinch of salt.  One must consider and understand all aspects of how a test was performed in order to take the results seriously.

I personally dont think the person that runs The PC Security Channel [TPSC] is manipulating the result for example ESET Internet Security v10 and v11 got good results on his tests.

v10 Review:

 

v11 Review:

 

 

Edited December 4, 2018 by razorfancy

[Buzzle 4]

1 hour ago, itman said:

Perhaps the non-detection is due to the fact both products have shown issues with malware detection in AV lab testing? Malware Research Group up until recently used to include both products in their quarterly 360 Full Spectrum tests: https://www.mrg-effitas.com/wp-content/uploads/2018/05/MRG-Effitas-2018Q1-360-Assessment.pdf

Remember that the conditions that an AV needed to pass our test is BOTH clean sheet on second-opinion scanners AND the system showing no sign of infection. Also, at the very end of that Kaspersky video, we also threw in Norton's Power Eraser, which also indicates a clean system. MBAM and HitmanPro detected CoreTemp which is a legitimate application (maybe they include adware/PUPs with their installer?) but on the ESET-protected system there was clear signs of malware/ransomware at the end of both Phase 1 and 2, and that's enough to disqualify it

24 minutes ago, razorfancy said:

I personally dont think the person that runs The PC Security Channel [TPSC] is manipulating the result for example ESET Internet Security v10 and v11 got good results on his tests.

v10 Review:

  Reveal hidden contents

v11 Review:

  Reveal hidden contents

 

We recently changed our testing methodology, which mainly focused on the Pro-Active protection components of the AVs, which can and will change many results that we have published in the past.

[itman 1,659]

Let's get back to the methodology employed in this test namely:

Quote

the Python script only plays the role of executing the malware in the ./Phase1 and ./Phase2 folders. It it NOT a ransomware. It's called malex for a reason (MALware EXecutor)

There are only three ways a Python script can be executed on a user device:

1. The user manually installed Python.

2. The attack involved downloading Python and installing it.

3. As posted previously, the attacker created an executable with the Python engine component and malicious script contained within.

Methods 1). and 2). are not viable scenarios since most users do not install Python and malware installation of it would be extremely "noisy."

That leaves  number 3). as the most likely way Python based malware would be deployed. And this method needs attention by Eset and other AV vendors. Also this is not an easy mitigation in that the presence of the Python engine code within the .exe is in itself not malicious. It is the code within the script that is malicious. And, it is a given that the script code will be hidden and will not reveal itself until the executable unhides the script code in memory prior to executing it via the Python engine. Compounding the issue is Python scripts cannot be examined by the Win 10 AMSI interface when executed this way; even if it did so which BTW, it does not. This leads to two possible detection scenarios: 

1. Improved sandboxing analysis for any executable that contains Python engine code remnants and detection of script code malware after it is unhidden in memory.

2. After sandbox detection of Python engine code remnants, alerting the user of possible suspicious process activity. This alerting could be conditioned upon a number of factors such as process reputation status; e.g. unknown, signing status; e.g. unsigned, etc..

I believe number 2). is the only viable solution given the capability of Python script code executed this way. Also it is not the norm for legit application software to be developed this way.

Edited December 4, 2018 by itman

[Buzzle 4]

3 minutes ago, itman said:

Let's get back to the methodology employed in this test namely:

There are only three ways a Python script can be executed on a user device:

1. The user manually installed Python.

2. The attack involved downloading Python and installing it.

3. As posted previously, the attacker created an executable with the Python engine component and malicious script contained within.

Methods 1). and 2). are not viable scenarios since most users do not install Python and malware installation of it would be extremely "noisy."

That leaves  number 3). as the most likely way Python based malware would be deployed. And this method needs attention by Eset and other AV vendors. Also this is not an easy mitigation in that the presence of the Python engine code within the .exe is in itself not malicious. It is the code within the script that is malicious. And, it is a given that the script code will be hidden and will not reveal itself until the executable unhides the script code in memory prior to executing it via the Python engine. Compounding the issue is Python scripts cannot be examined by the Win 10 AMSI interface when executed this way; even if it did so which BTW, it does not. This leads to two possible detection scenarios: 

1. Improved sandboxing analysis for any executable that contains Python engine code remnants and detection of script code malware after it is unhidden in memory.

2. After sandbox detection of Python engine code remnants, alerting the user of possible suspicious process activity. This alerting could be conditioned upon a number of factors such as process reputation status; e.g. unknown, signing status; e.g. unsigned, etc..

I believe number 2). is the only viable solution given the capability of Python script code executed this way. Also it is not the norm for legit application software to be developed this way.

Looks like you still don’t quite understand what the Python script is for. Its only function is to automate the execution of the malware in ./Phase1 and ./Phase2, just like there is a user that would go into the folders and run the executables one by one. The malware use for the test is all in the Phase1 and Phase2 folders. The Python script could be replaced with a PowerShell/bat/$whatever script and it won’t affect the results whatsoever. 

Also, who even code ransomware in Python? It would probably take forever to encrypt data

[Andrew3000 4]

8 minutes ago, itman said:

Let's get back to the methodology employed in this test namely:

There are only three ways a Python script can be executed on a user device:

1. The user manually installed Python.

2. The attack involved downloading Python and installing it.

3. As posted previously, the attacker created an executable with the Python engine component and malicious script contained within.

Methods 1). and 2). are not viable scenarios since most users do not install Python and malware installation of it would be extremely "noisy."

That leaves  number 3). as the most likely way Python based malware would be deployed. And this method needs attention by Eset and other AV vendors. Also this is not an easy mitigation in that the presence of the Python engine code within the .exe is in itself not malicious. It is the code within the script that is malicious. And, it is a given that the script code will be hidden and will not reveal itself until the executable unhides the script code in memory prior to executing it via the Python engine. Compounding the issue is Python scripts cannot be examined by the Win 10 AMSI interface when executed this way; even if it did so which BTW, it does not. This leads to two possible detection scenarios: 

1. Improved sandboxing analysis for any executable that contains Python engine code remnants and detection of script code malware after it is unhidden in memory.

2. After sandbox detection of Python engine code remnants, alerting the user of possible suspicious process activity. This alerting could be conditioned upon a number of factors such as process reputation status; e.g. unknown, signing status; e.g. unsigned, etc..

I believe number 2). is the only viable solution given the capability of Python script code executed this way. Also it is not the norm for legit application software to be developed this way.

What's wrong with you? The python script is a simple file executer nothing else, it's like double clicking with the mouse but  the process is automatted

[Marcos 5,074]

In order for us to provide an official response on the test, we would need the following:

Obviously the following concerns were not addressed since the "tester" didn't download files from actual urls serving the payload, ie. real-world conditions were not fulfilled and one of the important protection layers was bypassed:

Quote

 

In general, YouTube testers often use flawed (partial) test methods. Most YouTube testers often download a malware pack, unzip this malware sample pack (with disabled antivirus), enable the AV-solution of choice and run (or even just scan) the malware executables. By doing so, they cut out many of the protection mechanisms of the AV solution tested. For example, most AV products will behave differently when a file is downloaded from an URL with a poor reputation. This is why a specific sample might bypass an AV-solution in an ‘unzip and scan/execute’ test, while the same sample with the same AV solution could be blocked in AV-Comparatives’ ‘Real-World Protection’ test (which mimics the infection chain and use the real source URL in the execution scenario).

 

Without logs, samples or hashes, and possibly further metada, everything said in this topic are just speculations. Respected testers would allow vendors of tested AVs to review the results, provide the necessary stuff for verification and give room for disputes with vendors. This was not the case. Having said that, we'll draw this topic to a close.