Jump to content

Very poor test result


razorfancy

Recommended Posts

  • Administrators

It looks like PowerShell was dropping a bunch of old malware plus Python was accessing it somehow. Doesn't look like a common infection scenario at all. Please provide the exact methodology how the test was performed and provide undetected files for verification so that we can check if they should be detected or not.

Link to comment
Share on other sites

  • Most Valued Members
34 minutes ago, Marcos said:

It looks like PowerShell was dropping a bunch of old malware plus Python was accessing it somehow. Doesn't look like a common infection scenario at all. Please provide the exact methodology how the test was performed and provide undetected files for verification so that we can check if they should be detected or not.

https://www.thepcsecuritychannel.com/testmethod

Link to comment
Share on other sites

As far as the first test phase, the malware .exe's were dropped in %AppDataUser% directories. So I don't know why those weren't detected. I personally have a HIPS rule that monitors any process startup in those directories.

As far as the Python based ransomware, it first needs to be verified if the tester had previously installed Python on the test rig. If so, then running of a malicious Python script would be much easier to accomplish. Note that the average user would not be installing Python.

Now there are malware attacks that can download the Python engine "on the fly" with a malicious script. However, this requires the previous to be "bundled" in a .exe. If the script was encrypted, obfuscated, packed, etc.., it would be hard to detect in memory since Win 10 AMSI interface does not scan Python scripts.:huh:

Link to comment
Share on other sites

  • Most Valued Members
Just now, itman said:

As far as the first test phase, the malware .exe's were dropped in %AppDataUser% directories. So I don't know why those weren't detected. I personally have a HIPS rule that monitors any process startup in those directories.

As far as the Python based ransomware, it first needs to be verified if the tester had previously installed Python on the test rig. If so, then running of a malicious Python script would be much easier to accomplish. Note that the average user would not be installing Python.

Now there are malware attacks that can download the Python engine "on the fly" with a malicious script. However, this requires the previous to be "bundled" in a .exe. If the script was encrypted, obfuscated, packed, etc.., it would be hard to detect in memory since Win 10 AMSI interface does not scan Python scripts.:huh:

Still if you are a user that uses Python you could get infected if you are not paying attention , this is where ESET should act but it didn't

Here are the results from the same test : https://www.thepcsecuritychannel.com/test-results

Link to comment
Share on other sites

I will say this.

If traces of the Python engine code are detected in an .exe, Eset should flag that .exe as suspicious.

Edited by itman
Link to comment
Share on other sites

  • Most Valued Members
4 minutes ago, itman said:

I will say this.

If traces of the Python engine code are detected in an .exe, Eset should flag that .exe as suspicious.

But it was running through PowerShell as a script , same thing goes for the .py script that can give you admin privileges without a password without ESET or Windows stopping it.

Edited by Rami
Link to comment
Share on other sites

1 minute ago, Rami said:

But it was running through PowerShell

This is immaterial per se. Although I do have a HIPS rule to monitor all PowerShell execution. Also, Eset has a KB article in regards to PowerShell HIPS rule monitoring as it applies to FileCoders. 

Link to comment
Share on other sites

  • Most Valued Members
Just now, itman said:

This is immaterial per se. Although I do have a HIPS rule to monitor all PowerShell execution. Also, Eset has a KB article in regards to PowerShell HIPS rule monitoring as it applies to FileCoders. 

But let's pretend you are a user who knows basics and have ESET installed and this person do use Python or have Python installed because it's required by some programs , you could get yourself infected with Python.

Link to comment
Share on other sites

  • Administrators

I will check with my colleagues if we are aware of this test and if was performed according to AMTSO standards (https://www.amtso.org/wp-content/uploads/2018/11/AMTSO-Testing-Protocol-Standard-for-the-Testing-of-Anti-Malware-Solutions-v1.1.pdf), especially with regard to

image.png

 

Link to comment
Share on other sites

3 hours ago, Marcos said:

I will check with my colleagues if we are aware of this test and if was performed according to AMTSO standards

Doubt this is the case.

From what I can determine, PC Security Channel is not an AMTSO member: https://www.amtso.org/members/

This test falls into the category of all ad hoc Internet tests whose results cannot be verified and therefore should be ignored. The only exception I can think of would be Runbenking's PC Magazine tests employing the Core Impact tools. He has been doing those for years and is very upfront on how and what he tests for.

Edited by itman
Link to comment
Share on other sites

AV-Comparatives has a write up on uTube security test sources. The most important point to note is these concerns are not formally recognized AV lab testing sources. As such, they don't adhere to formalized and verifiable testing standards.
 

Quote

Below are some YouTube tech channels that readers may find interesting. Please note that by making these links available here, AV-Comparatives does not necessarily agree with any methods or opinions expressed in any of these channels, and does not necessarily endorse (or criticise) any products or services mentioned in them. Readers should employ their own judgement when considering the validity of any points expressed by the channel’s authors.

The PC Security Channel

As its name suggests, this channel focuses very much on PC security.

Full Speed PC

This covers Windows and Mac platforms, and covers maintenance as well as security.

Malware Blocker

The emphasis here, as suggested by the name, is protecting computers from malware.

Malware Geek

This is another malware-oriented channel.

Computer Solutions

Actual tests of antimalware programs, against phishing and malware URLs and other threats, are demonstrated in this channel.

Security Now

A series of regular interviews, discussing various security related issues.

SemperVideo

German-language channel with videos on individual security flaws, tips and tricks.

https://www.av-comparatives.org/youtube-security-channels/

Edited by itman
Link to comment
Share on other sites

Although The PC Security Channel [TPSC] is not an official AMTSO member, it is a worth noting channel that uses a consistent methodology to test security products. It is a respected chanell to IT, programmers and av-fans people. I starting using ESET products this year and bought 8 licenses in total and I am disappointed that ESET failed this test. As a programmer I also code in Python and I am worried by the failed test.

Link to comment
Share on other sites

1 hour ago, ECELeader said:

Although The PC Security Channel [TPSC] is not an official AMTSO member, it is a worth noting channel that uses a consistent methodology to test security products.

That said, I see a few "irregularities."

TPSC has affiliations with Bitdefender, Kaspersky, and Sophos. Next as show in the below screen shot, Kaspersky only scored in 80.46% versus Eset's 95.6% in Phase 1 testing but passed overall testing? Appears that because Eset failed the Python ransomware test that was justification for the overall failure rating. Is this a standard AV lab testing methodology? Or is what we have here is a polished presentation using a pre-evaluated ransomware sample that my sponsors product detected but its major competitor did not?

TPSC.thumb.png.cb0ab861d9055164dc637531b3ebb6ab.png

 

Edited by itman
Link to comment
Share on other sites

1 hour ago, itman said:

TPSC has affiliations with Bitdefender, Kaspersky, and Sophos...

...says somebody who has affiliation with ESET , trying to justify the questionable result .....

Link to comment
Share on other sites

2 hours ago, itman said:

That said, I see a few "irregularities."

TPSC has affiliations with Bitdefender, Kaspersky, and Sophos. Next as show in the below screen shot, Kaspersky only scored in 80.46% versus Eset's 95.6% in Phase 1 testing but passed overall testing? Appears that because Eset failed the Python ransomware test that was justification for the overall failure rating. Is this a standard AV lab testing methodology? Or is what we have here is a polished presentation using a pre-evaluated ransomware sample that my sponsors product detected but its major competitor did not?

TPSC.thumb.png.cb0ab861d9055164dc637531b3ebb6ab.png

 

The score shows only the results of proactive detection (malware was successfully blocked by the test product, prior to execution). A product is considered to pass phase 1 if after running the python script manages to keep the system clean in the end (clean sheet). You can read about the test method here.

Link to comment
Share on other sites

I don't want blame Eset or anything else, but I think Eset Failed hard. However, this should make us reflect as TPSC has also been surprised by the result. The pro active defense is one of the highest he has tested but despite this the PC was infected. At this point a question arises: what is the problem of Eset? In my opinion, it can not stop the execution of Malware that bypasses the protection by infecting the device

Link to comment
Share on other sites

  • Administrators

Below find my personal comments that may not represent an official response of the company on this test.

1, It's not a real world test and it appears that some protection layers were bypassed (e.g. web protection with more aggressive detection and url blocking), ie. the results might not reflect how ESET would protect users in real life. Also the question is if the missed sample was actual or synthetic threat. Since we didn't get missed samples for verification, we don't know how prevalent in the world they are.

2, A false positive test was not a part of the test. It's easy to detect 100% of malware if also clean files are detected.

3, The author works for Emsisoft. Despite the claims of being independent, it's hard to believe that this did not affect the test in any way. It's also interesting that Bitdefender got best results and Emsisoft uses its engine as well.
Employees of AV companies should not perform tests that they proclaim to be independent and unbiased. Only prestigious and respectful AV testing organizations should do that where independence is ensured. It would not be too difficult to make a test where an AV scoring 100% in other tests would get 0% if the "right" samples were picked in the test set.

4, "If a sample successfully makes it to memory and begins execution, it is considered a miss." This is a flawed methodology. A file has to be first unpacked in memory before it is executed. Advanced memory scanner triggers a scan only after a file has been executed and unpacked in memory.

 

I strongly recommend taking tests from youtube or performed by other than non-professional testers with a pinch of salt.  One must consider and understand all aspects of how a test was performed in order to take the results seriously.

Link to comment
Share on other sites

  • Administrators
6 hours ago, novice said:

...says somebody who has affiliation with ESET , trying to justify the questionable result .....

I strongly disagree with this accusation. Itman is not an employee of ESET and has no other relation to the company. He's become an active user in our forum and is a person who 's always willing to help the other with issues they have for no profit.

Link to comment
Share on other sites

2 hours ago, Marcos said:

I strongly disagree with this accusation.

It is not an "accusation" is merely an observation.

2 hours ago, Marcos said:

Itman is not an employee of ESET and has no other relation to the company

Nobody said that Itman is an ESET employee.

Itman is a valuable member of this community , however I noticed his tendency of defending ESET no matter what and having a biased attitude.

Link to comment
Share on other sites

  • Most Valued Members
23 minutes ago, galaxy said:

These tests don't matter. Why, because it is never shown how to capture the viruses hot or similar, so no real tests

Why? , It's samples that are loaded into a script and then the script start running and running the samples within , when the anti-virus fails to detect one of the samples and the sample can infect the machine and encrypt it , then the AV did fail it's job , even though that the virus did come from another way which is Python , but still it's a malware/virus and the machine got infected

So if a user downloaded a suspicious Python file, and did run it , then he would get infected.

It could be that the signature is not in the database so probably that's why ESET missed it , but still yet LiveGrid didn't act or real-time protections didn't notice the changes made by Python ransomware

Edited by Rami
Link to comment
Share on other sites

  • Administrators
35 minutes ago, Rami said:

Why? , It's samples that are loaded into a script and then the script start running and running the samples within...

I'd like to bring the following write-up by AV-Comparatives into your attention which clarifies why unprofessional tests are flawed: https://www.av-comparatives.org/spotlight-on-security-why-do-av-products-score-so-highly-in-professional-tests/

Compare the methodology used in this "test" with what AV-Comparatives wrote:

Quote

In general, YouTube testers often use flawed (partial) test methods. Most YouTube testers often download a malware pack, unzip this malware sample pack (with disabled antivirus), enable the AV-solution of choice and run (or even just scan) the malware executables. By doing so, they cut out many of the protection mechanisms of the AV solution tested. For example, most AV products will behave differently when a file is downloaded from an URL with a poor reputation. This is why a specific sample might bypass an AV-solution in an ‘unzip and scan/execute’ test, while the same sample with the same AV solution could be blocked in AV-Comparatives’ ‘Real-World Protection’ test (which mimics the infection chain and use the real source URL in the execution scenario).

 

Link to comment
Share on other sites

I'm Buzzle, moderator of TPSC Discord server. Let me just quickly clarify a few points here:

First, the Python script only plays the role of executing the malware in the ./Phase1 and ./Phase2 folders. It it NOT a ransomware. It's called malex for a reason (MALware EXecutor)

Second, the reason why Kaspersky only get a Pro-active detection ratio of ~80% but still pass Phase1 is due to the fact that BOTH HitmanPro and MalwareBytes detect nothing after Phase 1 and the system have no sign of infection (https://youtu.be/DtHHsJIw1RI?t=249) but ESET clearly did ( https://youtu.be/e0irhY1GOuI?t=166 +  https://youtu.be/e0irhY1GOuI?t=401 ). In fact, ESET did not even qualified for Phase2 but we did it anyway

Third, if you think the reason Bitdefender, Kaspersky, and Sophos passed because of affiliations with TPSC, then explain Norton and Comodo's result. We use the same testing method (running all the malware through malex) in every test since Windows Defender. There's no "irregularity" here like you said.

 

If you want to discuss with us, jump on our Discord server. Link is in the description of the video.

Link to comment
Share on other sites

3 minutes ago, Marcos said:

I'd like to bring the following write-up by AV-Comparatives into your attention which clarifies why unprofessional tests are flawed: https://www.av-comparatives.org/spotlight-on-security-why-do-av-products-score-so-highly-in-professional-tests/

ESET did not perform well also in Av Comparatives for Sep 2018   (98.5%)  , so why everybody is so surprised now?????
 

 

Link to comment
Share on other sites

Guest
This topic is now closed to further replies.
  • Recently Browsing   0 members

    • No registered users viewing this page.
×
×
  • Create New...