MRG Effitas 360 Assessment & Certification Q3 2016

[adnage19 1]

https://www.mrg-effitas.com/wp-content/uploads/2016/11/MRG-Effitas-360-Assessment-Q3-2016.pdf

ESET failed again in this test.

[Marcos 5,257]

Tests are tests, reality is often different. Just looking at our forum and forums of "top performers" and comparing the number of complaints about infection is interesting, isn't it? It's a matter of fact that there's nothing like 100% protection. Even if a particular AV missed just a few threats from the total number of threats in the world and you would get infected by it, you wouldn't care if it detects 100% or 70% of malware in tests.

As for this particular test, it's said we didn't get samples nor any information about samples for verification which is strongly recommended by AMTSO:
hxxp://www.amtso.org/download/amtso-fundamental-principles-of-testing/?wpdmdl=1151

 

Q. Must testers provide feedback and/or samples to vendors?
A. No. However, AMTSO encourages testers to provide vendors with constructive and adequate feedback in a timely fashion about specific faults and deficiencies (e.g. crashes, false positives, false negatives, etc.) This feedback can be in the form of technical details, reproduction steps, log files, memory dumps, samples, etc.

 

Without this data, we cannot tell if all samples were suitable for being included in a test set or if any of the samples was found on a computer with ESET installed.

[itman 1,746]

Since I am no great fan of MRG's testing, I am going to "chime in" on this.

 

This test was done using ver. 9 Smart Security. Ver. 10 has incorporated js, ws, and Powershell script protection which is the predominate threat mechanism ransomware uses. However, ver. 9 still scored a 95% detection rate which ranked it in the upper third of all products tested. 

 

Now for the financial malware tests. The MRG report does not state whether the tests were done using the AV's financial protection secured browser feature. I assume the 360 tests did not since MRG does a separate test for such protection. However even if the tests were performed using Eset's OPP feature, the browser MRG used for testing was Microsoft Edge. Eset specifically states that the OPP feature is not supported for the Edge browser. So take those financial malware test results "with a grain of salt." 

 

Bottom line - when evaluating AV protection test results, you have to first and foremost consider the source of the testing.

 

Here's a link to testing performed by SE Labs in the U.K. for Smart Security ver. 9: https://selabs.uk/en/reports/consumers . It was the top scoring product.

Edited November 8, 2016 by itman

[cvvorous 4]

The MRG report does not state whether the tests were done using the AV's financial protection secured browser feature. I assume the 360 tests did not since MRG does a separate test for such protection. However even if the tests were performed using Eset's OPP feature, the browser MRG used for testing was Microsoft Edge. Eset specifically states that the OPP feature is not supported for the Edge browser. So take those financial malware test results "with a grain of salt." 

 

IMO, it doesn't matter if the test's scope is only whether the system gets infected and whether the product remediated the infection after 24h. As you said, the payment protection is tested as part of the banking-specific report, and that particular test is conducted on systems running Win7 x64 w/IE 11 as the browser.

 

This particular one seems to me as straightforward a test as any - does the product protect the system from infection? If not, does it remediate the infection after 24h?

 

From the report:

 

This assessment measured the ability of security products to protect an endpoint from a live infection, and, in the event of a system being compromised, the time taken to detect the infection and remediate the system. The timeto-detect-and-remediate component relied on each security product being manually forced to conduct a scan every thirty minutes over a twenty-four hour period.

[TomFace 539]

As Marcos said,"test are test". They do not reflect day to day reality. Any test on any given day can be skewed to give this result or that result. That's one reason why I do not read Consumer Reports. I will not spend my valuable time chasing results.

Edited November 8, 2016 by TomFace

[itman 1,746]

Here's a link to A-V Comparatives latest Real World protection test: https://chart.av-comparatives.org/chart1.php . Eset scored 99.7%.

[itman 1,746]

As for this particular test, it's said we didn't get samples nor any information about samples for verification which is strongly recommended by AMTSO:

hxxp://www.amtso.org/download/amtso-fundamental-principles-of-testing/?wpdmdl=1151

 

Without this data, we cannot tell if all samples were suitable for being included in a test set or if any of the samples was found on a computer with ESET installed.

FYI - MRG responded on Wilders that they did provide Eset test samples: hxxp://www.wilderssecurity.com/threads/mrg-effitas-360-assessment-certification-q3-2016.389784/#post-2629924

[Marcos 5,257]

FYI - MRG responded on Wilders that they did provide Eset test samples: hxxp://www.wilderssecurity.com/threads/mrg-effitas-360-assessment-certification-q3-2016.389784/#post-2629924

Yes, but that statement is not correct. We have been in contact with Sveta and waiting for correction of the statement. I won't go into details as this is rather something that should not be discussed publicly.