A Few Recent AV Lab Tests Where MSE and WD Didn't Do Very Well

[itman 1,659]

Since a few forums members like to note how well that Windows Defender and MSE have been scoring lately in recent AV lab tests, it is appropriate to show that test results are very much AV Lab dependent.

First up is the latest comparative done by SE Labs in the U.K.. This lab does not receive compensation from any of the AV vendors tested. As such, it has no incentive to "stack the malware sample" deck by choosing samples that MSE and WD are most sensitive to; namely those that are "most prevalent."

Test result - MSE scored last.

Ref.: https://selabs.uk/download/consumers/apr-june-2017-consumer.pdf

The next test is the most recent Malware Research Group 2nd Quarter 360 test. This test has a number of unique features such as noting security product effectiveness by malware category; general, ransomware, financial, and PUA's. Also was tested product effectiveness from external device based malware.

For the first time every, MRG separated test results for Windows Defender with and without SmartScreen enabled. This was done as a result of complaints by some in the various web security forums that WD was not being fairly tested with SmartScreen disabled. The important thing to note is the following:

• This test was done on Win 10 with Edge as the browser employed. So both native and browser based SmartScreen were being employed in regards to Windows Defender detection scoring. If you are using a non-Microsoft browser or SmartScreen is disabled in IE11 or Edge, the Windows Defender column w/o SmartScreen should be used in determining what protection is given.

Test result - WD scored last among major AV products tested whether SmartScreen was enabled or not. Without SmartScreen employed, WD had a failure rate of 23%.

Ref.: https://www.mrg-effitas.com/wp-content/uploads/2017/08/MRG-Effitas-360-Assessment_2017_Q2_wm.pdf

Eset along with all the other major AV vendor products do not condition their malware effectiveness based on what browser is used or for that matter, what ver. of Windows is being used. WD maximum effectiveness is only achievable on the Win 10 ver. 1607+ platforms since only those incorporate the recent security enhancements Microsoft did to WD.

Edited August 27, 2017 by itman

[0xDEADBEEF 43]

The result from SE Lab is a bit counter-intuitive. e.g. Norton generally generates many of FPs in real life use (not only I myself, but also the case from the feedback of many other people). This makes me wondering what their sampling method and sample size are.

This also reminds me of AVC's malware protection test. Their test shows that ESET's score is identical before/after execution (so no dynamic detection, unless they also count AMS into "scan"). This strongly implies that their testing samples are too old to reflect the real-world situation.

Edited August 28, 2017 by 0xDEADBEEF

[itman 1,659]

13 hours ago, 0xDEADBEEF said:

strongly implies that their testing samples are too old to reflect the real-world situation.

They get their samples same place as AV-C and AV-Test; the AMTSO malware database. If the samples were old, I would expect a higher MSE score. If you follow VT signature detection stats, Microsoft is always the latest to detect.

Don't know what to say about Norton FPs. In the last AV-C test, they have 5. However on the last Win 10 test done by AV-Test, they had 0. So I guess it depends on whatever samples exist in the AMTSO database at the time the test was performed. -EDIT- SE Labs also uses a rather complex system for arriving at a FP determination. You can read about that starting at section titled, 5.1 Interaction Ratings. Based on what I read there, I would say they are a bit more "lenient" in their FP classification that other labs are.

Edited August 29, 2017 by itman

[SCR 195]

As far as I'm concerned Microsoft lost any credibility they may have had as a company on the sneaky back door Win 10 release. That's just not the correct way to treat customers.

Looks like they're still at it.

[novice 20]

3 hours ago, SCR said:

As far as I'm concerned Microsoft lost any credibility they may have had as a company on the sneaky back door Win 10 release. That's just not the correct way to treat customers.

Looks like they're still at it.

What is the relationship with the subject at hand????

[SCR 195]

14 hours ago, John Alex said:

What is the relationship with the subject at hand????

It's more then obvious to me.