ESET behind Microsoft in July AV-comparatives

[novice 20]

ESET:99.1%

MSE: 99.7%

https://chart.av-comparatives.org/chart1.php

[cyberhash 188]

My Dogs reaction to this news

[novice 20]

Yes, is sad news, indeed.

[SCR 195]

Wow, I sure am glad July was over 15 days ago considering that earth shattering news  

[itman 1,659]

There is an A-V Comparatives disclaimer posted in the full .pdf report version immediately below the test result chart shown by the previously posted link. I can't copy it verbatim since the report is copyrighted. What the disclaimer states in effect is that the AV products tested could be 100% effective against other malware samples. Further, the test results only apply to the 378 samples used out of the millions in existence.

In other words and entirely possible is that the test samples used were ones almost all AV products would detect; not just the ones specifically tested. This assumption is further supported by the 95+% detection by all products tested. Of additional note is that AV-C in their new revised tests no longer penalize AV vendors for false positives. If those were factored in the test results, MSE having the third highest recorded rate would most certainly be ranked in the bottom third of products.

To state that MSE has a better malware detection rate than Eset based on this test is naïve and ludicrous. 

Edited August 16, 2017 by itman

[0xDEADBEEF 43]

55 minutes ago, itman said:

There is an A-V Comparatives disclaimer posted in the full .pdf report version immediately below the test result chart shown by the previously posted link. I can't copy it verbatim since the report is copyrighted. What the disclaimer states in effect is that the AV products tested could be 100% effective against other malware samples. Further, the test results only apply to the 378 samples used out of the millions in existence.

In other words and entirely possible is that the test samples used were ones almost all AV products would detect; not just the ones specifically tested. This assumption is further supported by the 95+% detection by all products tested. Of additional note is that AV-C in their new revised tests no longer penalize AV vendors for false positives. If those were factored in the test results, MSE having the third highest recorded rate would most certainly be ranked in the bottom third of products.

To state that MSE has a better malware detection rate than Eset based on this test is naïve and ludicrous. 

Yes the sample size is small.. but if you look at past tests (starting this January), you will find that ESET is consistently low-ranked in these tests. What I want to know is that in their tests, if the "compromise" means the malwares are indeed effective (in other words, mimic the real world scenario) and penetrate the protection layers (e.g. somehow successfully executed without being detected). If both are true, then ESET indeed consistently missed something and needs improvements.

[SCR 195]

What I would like to know is where does the money come from to support these so called "testing" organizations.

[itman 1,659]

24 minutes ago, SCR said:

What I would like to know is where does the money come from to support these so called "testing" organizations.

You're getting "warm" .................... Microsoft is a major research grant funder. So its contributions to the below listed organizations can in turn "influence" their funding of AV-C. Always "follow the money flow." 

Quote

 

Sources of Income 

In the beginning, Andreas Clementi was hoping to finance the project with donations from users. Unfortunately, this was not viable, as only two or three users made significant donations, despite an appeal. Other sources of income had to be found to keep the project going. There was much debate as to how to support AV-Comparatives without compromising its most important quality, namely its neutrality. Payment must not be allowed to have any influence on test results.

The solution actually turned out to be very simple: if all manufacturers pay the same fee in order for their product to be tested, none of them can be advantaged or disadvantaged. In several cases it happens that a vendor is tested even if it do not apply for it. In this case, the costs will be covered either by the magazines or by other independent parties, which requested the results.

As AV-Comparatives also receives subsidies (see below), the test fees are amongst the lowest of all such testing institutes, despite the sophisticated testing methods.

Subsidies

Subsidies are an important part of AV-Comparatives‘ income. We are supported by academic and public bodies, and the Austrian state. Subsidies may be direct or indirect. They may take the form of financial support for research and development, or the payment of staff costs, or support from academic staff of the universities for specific projects. In co-operation with the universities, bachelor’s and master’s theses are used for working on specific problems.

• University of Innsbruck, Research Group for Databases and Information Systems
• University of Innsbruck, Research Group Quality Engineering
• Hong Kong University of Science and Technology
• National Taiwan University of Science and Technology
• Brandeis University Mississippi, Computer Science Department
• Management Center Innsbruck

 

https://www.av-comparatives.org/funding/

Edited August 16, 2017 by itman

[itman 1,659]

As far as security solution testing overall, there is only one AV Lab that I am aware of that does it properly; NSS Labs in Austin, TX.

To properly test security software, there should be no selection of samples and the determination of what samples are appropriate. Rather malware detection should be performed on a 7/24 continuous basis over a pre-determined interval. This usually is 3 months. To facilitate like monitoring, an array of "honeypot" servers is utilized to capture malware "in-the-wild." When this technique is employed what will be observed is not only an "ebb and flow" in malware activity, but the same like behavior in security software detection. At the end of the pre-selected period interval, an overall average detection rate is calculated for security products being tested.

There is documented scientific evidence why continuous monitor is the only correct way to detect malware. It is based on the fact that malware will modify its delivery and behavior depending on its intended targets. This is most pronounced during the initial infection campaign phase which usually is no more than a couple of weeks and many times, only a few days. After that period, malware for delivery purposes "is used up" as far as effectiveness goes. Continuous monitoring is also extremely effective is determining mean-time- to-mitigation(MTTM) effectiveness for a given security product as evidenced by time for signature deployment or behavior detection "tuning." Obviously, this would be shown as a zero time interval for immediate detection of 0-day malware. 

NSS Labs uses its continuous security validation platform(CAWS) to perform the above activities. AV-Test also claims to monitor malware on a continuous basis via honeypot method but I believe that is to capture malware samples only. Eset does participate in NSS Labs testing but only for its Endpoint products. NSS Labs funds itself by charging $$$$ for product test reports with most starting at $1000 upwards. "Once in a blue moon" NSS Labs will run a consumer security product comparative with the report available free to the general public. I believe the last time they did so was a couple of years ago and Eset was a top ranked scorer.    

[SCR 195]

Thanks itman.

That pretty much falls in line with my thoughts as to the source of their funds. It also confirms my feelings toward the results of their tests. That is, read them, think a bit about them and then return to "my" own on going "Test."

Test Status: Perpetual

Method: Daily Use of two Windows 7 Computers and several prior versions of a Windows OS

Product Tested: Nod32, ESS, EIS and several beta's of each

Length of test to date: In excess of 10 years, probably 15 or more I can't remember.

Number of Infections: Zero

Number of False Positives: Zero

Results to Date: Eset Products prevented infection to my systems 100% of Real World time.

Conclusion: I'll continue to use the 100% effective Eset products.

Edited August 16, 2017 by SCR
Info Correction

[0xDEADBEEF 43]

39 minutes ago, itman said:

As far as security solution testing overall, there is only one AV Lab that I am aware of that does it properly; NSS Labs in Austin, TX.

hmm interesting.. Reminds me of this: https://www.eset.com/ca/about/newsroom/corporate-blog/esets-position-on-nss-labs-advanced-endpoint-protection-10-test/

[0xDEADBEEF 43]

32 minutes ago, SCR said:

Eset Products prevented infection to my systems 100% of Real World time.

That's interesting also, seems my test samples are so special that I occasionally encounter ones that bypass all protection layers of ESET

[Marcos 5,074]

12 minutes ago, 0xDEADBEEF said:

That's interesting also, seems my test samples are so special that I occasionally encounter ones that bypass all protection layers of ESET

Which one(s)? I'm eager to check them closer. Also are you talking about executables? Because the situation is a bit different if, for instance, a script malware (downloader) is not detected but the payload is so in the end the computer doesn't get infected.

[itman 1,659]

17 minutes ago, 0xDEADBEEF said:

hmm interesting.. Reminds me of this: https://www.eset.com/ca/about/newsroom/corporate-blog/esets-position-on-nss-labs-advanced-endpoint-protection-10-test/

I strongly suspect NSS Labs regrets getting involved with the testing "shenanigans" done earlier this year in regards to Cylance and a select group of other Next Gen/AI vendors. They got publically "blasted" and sued by Cloudstrike over the same. But who knows, "money talks" and definitely influences any corp. activities. But in the case of many of the Next/Gen AI vendors it is also "political hooks" they have since many are managed by ex-CIA and other "Black Ops" employees.

[itman 1,659]

Getting back to Windows Defender specifically, Microsoft has publically announced all the new protections that will be available in Windows Defender ATP with this fall's release of Win 10 CE. Note that WD ATP is only available to Enterprise OS versions by paid subscription:

Quote

Detecting suspicious PowerShell activities, code injection, and malicious documents

Ref.: https://blogs.technet.microsoft.com/mmpc/2017/08/03/windows-defender-atp-machine-learning-detecting-new-and-unusual-breach-activity/

I find this illuminating for a couple of reasons. It clearly implies that "plain" Windows Defender prior, currently, and in the future does not and will not have like protections or any of the new "advanced" protections MS is implementing.

The following truism is most appropriate in regards to these discussions " you can lead a horse to water, but you can't make him drink it." 

Edited August 16, 2017 by itman

[itman 1,659]

45 minutes ago, 0xDEADBEEF said:

That's interesting also, seems my test samples are so special that I occasionally encounter ones that bypass all protection layers of ESET

One comment in this regard. Testing malware out of the context of how it was delivered in the actual malware attack can and will affect the detection of same by security solutions.

[0xDEADBEEF 43]

29 minutes ago, Marcos said:

Which one(s)? I'm eager to check them closer. Also are you talking about executables? Because the situation is a bit different if, for instance, a script malware (downloader) is not detected but the payload is so in the end the computer doesn't get infected.

Yes I am talking about executables. I don't consider those macro malware or script downloader that eset failed to detect initially as a "bypass", unless the downloaded payload execute successfully and do something bad without being detected.

Generally, ESET is impressive in detecting most threats (it is much harder for me to capture a fresh malware before ESET does after all). One example is my post of that ransomware sample last time, there are some others (very few), and I'd be glad to notify ESET if I spot others in the future.

[0xDEADBEEF 43]

1 minute ago, itman said:

One comment in this regard. Testing malware out of the context of how it was delivered in the actual malware attack can and will affect the detection of same by security solutions.

Agreed. But it is hard to reproduce exactly the same scenario of how it is originally delivered. Also, user initiated download and execute is still one major way of getting infected. I personally consider the protection against payload itself as equally important, even though it might be intercepted by other defense layers first.

[SCR 195]

1 hour ago, 0xDEADBEEF said:

That's interesting also, seems my test samples are so special that I occasionally encounter ones that bypass all protection layers of ESET

Who said anything about test samples? I'm conducting a Real World Test. Our test methods are completely different. I'm not throwing rocks at my front window because they told me it was unbreakable.

As the end user I am more interested in how a product works in my world which, of course, is the only Real World that matters to me. My result is of an average user doing average things on a high speed Internet connection. I'm not an IT Security Specialist just a happy Eset customer.

To clarify, I am currently using EIS on both systems one set to Automatic Mode and one to Interactive Mode. I don't go around the Internet looking for trouble, no rock throwing allowed.

Eset has sent up warnings a few times in each system. We follow the advice of the warning and carry on. That's it 100% protection in my Real World.

To be honest the real World Test is on the machine set on Automatic mode and uses the default setup, shy of me setting up the network and printer, and used by my wife who has had some very basic security training. Basically if anything pops up, especially in red, to  ask me. If I'm not around close the browser and open a new one and don't go wherever you were.

I really have no idea why people manage to get their machines infected. There is plenty of information on line, in books, etc. to get a clue of how to safely surf and use the Internet. I post here because average users read these posts. This negativity about one program having a better detection rate then Eset by 0.07% prompted me, a average user, to let them know the Tests conducted by these Testing companies does not necessarily reflect what their experience will be as an average user. Most of this stuff is way over my head however I am curious as to what makes things tick.

[persian-boy 22]

 

23 hours ago, MSE said:

ESET:99.1%

MSE: 99.7%

https://chart.av-comparatives.org/chart1.php

Personally, I don't give a..... about these av test companies... you should read the Testing Methodology then judge! these tests are totally bs! just stay away from it:P

Edited August 16, 2017 by persian-boy

[novice 20]

7 hours ago, itman said:

To state that MSE has a better malware detection rate than Eset based on this test is naïve and ludicrous.

And why is that???

Feb to June 2017 .......MSE 98.8%     ESET  98.5%

June 2017 ......            MSE  100%     ESET  98%

May 2017                    MSE  99.5%    ESET  97.7%

Apr 2017                      MSE  99.3%    ESET 98.7%

 

"You can fool all the people some of the time, and some of the people all the time, but you cannot fool all the people all the time."
 

Edited August 16, 2017 by MSE

[Marcos 5,074]

I don't know what you are trying to prove but it's not a big problem to "make" a test where ESET will detect 100% of actual new borne malware while MSE none and publish such "tests" with different malware on a regular basis.

[cyberhash 188]

My dog is depressed with the av-comparatives test forum posts as you seen with the photo above. He thinks that people should consider looking at these tests only "IF" their current av product has actually let them down and ended up infected because of the "Supposed" poorer detection.

 

[novice 20]

1 hour ago, Marcos said:

I don't know what you are trying to prove but it's not a big problem to "make" a test where ESET will detect 100% of actual new borne malware while MSE none and publish such "tests" with different malware on a regular basis.

This is not a "made" test to fit MSE, is a test performed by AV comparatives , a reputable company, and all players had the same chance to prove their detection rate.

Leaving  MSE alone, from Feb to June ,ESET is on 16th place from 21, with 7 antiviruses having 100% detection.

As an ESET developer, I fell for you, but the reality is that ESET is no longer what it used to be.

 

Thanks,

 

 

 

Edited August 16, 2017 by MSE

[TomFace 539]

2 hours ago, MSE said:

This is not a "made" test to fit MSE, is a test performed by AV comparatives , a reputable company, and all players had the same chance to prove their detection rate.

Leaving  MSE alone, from Feb to June ,ESET is on 16th place from 21, with 7 antiviruses having 100% detection.

As an ESET developer, I fell for you, but the reality is that ESET is no longer what it used to be.

 

Thanks,

 

 

 

Oh yes...every one is a "reputable company". because they said they were. You sound a whole bunch like malkil another new puppy around here. Are you 2 one in the same being?

80% or so of the A/V test mean nothing in the real world. This has been gone over and over for years.,

Just why are you (and malkil for that matter) here?  What are you trying to prove? Are you just here to TROLL?

If so you are not welcome here.

This forum is to help users of ESET products with real usage issues. Not to be trolled.

 

Edited August 17, 2017 by TomFace