ESET Smart Security performance in the MRG Effitas 360 Assessment & Certification Programme Q2 2014

[Chris_MRG 6]

We published the MRG Effitas 360 Assessment & Certification Programme Q2 2014 last night.

 

You can download the report from our website - www.mrg-effitas.com

 

Congratulations to ESET for passing the test.

 

Best regards,

 

Chris.

 

 

[rugk 397]

Thanks for sharing.

 

Here is the direct link: https://www.mrg-effitas.com/wp-content/uploads/2012/06/360-Q2-2014.pdf

 

ESET passed the test, but wasn't certificated (so only "level 3 pass").

But as I see level 3 is still quite good, so nothing to worry.

 

PS: You're post was pinned.

[Marcos 5,071]

1, ESET ranked 2nd if you consider that the 2nd product had 1,3% of user-dependent detections/blocks which is 98,7% if you don't take such blocks into account (imagine how many ordinary users can respond properly to UAC, firewall, HIPS prompts).

2, The difference between the first and 3rd place are 3 missed samples which is nothing compared to dozens of thousands of new malware variants that emerge on a daily basis.

3, There is nothing like 100% protection against malware and it all boils down to the test set.

4, Achieving level 3 is great, otherwise you'd expect 100% malware detection which is impossible to achieve without false positives. 100% detection without FPs can only be achieved on quite limited test sets.

[rugk 397]

Okay, great!

[Chris_MRG 6]

Hi Marcos,

 

I see what you say. There is no perfect test methodology, but our friends at AVC etc use this same metric.

 

We discussed confidence intervals last week at WATeR after the AMTSO meeting and we will continue to lease with Juraj, Righard etc concerning testing matters.

 

Again, thank you for your observations.

 

Cheers,

 

Chris.

[Marcos 5,071]

Hi Marcos,

I see what you say. There is no perfect test methodology, but our friends at AVC etc use this same metric.

 

Yeah, I didn't mean to object the methodology at all as we all understand that it's not possible to test every single piece of malware for obvious reasons Personally I like these tests as you use actual malware that you download from the internet and then you also execute samples.

[Chris_MRG 6]

Indeed, this is, in our opinion the only valid way of testing. Its time consuming (do the maths and you will see that in this single assessment, our guys had to perform just under 12,000 individual tests!) but you get results that map most closely to real world use cases.

 

Also, we think including the time to detect missed samples is an interesting metric - as you don't want these things running on your endpoint for any great length of time - particularly financial malware.

 

We really strive to make testing accurate because if you cant measure a products efficacy accurately and appropriately, how can we help the vendors actually improve their products performance in a way that benefits their clients?

 

We have been involved in recent industry discussions and it emerges that some vendors spend literally millions $ in engineering time, altering their product so that it performs better in certain types of tests (I'm thinking here of tests which measure the increase in system boot time and steady state load etc) - whilst these tests do have a value in some sense, some vendors feel this money and resources would be better spent in other areas which actually helped improve protection for clients.

 

The consequence of the above is that vendors can see testing as a "cost" rather than an "investment" - in our view this is why we have always tried to present ourselves as an efficacy assessment and assurance house as opposed to simply a testing lab.

 

The majority of our work is private external quality assessment services - to serve as an independent and on-going benchmark, helping vendors ensure their product is running as they would like. We also are the worlds largest supplier of malicious URLs and binaries (300-500 thousand unique binaries each day and around 250 thousand URL each day) and supply these to the majority of vendors to help them protect their customers better. We also supply our feed to some testing labs - for instance, if you look at the tests our friend Neil Rubenking conducts at PC Mag, you will see he uses our feed.

 

In any case, we are always open to discussion and welcome constructive feedback.

 

Cheers,

 

Chris.

Edited October 29, 2014 by Chris_MRG

[rugk 397]

Very thanks Chris and the complete Effitas Group.

I'd like to say that not many av test institutes go into the forums of the av vendors and personally post their tests, even explain how they test and discuss a bit in a very friendly way.

Thank you again!

[Marcos 5,071]

Also, we think including the time to detect missed samples is an interesting metric - as you don't want these things running on your endpoint for any great length of time - particularly financial malware.

 

Indeed. This is another thing I like a lot about these tests. It really matters if a product starts to detect missed malware within minutes/hours or days/weeks and this aspect is usually not reflected in other tests that other testers conduct.

[SweX 871]

We also supply our feed to some testing labs - for instance, if you look at the tests our friend Neil Rubenking conducts at PC Mag, you will see he uses our feed.

Yeah he does mention that in his AV reviews. 

 

And only because you mention him, I thought why not link to his ESET V8 reviews of both NOD32 and ESS that I read a while back.

 

NOD32

hxxp://www.pcmag.com/article2/0,2817,2469847,00.asp

 

ESS

hxxp://www.pcmag.com/article2/0,2817,2469978,00.asp

 

I generally think that what the testing orgz does is of higher quality these days compared to a couple of years back, and that it is good they spend a lot of time on the tests, because even if it may be time consuming I do think that the time spent is needed or else the quality will be lower, though there are some exceptions that I think can improve quite a bit in several ways, but I keep that to myself 

 

And welcome to the ESET forum Chris 

Edited October 30, 2014 by SweX

[rugk 397]

From the NOD32 test:

ESET apparently doesn't exclude known safe files from future scans the way F-Secure Anti-Virus 2015 and Trend Micro do.

Yes it does! (It excludes known safe files!)

Maybe it only didn't do this, because he ran an in-depth-scan before he tested that.

Edited October 30, 2014 by rugk

[SweX 871]

From the NOD32 test:

ESET apparently doesn't exclude known safe files from future scans the way F-Secure Anti-Virus 2015 and Trend Micro do.

Yes it does! (It excludes known safe files!)

Maybe it only didn't do this, because he ran an in-depth-scan before he tested that.

Yeah, I also saw that when I read it 

[scottls59901 1]

ESS vs Kaspersky vs Emsisoft (I've used all 3 extensively)-

1. Kaspersky (KIS 2014)-

-Gives Excellent 100% protection, but slows your system.

-Updates every 2hrs, but the system is Slow for 10min... afterwards (makes it's not-so-Quick scan).

-Fast Scans!- Remembers scanned files, and doesn't scan them again unless modified... (I don't know if this good, as new definitions may detect...?).

-You have to Open/close your browser 3 times (for 3min...), before it gets Fast (Hotmail...), and drove me Nuts!

 

2. Emsisoft, with firewall (bought old Online Armor FW)-

-Good detection rate, and system is fairly fast.

-Bombards you with alerts, that Really are hard to answer (tells you what others have answered...!?- Lemmings...?).

-Written in old A2 non-MS language, and is Not always recognized in Security Center (warnings, and they want you to disable)!

-Firewall is problematic!- They want you to Uninstall in Safe Mode, with every update, then reinstall frequently.

 

3. ESS-

-Good detection rate, and user friendly (doesn't slow my systems much).

hxxp://threatcenter.crdf.fr/?Stats#null (rates NOD #1!?).

-Slow to respond to new threats with infrequent updates (Consumer Reports dinged ESS on this!)!

 

My Bottom Line- I'll stick with user friendly ESS, and do on-demand scans with free-

old Fast MBAM v1.75 (No Pro!) hxxp://filehippo.com/download_malwarebytes_anti_malware/comments/14815/,

Kaspersky hxxp://usa.kaspersky.com/downloads/free-anti-virus-scan,

Emsisoft Emergency Kit hxxp://www.emsisoft.com/en/software/eek/

Edited October 30, 2014 by scottls59901

[Marcos 5,071]

ESS vs Kaspersky vs Emsisoft (I've used all 3 extensively)-

1. Kaspersky (KIS 2014)-

-Gives Excellent 100% protection, but slows your system.

There's nothing like 100% protection. There are many malware samples that KAV won't protect you from while ESET would and vice-versa.

 

3. ESS-

-Good detection rate, and user friendly (doesn't slow my systems much).

hxxp://threatcenter.crdf.fr/?Stats#null (rates NOD #1!?).

-Slow to respond to new threats with infrequent updates (Consumer Reports dinged ESS on this!)!

Not sure what you mean by slow response to new threats. ESET uses mechanisms to ensure that recognition of new threats is added immediately while it takes hours or even days or weeks for some other vendors to add detection.

[SweX 871]

 

3. ESS-

-Good detection rate, and user friendly (doesn't slow my systems much).

hxxp://threatcenter.crdf.fr/?Stats#null (rates NOD #1!?).

-Slow to respond to new threats with infrequent updates (Consumer Reports dinged ESS on this!)!

Not sure what you mean by slow response to new threats. ESET uses mechanisms to ensure that recognition of new threats is added immediately while it takes hours or even days or weeks for some other vendors to add detection.

 

I assume he means that ESET does not download a VSD update every hour like some other products. But afaik the rapid response module can get updated on the hourly checks even if no new VSD is downloaded. And we have Live Grid wich can block malware for all users before a VSD has been released. Plus all the other tech inside. I guess we can say that the protection and detection abilities does not stand still only because the VSD number doesn't change every hour. 

 

And it's not that 1 signature added to the VSD is able to detect 1 piece of malware it doesn't work like that.

Edited October 31, 2014 by SweX

[rugk 397]

Just FYI ESET downloads new signatures every hour, but there isn't released a new VSD every hour.
(Normally 3-6 VSDs per day)

Edited December 18, 2014 by rugk

[SweX 871]

MRG Effitas Online Banking Browser Security Certification Project Q3 -2014

 

https://www.mrg-effitas.com/mrg-effitas-online-banking-browser-security-certification-project-q3-2014/

[Utini 1]

API hooking seemed to fail a bit with ESET but the rest look very very good?

[Marcos 5,071]

API hooking seemed to fail a bit with ESET but the rest look very very good?

 

Who cares HIPS, Advanced memory scanner (AMS) or Exploit blocker (EB) are not behavior monitors that would spring into action when a suspicious behavior is detected. Unless the code in memory is indeed malicious and resembles known malware, it will be detected and suspended by AMS which cannot be the case of a simulator. Likewise EB is triggered when malware attempts to exploit a known vulnerability which is again not the case of the simulator used in this case.

[Utini 1]

 

API hooking seemed to fail a bit with ESET but the rest look very very good?

 

Who cares HIPS, Advanced memory scanner (AMS) or Exploit blocker (EB) are not behavior monitors that would spring into action when a suspicious behavior is detected. Unless the code in memory is indeed malicious and resembles known malware, it will be detected and suspended by AMS which cannot be the case of a simulator. Likewise EB is triggered when malware attempts to exploit a known vulnerability which is again not the case of the simulator used in this case.

 

 

Awesome The more I use ESS the more I like it

[SweX 871]

API hooking seemed to fail a bit with ESET but the rest look very very good?

I know too little to even comment about them, but it has been the case with products before, that they react differently when the "real deal" shows up, compared to a simulator that is supposed to simulate a situation. So it is possible that applies to several of the other products that failed that part too. But I know too little so I am not going to speculate that much about it.

[Patch 16]

Who cares HIPS, Advanced memory scanner (AMS) or Exploit blocker (EB) are not behavior monitors that would spring into action when a suspicious behavior is detected. Unless the code in memory is indeed malicious and resembles known malware, it will be detected and suspended by AMS which cannot be the case of a simulator. Likewise EB is triggered when malware attempts to exploit a known vulnerability which is again not the case of the simulator used in this case.

 

They do state that they have actually set up a malicious network and record fails when they can get a password from the system under test to their servers. An impressive test setup, and to fail it I would care.

 

I may have missinterpreted their description though, as you are implying that test protocol did not apply to the API testing, perhaps only applying to the Botnet test.

Edited December 20, 2014 by Patch

[Chris_MRG 6]

Guys, we have published our Q3 360 data on our website www.mrg-effitas.com.

 

Well done to Righard, Juraj and the team - ESET passes and gets our certification!

 

Cheers,

 

Chris.

[SweX 871]

Guys, we have published our Q3 360 data on our website www.mrg-effitas.com.

 

Well done to Righard, Juraj and the team - ESET passes and gets our certification!

 

Cheers,

 

Chris.

Thanks Chris and Merry Christmas.

 

I started a thread in the General (non-ESET support-related topics) section earlier today: https://forum.eset.com/topic/3842-mrg-effitas-360-assessment-q3-2014/