Jump to content

AV-Test results for July & August (ESS Version 7 tested) - thoughts?


TJP

Recommended Posts

  • ESET Insiders

Full scoreboard of results: hxxp://www.av-test.org/en/?avtest[type]=3&avtest[platforms]=5-4-1&avtest[series]=10-7:1396310400.2,5-4-1:1406851200.2

 

I find most AV tests interesting to view however I don't place a lot of stock in the results. This is because of different testing methodologies that are used as well as tests being paid for by a sole AV vendor (which may or may not skew test results). That said, I do find the latest AV-Test for July and August very curious.

 

AV-Test latest report reveals ESS earnt 5.5/6.0 for both detection (99% for July & 100% for August) and useability and just 3.5/6.0 for performance, dropping the overall result of ESS down the list. I'm sorry to rebuke their findings as I'm no expert it AV testing but I'm at a loss to explain the ESS performance rating.

 

I know what I'm about to post is purely anecdotal - I've found ESS is the lightest of any AV I've had on my PC which is why I bought a licence at the beginning of 2014 after many years using a well known Russian AV solution. The aformentioned Russian AV scores a perfect 6.0/6.0 with zero performance loss according to AV-Test and yet it's installed on my father's PC and kills system performance stone dead and is well known on various security forums for being a drag on system resources.

 

Can anyone explain why ESS continues to score so poorly in the performance in this test?

 

Cheers.

Edited by TJP
Link to comment
Share on other sites

Hi TJP!

 

You are not the only one wondering about this that's for sure.

 

The "performance" part in the tests by AV-Test has always been a bit confusing.

 

I asked Marcos couple years ago why ESET always get such negative performance results and he had no idea.

 

They changed this since a year or so back, the "performance" part was included under usability before.

 

Now the industry average is 3s, before the average was 10s afaik, so products that really affected the performance could score 12s or more. ESET was mostly between 6-8s.

 

Before they changed no product got 0s like several products does now, you can even wonder how a product can get 0s and what does that mean exactly.   :huh:

 

I would rather see a percentage system the higher the percentage the more the product affects the performance. Like AV-C does it's easier to understand for most people.

 

Edit: all older tests are on the site if you can figure out how it works(I did after a while), it's about load/unload the content, click on one test and it will load, click again and it will unload. And you maybe got some seriously scrolling to do if you happen to load several tests at the same time. And click "search the archive for past tests" several times and they will popup one by one.

 

One can see that the "industry average" on the performance/usability part has gone from an average of 251s to 3s. And some products get 0s in the recent tests.

 

Let's pick out a few ESET examples from the past to the latest.

 

Win 7 Aug 2014

Avg 3s - ESET 5s

3.5 / 6.0

 

Win XP June 2014

Avg 3s - ESET 3s

4.5 / 6.0

 

Win 8.1 April 2014

Avg 3s - ESET 6s

3.5 / 6.0

 

Win 7 June 2013

Avg 6s - ESET 11s

2.5 / 6.0

 

And to answer your question, no I can't explain as I don't understand the performance part either. ;)

 

Cheers

Edited by SweX
Link to comment
Share on other sites

Yes it's quite strange.

 

Here an overview with ESET, Kaspersky (the Russian AV mentioned above) and AhnLab (as a bad example) from a few tests:

hxxp://www.av-test.org/en/?avtest%5Btype%5D=3&avtest%5Bprods%5D=11,15,29&avtest%5Bplatforms%5D=3,10-7,5-4-1&avtest%5Bseries%5D=5-4-1:1406851200.2,5-4-1:1391252400.2,10-7:1396310400.2,10-7:1385899200.2,3:1401580800.2,2:1291201200.3

 

PS: Also interesting that they didn't tested the cooperate solution from ESET.

 

@SweX Yes 10 sec. was e.g. average performance inpact in 2012. See the test of ESS in this year.

And there they had "repair" instead of "performance".

Link to comment
Share on other sites

  • ESET Insiders

Thanks SweX and rugk!

 

The reason why I call into question these results are they simply do not gel with my own long term experience; anyone skim reading these tests (and who outside an enthusiast is going to read every word of each test result?) would believe ESS is a terrible resource hog which it simply isn't. Other vendors score a perfect 6.0 which doesn't come close to what I've seen in my own real world usage.

 

I don't name Kaspersky outright because I want to avoid comments that could be viewed as libelous and/or slanderous. I used their products for 5 years (and Nod32 prior to that) and had mostly trouble free years until early this year. I'm still very happy they forced me to change as I couldn't be happier with ESS.

Link to comment
Share on other sites

Hey,

 

Yes that's my point too, if ESET affected the performance like the results shows, then no one of us would probably use ESET.

 

Good system performance and lack of bloat is very important for me, and most likely to all of the above average ESET users too. Unlike the average user that usually don't know the difference between any of it or don't care much about that stuff.

Edited by SweX
Link to comment
Share on other sites

Yes also interesting is this twitter post from AV-test:
 

The results published at our website are correct, you can find them online and also downloadable at hxxp://www.av-test.org/en/press/test-results/

https://twitter.com/avtestorg/status/517386006296018945
 
Has somebody said something else? ;) 

Link to comment
Share on other sites

Has somebody said something else? ;)

Maybe someone sent them the link to this topic. ;)

I can name at least 6 products have higher performance mark than ESET but my real experience about them is way worse than ESET.

I have one desktop which I bought in 2006 is using Celeron CPU.

And the only AV I can install is ESET.

I tried some "less resource hunger" AVs but my computer was slow down significantly.

All I could do was to look at the monitor and wait. :D

Sometime, I wonder if some AVs have mechanism of detecting testing tools so that they have behave in difference way which uses less resource.

Edited by klv12gcn
Link to comment
Share on other sites

@klv12gcn

 

Nice test. I assume you have tested (some of) the AVs where AV-Test "claims" that it is better than ESET.

Just for completeness: How many GBs of RAM had your test PC?

 

Sometime, I wonder if some AVs have mechanism of detecting testing tools so that they have behave in difference way which uses less resource.

Interesting speculation, but I think it would be more useful to adapt this based on the system performance. (e.g. when a computer has many RAM and it isn't used actually so much, it could temporarily use a bit more RAM, but if the computer has not so many RAM then it could use less)

Link to comment
Share on other sites

@klv12gcn

 

Nice test. I assume you have tested (some of) the AVs where AV-Test "claims" that it is better than ESET.

Just for completeness: How many GBs of RAM had your test PC?

 

Sometime, I wonder if some AVs have mechanism of detecting testing tools so that they have behave in difference way which uses less resource.

Interesting speculation, but I think it would be more useful to adapt this based on the system performance. (e.g. when a computer has many RAM and it isn't used actually so much, it could temporarily use a bit more RAM, but if the computer has not so many RAM then it could use less)

Bold, if I read what you write correcly, then that is how other AV's that like to show off with low ram usage does it, and it never increases the system performance only the opposite. The ram usage will be lower at times, but the system will not feel any faster. 

 

I'm no expert but someone said it's about loading/unloading parts of the program in RAM that's not needed at the moment.

I find that approach pointless because every part of the program is needed at some point. Let's say you have ten workers, but only 5 is working in the ram, when you need more workers you have to call them in to work, why do that when all ten workers can be in the ram, then you don't have to call them in as all ten workers are loaded in ram and ready to work all the time=better system performance.

 

"Look our AV is so light on the system"  :huh: Light on the system / and low resource usage, is two different things.

 

Regarding this test, afaik they don't give results based on the RAM usage(that would be ridiculous), but how fast/slow the program is in daily use (if it slows you down or not), 

 

For example...

1. AV can use 5mb of ram but slows down operation = bad results.

2. AV that uses 5mb of ram but doesn't slow down operations = good results.

3. AV that uses 100mb of ram but doesn't slow down operations = good results.

4. AV that uses 200mb of ram but doesn't slow down operations = good results

 

My experience is that AV's that are loaded in the RAM all the time will feel faster on the system despite (or thanks to) the higher ram usage. 

Vendors should have a focus to keep the CPU and I/O usage to a minimum, and that is hard to do if you aim to keep the RAM usage to a minimum.  

Edited by SweX
Link to comment
Share on other sites

Nice test. I assume you have tested (some of) the AVs where AV-Test "claims" that it is better than ESET.

Just for completeness: How many GBs of RAM had your test PC?

 

Interesting speculation, but I think it would be more useful to adapt this based on the system performance. (e.g. when a computer has many RAM and it isn't used actually so much, it could temporarily use a bit more RAM, but if the computer has not so many RAM then it could use less)

Good day, when I first used ESET for my PC, it had only 256 MB. And it worked fine! :D

I managed to upgrade it to 1GB few months ago.

Since then, I tested: 1 US product (its founder now is kind of playboy), 2 German AVs, 1 Romanian AV, 1 Russian AV, 1 Czech AVs. (I had a lot of free time back there)

My experience about performance is: ESET is the best, 1 German AV comes 2nd, Czech AV comes 3rd. The other is...

Edited by klv12gcn
Link to comment
Share on other sites

 

Nice test. I assume you have tested (some of) the AVs where AV-Test "claims" that it is better than ESET.

Just for completeness: How many GBs of RAM had your test PC?

 

Interesting speculation, but I think it would be more useful to adapt this based on the system performance. (e.g. when a computer has many RAM and it isn't used actually so much, it could temporarily use a bit more RAM, but if the computer has not so many RAM then it could use less)

Good day, when I first used ESET for my PC, it had only 256 MB. And it worked fine! :D

I am not surprised, just so you know my signature is 100% true ;)

Link to comment
Share on other sites

@Arakasi Yes nice to see you.

 

@SweX

Yes, you're right, but if a computer has not so much RAM and the AV is already using much of it can slow down the complete computer very much.

However it is a mixed blessing. If you have less RAM usage you may not have all things in RAM and this can speed down thing, but if you have very much in RAM the RAM usage could be too much.

 

@klv12gcn

And based on this it makes more sense - although it is a very unfair sense - too use many RAM if a speed test with a testing tool or something, so that e.g. copying is more fast.

 

However it may be difficult to detect this tools.

You don't know how exactly they test it and so we can only speculate. In their test results only stands what they tested:

Performance

Average influence of the product on computer speed in daily usage

 

Use cases: visiting websites, downloading software, installing and running programs and copying data
5 samples used

 

Interesting - 5 samples - ... quite less, aren't they? :huh:

 

So if we count:

  1. One time visiting a website
  2. One time downloading a software
  3. On time installing a software
  4. Running one program
  5. And copying one time (some) data

Or how do they mean these 5 "samples"?

Edited by rugk
Link to comment
Share on other sites

  • ESET Insiders

Performance

Average influence of the product on computer speed in daily usage

 

Use cases: visiting websites, downloading software, installing and running programs and copying data

5 samples used

 

Interesting - 5 samples - ... quite less, aren't they? :huh:

 

So if we count:

  1. One time visiting a website
  2. One time downloading a software
  3. On time installing a software
  4. Running one program
  5. And copying one time (some) data

Or how do they mean these 5 "samples"?

 

I installed the very latest version (2015) of the Russian internet security on my father's PC (i5 CPU, 8GB RAM, Win 8.1 x64) two weeks ago and I can tell you that it froze the PC on the following dot points:

 

2 - froze Firefox and the entire OS when downloading an ini file for ccleaner;

3 - froze the installation of Libre Office to scan the files (I presume);

4 - froze Firefox when adding it to the 'trusted programs' list.

 

The same program takes an age to load when Win 8.1 first boots and you can't access the UI while the green loading bar moves. On my PC with similar specs it takes ESS less than a third of the time of the Russian AV to load at start-up. If this constitutes a perfect score in performance, there is an Opera House in Sydney I'd like sell to you.

Edited by TJP
Link to comment
Share on other sites

So if we count:

  1. One time visiting a website
  2. One time downloading a software
  3. On time installing a software
  4. Running one program
  5. And copying one time (some) data

Or how do they mean these 5 "samples"?

 

The only way to get an "average" on the products is to perform the same test several times. Visiting a few websites ONE time will not give them an average, if they do it five times then they can get an average. And same for the other tests.

 

@SweX

Yes, you're right, but if a computer has not so much RAM and the AV is already using much of it can slow down the complete computer very much.

However it is a mixed blessing. If you have less RAM usage you may not have all things in RAM and this can speed down thing, but if you have very much in RAM the RAM usage could be too much.

 

Well if a user would run out of ram so the system stalls or similar I wouldn't blame that on the AV for being loaded in ram all the time, but that you have too little ram to do multitasking to begin with.  A web browser alone can easily use 500mb+ of ram.

 

If other vendors would load their AV in ram like ESET and how high the ram usage would be for those vendors (afaik) depends on several things and one is the database. If ESET's database is small as a Pea but another AV's database is big as an Apple, then the ram usage wouln't be around 100mb like ESET but way more. So even if it probably would increase that AVs system performance, users would complain on a unreasonable high ram usage 300+mb wich I guess is why so few vendors does that.

Edited by SweX
Link to comment
Share on other sites

So even if it probably would increase that AVs system performance, users would complain on a unreasonable high ram usage 300+mb wich I guess is why so few vendors does that.

Yes, I would do. :P

But I think the ~100MB RAM usage of ESET are very good for an AV.

And browsers are something different. They are not running compulsive in the background every time.

 

For me I'm always making a difference of background and foreground processes when observing the RAM usage.

 

But BTT:

 

So if we count:

  1. One time visiting a website
  2. One time downloading a software
  3. On time installing a software
  4. Running one program
  5. And copying one time (some) data

Or how do they mean these 5 "samples"?

 

The only way to get an "average" on the products is to perform the same test several times. Visiting a few websites ONE time will not give them an average, if they do it five times then they can get an average. And same for the other tests.

 

And... how did they do it?

Link to comment
Share on other sites

 

So even if it probably would increase that AVs system performance, users would complain on a unreasonable high ram usage 300+mb wich I guess is why so few vendors does that.

Yes, I would do. :P

But I think the ~100MB RAM usage of ESET are very good for an AV.

And browsers are something different. They are not running compulsive in the background every time.

Yeah I figured that  :P. Well, I only compared to the browsers ram usage wich can be extreme sometimes, but unlike the AV wich monitors everything in realtime, the browser process will disappear when you close it. So it doesn't matter that much if the ram usage is high while you use it. 

 

Yes of course, you and I know that. But 100mb can seem like a lot for inexperienced users that don't know how each AV works when they compare to for example Avast that flushes the ram every couple of seconds so the ram usage rarely goes above 10mb.

 

I am not sure how they do it exactly. We have to investigate.

 

This is the best info I could find from AV-Test at the moment:...........

 

"Performance Testing

13 typical actions are used to measure the influence on the system speed, for example:

  • Downloading various files from the Internet
  • Copying files locally and in a network
  • Installing programs
  • Running and using programs such as Word or Excel

Functions of the operating system, the protection program and other programs that may be disruptive are closed down prior to the beginning of the test. This includes both automatic updates and planned actions such as scans or backups. The tests are carried out on a limited number of computers that are identical in construction and have been verified as having the same system speeds in order to give all products the same chances.

Every individual action is carried out at least seven times so that a reliable average can be generated. If the standard deviation of the individual values exceeds a specific threshold, this indicates an error and the test is repeated in full. If the test achieves reliable values for all products and all test cases, these are compared with the reference system values and the difference is calculated. This difference then specifies the slowing down of the system in the case of the actions tested."

 

hxxp://www.av-test.org/en/test-procedures/test-modules/performance/

Edited by SweX
Link to comment
Share on other sites

  • 4 weeks later...

People who judge how light or heavy an AV is based on how much RAM it uses are simply clueless and know nothing about performance. For all I care, the AV can use all the RAM it wants as long as I feel that my system is snappy. that's what matters.

 

I know very light AVs when it comes to RAM usage but they will slow your computer down to a crawl

Link to comment
Share on other sites

Guest
This topic is now closed to further replies.
  • Recently Browsing   0 members

    • No registered users viewing this page.
×
×
  • Create New...