AV-Comparatives: Investigation in progress

[TJP 143]

Interesting times in anti-virus testing: hxxp://weblog.av-comparatives.org/?p=501

 

"AV-C has uncovered an infringement of the testing agreement by one of the vendors participating in its tests. It has been found that a product submitted for testing by the vendor had been specifically engineered for the major testing labs, including AV-C".

 

I bet industry insiders know who the company is

[SweX 871]

"specifically engineered for the major testing labs, including AV-C".

 

Specifically engineered to look good in tests, and users that download their product from the vendor website probably thinks what they get is the same product that was tested, but nope.

 

I hope AV-C will exclude this vendor right away and ban them from participating in any future testing by AV-C if this turns out to be true. And that goes for other testing labs that may be affected by this as well. Cheating like this (if true) can't be tolerated. And even if the vendor thought that doing this was "OK" and not against any rule, they should get kicked out. They don't deserve a second chance since they clearly know what they were doing participating with a "special" version made for the tests!

Edited May 3, 2015 by SweX

[sky7 19]

The security vendor is Qihoo 360

 

hxxp://www.av-comparatives.org/weblog/wp-content/uploads/2015/04/VB-AVC-AVT-press-release.pdf

Edited April 30, 2015 by sky7

[SweX 871]

AV-Test:

 

Unfortunately we have to post an update to our current findings. So far we checked the possible manipulation of our protection tests. This is what is being reflected in the posted statement.

We have now started to evaluate the possible manipulation of our performance testing. We have found strong evidence that another company, not Qihoo, is optimizing their product to do well in our performance test by excluding certain files and processes from checking. This is based on filenames and process names and can pose a security risk as well! We will check with AV-Comparatives and VB100 to verify our findings and will let you know as soon as we have the final data.

 

 

https://www.facebook.com/avtestorg/posts/952718894780899

[SweX 871]

The security vendor is Qihoo 360

 

hxxp://www.av-comparatives.org/weblog/wp-content/uploads/2015/04/VB-AVC-AVT-press-release.pdf

Just like 99% expected it would be. Not shocking at all. 

On requesting an explanation from Qihoo 360 for their actions, the firm confirmed that some settings

had been adjusted for testing, including enabling detection of types of files such as keygens and cracked

software, and directing cloud lookups to servers located closer to the test labs. After several requests for

specific information on the use of third‐party engines, it was eventually confirmed that the engine

configuration submitted for testing differed from that available by default to users.

 

I expected the testings labs to exclude them from participating immediately, but no....

Qihoo’s awards and certifications attained since the start of 2015 will thus be stricken from the records

by all three testing bodies, and all three will be imposing stricter demands on test participants to avoid

any further gaming of results by vendors.

 

So the testings labs thinks they deserve a second chance. 

 

Why should they get a second chance when they knew what they were doing.

They should be kicked out and banned from future testing.

Edited May 1, 2015 by SweX

[yongsua 16]

Can't wait to read how the users respond in Wilders.

[LabVIEW707 13]

Funny thing is I warned people especially at Neowin about Qihoo. They have been taken to court about 10x now throughout these past years. But everyone just see's charts and graphs and goes "Oh yeah I so gotta use that. And its FREE". Lol. I tried 360 Internet Security when it came out. I had about 10 false positives and it did not work in a standard account. They do not have a support forum so I emailed them. A week later I got a reply saying "We are working on it". With every new release I once again tried it. Same results and many others with the same problem. Mainly the standard account issue. What kind of security product does not work for all amounts? Lol. So a year later they finally fixed it but still had false positives. Then if you actually took the time to read the AV Compairives PDF and not look at the pretty graphs you would see that the link for Qihoo was to their Chinese version and not 360 Internet/Total Security which you see on Neowin, Majorgeeks and Softpedia. Heck even PCMag highly rated it. Now look at all these fools. Ha. I hope Eset always remains true. 

[SweX 871]

What they say here is nothing else but baloney!

 

hxxp://www.hihuadu.com/2015/05/02/360-withdrew-av-c-reviews-traditional-anti-virus-evaluation-criteria-behind-20812.html

 

Recently foreign anti-virus testing organization AV-C combined with AV-TEST, VB100 claimed that 360 anti-virus evaluation versions of the engine settings and domestic ordinary user version, canceled 360 reviews Awards in 2015. In this regard we believe that the traditional antivirus method AV-C test has been far behind the cloud security of the times, the two sides can not agree on evaluation criteria, 360 decided to withdraw from AV-C reviews,

 

Are they having a laugh or what ? The methodology has not changed since testing began this year, but all of a sudden AV-C testing method is not suitable for their product. Then why did they even choose to participate in the 2015 test series at all if it is not suitable for them!

Edited May 3, 2015 by SweX

[TJP 143]

The entire situation makes a mockery of anti-virus testing.

 

Get caught fudging the tests and it's a slap on the wrist and all is forgiven. AV-C and likely others are paid by the vendors to participate in the tests so I can guess why the cheaters get a second chance.

[LabVIEW707 13]

I love Qihoo's cloud explanation. Obliviously AV-C tests cloud products cause Panda Free scored #1 several times and other times it was in the top 5. And that is a FULL cloud av. Lol

Edited May 5, 2015 by LabVIEW707

[SweX 871]

Tencent is the latest Chinese software company developing an anti-virus product to have been censored by independent testing agencies in less than a week.

 

Now the testing bodies have named Tencent as guilty of gaming tests to falsify its abilities in performance tests. Worse still, according to the testers, the tweaks made for the testers could actually make the product's real-life detection rates *worse*.

 

"After in-depth investigations, certain optimizations have been identified in Tencent products which are clearly designed to improve their ratings in AV-TEST's performance testing. These optimizations, which have been found in all recent public versions of the products, provide minimal benefit to normal users and could even degrade the level of protection offered by the products.

 

All three testing labs involved in these investigations - AV-TEST, AV-Comparatives and Virus Bulletin - expect participants in their tests to behave in an open and ethical manner at all times, and consider this sort of "gaming" of tests to be unhelpful to both developers and users. The labs will be imposing stricter controls on participants to reduce opportunities for such actions, and will revoke all affected certifications and awards granted so far in 2015."

 

https://grahamcluley.com/2015/05/anti-virus-cheating-independent-tests/

Edited May 5, 2015 by SweX

[SweX 871]

Why we cannot tolerate unethical behavior in the anti-malware industry

6 May 2015, San Francisco

 

Cheating goes against everything AMTSO stands for. Our mission is to improve the relevance and accuracy of anti-malware products testing. It is something that benefits all the community:

• product testing drives anti-malware vendors to improve their solutions:
• encourages innovation so vendors can offer better solutions than their competitors
• finds product issues and bugs to be resolved by vendors before products are dispatched to users
• better solutions provides more resilient threat protection for users; and
• better testing provides users with data to make an informed decision about what solutions fit their specific needs in order to get the best protection.

Conversely, bad testing harms us all:

• anti-malware vendors focus on ways to obtain better testing results, rather than researching and developing better protection for users;
• bad testing provides consumers and businesses with misleading information; and
• the reputations of all involved are tarnished.

 

hxxp://www.amtso.org/PR20150506

Edited May 7, 2015 by SweX