0xDEADBEEF

It is hard for testers to correlate the testing case with each individual user's pattern. They have to assume a virtual user who faces all possible threats equally (and based of the prevalence of the threat). I see no problem doing this simplification in reality. An antivirus that needs users' frequent intervention is not antivirus, but more of a system control tool. Since detecting malware itself is an undecidable problem, it will be ironic to let this responsibility fall back again to users, who already paid some money for letting experts do the job through their product. A good HIPS should not be per-step popups, and there are plenty of vendors that have implemented more intelligent ones and achieved good results. All I can say is that ESET could have done better Finally, I feel that the statement "the user should blame themselves if their computer got infected" is really weird. Ideally it is antivirus's job to help users distinguish good from bad. If users have to force themselves to behave like an "ordinary" user who never visit suspicious website, I don't think they would bother to pay for the protection. Antivirus gives users more freedom, not shackles. If users got infected, it is their right to question the service they have paid for.

Thanks for the info. I understand that one should always balance between detection and FPs, and sometimes sacrificing detection rate a bit is unavoidable for usability. However, I also see that some products like Bitdefender or Kaspersky achieved good detection rate while also maintaining decent FP rate in AVC (Kaspersky's FP is on par with ESET, and Bitdefender is slightly higher). Doesn't this imply that sacrificing less detection to achieve the similar FP is doable? Especially if the FP testing samples are cherry picked towards the gray zone, it implies their ways of suppressing the FP also work pretty well. They have something that ESET currently doesn't. K has cloud-based HIPS and rollback mechanism. B has a complex weight-based process and inter-process scoring algorithm to deal with post-execution scenario. Although AMS is a similar technique, it still suffers in some cases while other behavior blockers may well handle these corner cases. I know it is much easier to say than actually implementing additional protection layers without introducing more FPs, but I just hope ESET can become better. It has decent HIPS modules, it has a good reputation system. Maybe it is a good idea to exploit more from these infrastructures. oh, but I really love ESET's low perf impact and good compatibility with sandboxes

Personal experience can be helpful, but not all the times. When you do a short-term test drive, you never know if there are some hidden engine problem that might outbreak and lead to disasters sometimes later. For a layman, finding a dealer with good reputation might be a safer option. Similarly, encountering cyber-threats might be rare and the threats are usually in stealth. Trying is definitely necessary, but the loss induced by these attacks might be too much to afford. Relying on trustable reviews with a systematic testing approach is a natural way to help make decisions. We see similar things in testing processor performance: product A could outperform B in metric 1, but could also be beaten in metric B. This is natural and is because many things are just too complex to be measured by a single standard. I don't think security product is an exception. I'd prefer to gradually learn and interpret these results and metrics (which might contradict each other if only looking at numbers) in an objective way and fits my need, instead of simply saying a no to these stuffs.

Personal experience is never representative. As some articles pointed out, no test can directly guide certain user's choice of AV products especially when correlating with his/her usage pattern. I have several experiences of my computer got infected by web trojan when using NOD32 full protection (when it was at ver3.0), but it doesn't mean anything to other people, nor do other individual's personal experience. That's why we need some 3rd party test. Generally it is assuming a single person facing all possible (sampled) threats and obtain the probability of infection. I personally view it as the evaluation in "the worst possible case". Well, ESET indeed does exceptionally well in balancing FPs and detection rate. But when a certified 3rd party test is consistently showing that it ranks relatively low in detection rate compared to other products, there should be some explanation. On tester's side, this could be due to bad samples, biased samples, inappropriate testing methodology, etc. But it might also be due to the real missing pieces on the vendor's side. I personally tend to suspect the sample quality first before I question the security product. But since I can get no more info from their reports (especially for the system performance impact evaluation), I can only post here and hope someone can give a more convincing explanation. My view is: if any test raises an issue, there should be some explanation. It could be the issue with the test itself, it could be the issue with the product, or it could be both. P.S. I've personally played with Microsoft's heur engine, and I know how funnily it performs

I don't either. Actually I care more about what kind of samples did ESET miss every time. Even though it is a very small portion of the whole sample set AVC uses each time, the consistent miss ratio makes me curious about if they are of the same type or not. I personally don't care too much about the AMTSO org results unless they look too bad. But I indeed heard some people posting negative comments about ESET saying "it performs even worse than the free Windows Defender"... So I think some reasonable explanations are good to have

OS might be a factor. But if this is the case, since there is still a decent amount of people using Win7, it does not make sense to provide a compromised protection in one system but not the other. Region (sampling bias) might also be a factor... But if this is the case, it is not a good explanation to north America users.. VB employs quite different testing methodology (# sample is small, only tests static detection, while AVC real-world test seems to test all protection layers, that's why ESET has dynamic detection). I am not familiar about other EU tests, will take a look. As for performance impact score, ESET has very bad score in AV-TEST but very good score in AVC... This is really funny. My experience is that ESET is very light weight (proved by its very low power consumption by looking at energy meter in a windows tablet), but apparently some tests disagree with this.

I have noticed that ESET has been placed in relatively low ranks in AVC's real world tests (from Feb. to Jun.). I am just wondering if this is due to ESET's relatively conservative detection strategy. Of course, the number of samples they use in real world test is pretty low (~400), and many times the detection rate of different products are pretty close (so the # of missed samples is actually very few). I have read David Harley's article about AV tests and understand that sampling bias and many other factors might affect a product's detection results in a test. But several months of similar rankings still make me wondering if ESET missed more than other vendors (of course the FP rate is always very decent compared to others). Another thing I noticed is that ESET seems to be a bit conservative in detecting macro malwares. Is it because ESET prefer to deal with this at later defense level? (like when payload is actually downloaded?) Another mysterious thing is that the performance test in AV-TEST and AV-Comparatives are utterly opposite. Is it because of the difference in their testing strategy?

Cool, thanks. Looking forward to seeing the new detection feature in the future endpoint releases.

Thanks. It means it will be available again sometimes in the future endpoint release? Will it also be available in personal products?

I once saw this option in the ESET Remote Administrator (in an old version). I am currently using ERA 6.5.522 with EES 6.4, but cannot find this option anymore. I remembered the option was here in the policy tab of windows product. Is it an abandoned feature?

I've noticed that some people's endpoint security has an optional high sensitivity heuristic in the threatsense parameter. However, I cannot find this option in my v6.5 endpoint security installation. Is this option only open to some companies or controlled by the administrator?

Thanks!

So Microsoft also has some in-memory detection mechanism? Is there a name for it?

Somewhat expected... BTW, I enjoyed reading the last part of the machine learning discussion from ESET

Really appreciate your response. I remembered ESET has introduced rule-based HIPS since V4, but till now, the HIPS's auto mode still does little in malware's post-execution scenarios (although there is a smart mode and a ransomware protection). Is it due to the concern of FPs so that ESET only leave this function to advanced users? I agree that it is a paradox to claim a product can know the threat before it appears. But some statements you gave seems to be based on the fact that malware makers can always fool the AVs. Will it be the case that a product is so hard to be fooled so that malware creators can hardly/are discouraged to bypass a certain product? For example, in the old days when AVs use traditional signatures (strings, API sequences) to detect threats, it might be intuitive to hide these signatures. But if the signature is less intuitive (some inherent file properties like entropy or something else), it is less intuitive for malware makers to realize what is triggering the detection. Will this ever be the case? A question about possibility to bypass AMS: from the description, it seems to me that AMS statically scans the new executable page. So how about self-modifying code? How about writing an emulator-like execution engine to execute the encrypted code with-in a small memory window so that AMS cannot gather enough features to do scoring? (I am not pro so please bear with me if these questions are dumb)

Seems to be so. I have set a local folder to be visible in the LAN (like "\\server\Runtime"), and added a rule to protect the local path of the folder (like "C:\Users\Username\Runtime"). And apply on all applications. When explorer.exe tries to create a new folder through the local path, the HIPS will prompt the window; but accessing from Network using explorer.exe doesn't prompt any window. Adding the network path to the protection doesn't help

Another question related to the product: when a malware bypasses the scan and detected by AMS, it is already at the execution stage. The executed malware will sometimes have some side-effect on the machine (registry, files, etc.) I have seen some vendors employ rollback mechanism, and some use standard repair procedures. In some cases, wild ransomware might successfully encrypt some files and then be detected through behavior detection, the rollback mechanism of those products will recover encrypted file (currently ESET is not). Is there a reason why ESET doesn't introduce such roll-back mechanism?

Really glad to see such detailed response from ESET. I am with ESET's view that static ML detection alone, which treat executables as data, is not that reliable against malware because it doesn't really look into the things happening under the hood. These methods are somewhat similar to anomaly detection and should not have been deployed to ordinary clients from my view, due to many potential FPs (ESET's low FP is one of the primary reason for me to stick to this product indeed). But does it mean that in order to control FPs, products should be mostly relying on the response speed of the vendor (in that sense there will always be some unfortunate first-time victims). One claim those startups made is their products are capable of detecting unknown threat even with very infrequent update. Of course these statements are hyped, but still makes me wondering if a reliable AV product can somewhat stay ahead of the newly born threats. I knew some vendors have provided sandbox as a mitigation measure for this scenario, has ESET ever considered this issue?

Yes, I mean they are generally good at detecting known threats and their variations. For most Cerber or Spora families, my experiences is that AMS will first kick in if it detects de-cloaked code in memory, and if not, rule-based HIPS will kick in, but at the cost of sacrificing some files. But I never see these two detect new family of ransomware (like if the author of the malware rewrite the core code or change its behavior dramatically, a typical example is the Jaff ransomware recently, AMS and HIPS generally kept silence until more signatures are added some days later). I don't expect ESET or any other security product to detect these new threats at high rate at the initial stage, but it would be good to have some better rule-based HIPS blocking mechanism for post-execution protection when a new threat bypasses AMS or the very conservative HIPS-based ransom protection. I know it is hard to do this well with very few FPs though... but I've seen other vendors use process API call monitoring and auto scoring system to monitor and detect these unknown threats

Well, it is hard to find a vendor which does not use machine learning techniques these days. I am asking for deep learning, but it is fine if ESET does not want to disclose more details about it. Modifying malware to avoid the detection of security products is common, but this cannot explain why those threats are not also tailored for products with similar or larger market share. The cost for these customizations will for sure rise if the protection layers are harder to bypass I saw many improvements in ESET products generation by generation, like the introduction of HIPS, AMS, and exploit blocker. But I feel that the behavior blocking of HIPS is still crude with respect to its rules after many generations. By default, the automatic mode barely does anything unless for modification on key areas. This is not very helpful for many malwares that do not touch these areas. The HIPS-based ransomware protection seems to be very conservative and blocks the threat at the cost of partial encryption of files. AMS is very effective but is restricted to known signatures. BTW, the HIPS rules for Windows Linux Subsystem is still not functioning correctly. Of course cybersecurity protection is a probability game. From the results from those AMTSO organizations with large sample set, ESET does exceptionally well in FPs and performance impact, but still has room for improvement in detection rate.

It is the description ESET made in whitepapers or other public materials make me think in this way. The blog in welivesecurity further implies that ESET is not interested in those deep learning techniques, which only began to be widely adopted in cybersecurity in recent 3 years or so. These are relatively "non-traditional" compared to those well developed ones. I didn't mean these techniques are superior, but am just wondering if ESET has ever adopted these in the detection process. One example is Wannacry. Although during the time it initially outbroke the exploit blocker can already block the SMB exploit and stop the propagation, the malware itself is not detected before the new virus db release (I tested the ESET snapshot right before the Wannacry.D virus db release, none of the protection layer from ESET detected the threat). This implies 1) if the author uses some non-public/zero-day exploit in Wannacry, then ESET can hardly detect this threat in a timely manner. Especially because this ransomware quickly infect LAN computers, the cloud blacklisting (like the ESET "suspicious object" detection) also can't help much 2) the entry point is not protected. That is, if a user initially download the wannacry payload to the computer and execute it instead of being infected through SMB exploit, ESET cannot detect it in a timely manner. As a comparison, some AV vendors can effectively detect and block the threat either at the heuristic stage, or at the behavior block stage (some people tested and verified that some product can even effectively detect and block Wannacry using the virus definition from last December). I am not sure if it is appropriate to mention their names here, but I believe you guys have some idea. I don't mean to criticize anything, but as a ESET user for over 10 years, I start to worry about what if this kind of outbreak happen again.

I've read the tech white paper, but it seems to me what ESET discloses is still close to traditional approaches. The heuristic that pre-execute the malware and do scoring based on the collected behavior has been used by traditional vendors for decades (can't deny that ESET is one of the best). Though I am not sure about adv mem scanner and other techniques, I feel that they are still based on same/similar techniques except for being applied at different stages. On the other hand, some vendors uses static engines to detect malware through their statistics features (like avg entropy or more complex ones), but I've never seen ESET mention these techniques. Although most of these static engines raises many more FPs, but I think it might enhance detection if used in collaboration with traditional approaches. That's why I am wondering if ESET has ever adopted these approaches in the current product. From what I've tested, although heuristic or adv mem scanner can deal with most threats, they are less effective with new families (especially those new ransomware families). HIPS-based anti-ransomware helps very little also, making me worrying about the protection against the explosion of new malware families, like Wannacry

Recent years I've seen many vendors started using new machine learning techniques to enhance their detection rate. e.g. RNN or other neural network variations, as can be seen from the patents they filed. I am wondering if ESET is keeping up with these techniques. From what I've seen in the articles posted in Welivesecurity about ESET's attitude to the machine learning, it seems to me that ESET is rather conservative in adopting these new techniques and a large portion is due to the concern about FPs. I know ESET is one of the vendors that has lowest FP rate (while those aggressive/paranoid designs often suffer from more FPs), but I am also curious if ESET is ever considering or even already has adopted these new techniques to overcome its own limitation currently. Any plans to add protection layers to deal with threats like wannacry that outbreaks so quickly and cannot be easily rescued by cloud blacklisting?