MMx
-
Posts
44 -
Joined
-
Last visited
-
Days Won
3
Posts posted by MMx
-
-
Back to the original topic.
We've had a discussion with Microsoft regarding this. They believe that the memory and CPU usage reported here is adequate to the size of the revocation list that is being processed. There are no plans to implement any changes in this part of Windows unless they are required for security. In their words it's not possible to avoid this behavior except disabling the cache which is not recommended.
I've identified some circumstances that were contributed to this problem. This will be solved in protoscan 1439. Unfortunately the problem might come back anyway since it's considered a normal behavior of Windows, although now it will be less likely.
It's possible to apply this workaround manually. To do that create a DWORD registry value called CryptnetCachedOcspSwitchToCrlCount under HKLM\SOFTWARE\Policies\Microsoft\SystemCertificates\ChainEngine\Config (you may need to create several missing path components) and set it to 1047 (the special meaning of this value is that it will be reverted to default when the product is uninstalled). Then run the following commands elevated and reboot:
certutil.exe -urlcache http://crl3.digicert.com/ssca-sha2-g6.crl delete
certutil.exe -urlcache http://crl4.digicert.com/ssca-sha2-g6.crl delete
This needs to be done for each user separately.It is also possible to completely disable the cache that is causing these problems. Doing it means that verifying certificates after reboot will be as slow as it is the first time they are encountered ever. This is not a recommended solution:
certutil -setreg chain\ChainCacheResyncFiletime @now+10000:0
To revert this use
certutil -delreg chain\ChainCacheResyncFiletime -
On 2/25/2022 at 10:37 PM, itman said:
This method doesn't allow to inject data into the connection. This has several disadvantages:
- No blocking pages (it's much easier to figure out which tab a message is related to if its displayed directly in the tab than in a separate dialog), no redirects for Banking & Payment protection.
- There's no way to implement HTTP2 flow control properly since we would be unable to send any messages: "Flow control is specific to a connection. Both types of flow control are between the endpoints of a single hop and not over the entire end-to-end path." (https://datatracker.ietf.org/doc/html/rfc7540#section-5.2.1) This could have unforeseen consequences.
- There's also no way to send HTTP2 RST message to inform the browser that it shouldn't use any data it has received so far.
-
If an HTTP2 response is completely tranfserred but we haven't finished scanning it yet, we are only able to delay the entire connection, blocking access to other resources which the browser could be parsing in the meantime. This would result in degraded performance. This might be one of the reasons why Avast slows down your browsing more than ESET (see Slowing-down when launching popular websites):
https://www.av-test.org/en/antivirus/home-windows/windows-10/december-2021/eset-internet-security-15.0-211609/
https://www.av-test.org/en/antivirus/home-windows/windows-10/december-2021/avast-free-antivirus-21.9-211603/
On 2/25/2022 at 10:37 PM, itman said:As already mentioned this research is couple of years old. Most of the findings there related to ESET were fixed before the paper was published (we were contacted by the author beforehand), all of them are fixed by now, some were never correct in the first place. Unfortunately the author didn't respond to our request for corrections or updates.
-
1 minute ago, itman said:
As such, what other reasons are there for employing SSL/TLS protocol scanning?
It's the only way to catch exploits targeting browsers before they have a chance to run.
-
Just a quick heads-up, we've been made aware of a malware campaign that hosts malicious payloads as images in imgur.com, so we're removing it from the list of domains for which TLS filtering is skipped. That means it won't behave as described in previous posts anymore.
-
8 minutes ago, Mr_Frog said:
So do you have any explanation so far what is causing the large memory consumption of the eset kernel when opening the browser???
Yes, I've written a detailed analysis here: https://forum.eset.com/topic/31282-memory-usage/page/4/#comment-147108
In a nutshell, this memory is occupied by Windows Crypto API cache that we don't have direct control over. I'm currently trying to figure out if it can be configured beforehand.
-
I see no AV criticism in the linked article or in your post. Also we're not using HTTP Public Key Pinning nor certificate pinning. Can you explain the relevance to the topic here?
-
On 2/18/2022 at 4:37 PM, itman said:
The issue is Eset is also performing cert. revocation checking on non-SSL/TLS filtered web sites and it should not be doing so.
Unfortunately you're wrong here. If someone performs a Man-in-the-Middle attack on your connection to imgur.com it's very likely they will try transferring malware or other content of security interest to the user. In that case we very much want to scan that connection. And the only way to detect that attack is to verify the certificate. We don't want to whitelist anything that's called imgur.com, we only want to whitelist the actual imgur server.
-
16 hours ago, itman said:
First, I posted in this thread a detailed example of certificate chain "patching" in this thread: https://forum.eset.com/topic/31282-memory-usage/?do=findComment&comment=146973 .
I tried to understand your reasoning to the best of my ability and I believe you've made one key mistake.
QuoteI had for a while noticed that Eset memory spiking occurred on web sites using a cert. chained to this Intermediate CA cert., DigiCert SHA2 Secure Server CA.
You're mistaking correlation for causality. Just because two things happen one after the other doesn't mean that the first is causing the second. For example they might be both caused by another factor you've not considered. For a more in-depth explanation please refer for example to https://amplitude.com/blog/causation-correlation
In order to put forth a convincing case you'll need evidence to support your claims. It might look like this:
1. The etl log created as ekrn was allocating the memory shows that about 64MB has been allocated as
- protoscan module was trying to verify a certificate using Windows API, which in turn
- called a function to verify revocation status which in turn
- called a function to download the corresponding Certificate Revocation List which in turn
- allocated the memory as the CRL was being loaded
(another 64MB can be traced to a similar call stack)
2. We know of a CRL that is this big: http://crl3.digicert.com/ssca-sha2-g6.crl Coincidentally this CRL has been issued by "DigiCert SHA2 Secure Server CA", so this might be the source of the correlation you're pointing out.
3. During an investigation of a different ticket we've looked into the allocation and found that it contains an exact copy of the downloaded CRL, so we know that this function is capable of loading the entire file into memory.
Unless you can provide more compelling evidence supporting your cause we'll have to treat claims such as these as your personal beliefs:
Quotethe problem is Eset "chokes" on the processing of intermediate root certs. thereby borking the entire chain validation processing.
I've also noticed several less relevant mistakes in your claims. I can explain those if you're interested, but I'd like to keep the discussion to the point for now so as to not moot the point. Ultimately they are of little consequence because of the wrong assumption you've made at the beginning.
-
8 hours ago, itman said:
It is fairly obvious that Eset, either directly or indirectly, is performing certificate chain validation checking as part of its background processing. It has also been doing this incorrectly for some time.
Unfortunately it is far from obvious for me that there's any problem with certificate validation. Please help me understand how you arrived at this conclusion. Can you post a step-by-step description of a scenarion that lead you to this conclusion and also describe what was the expected result and what was the actual result?
8 hours ago, itman said:The "mystery" at present is why Eset is now reverting to loading the entire aggregate Windows cert. revocation list into memory when this cert. patching occurs.
Please note that the behavior related to loading CRLs into memory you're describing is implemented inside the Windows API function CertGetCertificateChain: https://docs.microsoft.com/en-us/windows/win32/api/wincrypt/nf-wincrypt-certgetcertificatechain I can try to see if it is possible to tweak it based on the information I've requested above, but please note that this is the default behavior of the OS that is also used for example by the Chrome browser: https://source.chromium.org/chromium/chromium/src/+/main:net/cert/cert_verify_proc_win.cc;l=1207?q=CertGetCertificateChain&ss=chromium
-
Thanks everyone for providing the dumps, using them we now have a theory about what is hapenning and a potential fix. It would be helpful if you could test it. To do that first disable all workarounds (like app verifier, enable startup scan) then download the appropriate zip file attached to this post.
If you have a directory called "c:\Program Files\ESET\[product name]\Modules\em005_64" then cleaner_test_dll_64bit is for you. Unpack the file into "c:\Program Files\ESET\[product name]\Modules" (not into the em005_64 subdirectory) with selfdefense disabled and reboot.
If you have a directory called "c:\Program Files\ESET\[product name]\Modules\em005_32" then cleaner_test_dll_32bit is for you. Unpack the file into "c:\Program Files\ESET\[product name]\Modules" (not into the em005_32 subdirectory) with selfdefense disabled and reboot.
If you have a file called "c:\Program Files\ESET\[product name]\em005_32.dat" then you need to use cleaner_test_dat_32bit.zip. Unpack it into "c:\Program Files\ESET\[product name]\" replacing the existing file with selfdefense disabled and reboot.
Then report back if the problem is fixed.
-
I'd like to clarify Marcos' post. You can find the app verifier installer here
32bit: https://drive.google.com/file/d/1c4wQGJteGQb5EurEmhYaYLcmAqUbAIY-/view?usp=sharing
64bit: https://drive.google.com/file/d/1Sh_Yyp7Ie69dbGqBaitN_Nv5iAzuRdwb/view?usp=sharing
Before you are able to use it, you'll have to disable self-defense and reboot. The changes you make will be applied after you click Save in the verifier and restart ekrn by rebooting Windows.
You can skip the manual registry import he's describing by extracting and importing the file attached to this post. Dumps will then be created in c:\dumps.
Edit: There's one more option that needs to be changed in the app verifier. After you've added ekrn.exe you'll need to expand Basics, right-click Heaps, Properties, and enable UseLFHGuardPages (see attached screenshots).
-
Thank you very much RCK for the dumps, they have been helpful. Unfortunately by the time they were created too many things have gone wrong to figure out what was the primary cause and was just a result. It would be helpful if you (or anybody else) could run the following command as admin as soon as possible after boot
procdump -ma -e 1 -n 10 ekrn.exe
Then replicate the problem, and send us all the dumps that will be created. Procdump can be downloaded from https://docs.microsoft.com/en-us/sysinternals/downloads/procdump.
Edit: Please disable selfdefense and reboot before using the procdump command, otherwise it would fail.
-
Also please note that supporting HTTP/2 means implementing full client and server functionality as specified by standards. This isn't directly related to certificates, and changing the TLS handshake to advertise HTTP/2 support is one of the easiest things to implement.
-
That might work in your case, but we try to tune our solutions to work for the majority of around 100 milion of our users. In particular we detect streams and avoid writing them to disk altogether. It's possible that the server in your case is using some less common ways to present the stream to you which our detection doesn't recognize. To investigate further it would help if you could let protocol filtering logging run for a couple of minutes while the temp files are being created:
- Enable F5 -> Tools -> Diagnostics -> Enable Protocol filtering advanced logging
- Make sure the temp files are being created, wait a while
- Disable the logging
- Send me the files "c:\ProgramData\ESET\ESET Internet Security\Diagnostics\*.pcapng"
-
33 minutes ago, zamar27 said:
I'm not sure why you hesitate to define a RAM disk size?
It's user dependant. Any constant I might give you is bound not to be enough for someone.
33 minutes ago, zamar27 said:While Eset may process simultaneously multiple downloads, it segments each download into small parts for processing (if I understand it correctly).
Not really, every htt???.tmp file is a separate download, but not every download creates such file.
33 minutes ago, zamar27 said:is it possible to direct to RAM disk only Eset activity related to web stream processing, but keep it as is all other activity related to various file downloads and web browsing?
Not really, because "The service control manager does not support passing custom environment variables to a service at startup." (from https://msdn.microsoft.com/en-us/library/windows/desktop/ms685990(v=vs.85).aspx). The closest you can get is by redefining the system-wide environment variables TMP and TEMP (in Control Panel -> System -> Advanced system settings -> Advanced -> Environment variables -> System variables). All of the common applications (like browsers) should be using the per-user temp directory, so that won't change.
-
8 minutes ago, zamar27 said:
However, in modern laptops, there is often no way to use SSD as a system drive, and a 2nd internal HDD as a service drive.
I'm not sure I follow. Are you saying that you only have the SSD drive?
10 minutes ago, zamar27 said:My question is, what RAMDisk size would you consider optimal? Will it work OK for Eset?
No one's ever tested that as far as I know, but it should work
Just make sure c:\windows\temp points to a valid place before ekrn.exe starts. Guessing the size is tricky. For protocol filtering you'll want it to fit all of your simultaneous downloads (plus files extracted if there are any archives).
13 minutes ago, zamar27 said:Can you guys look at the Process Monitor log I will send, to suggest what is the bulk of data System Process writes to disk, what activity its related to?
That's fairly easy to do yourself. Filter the procmon log to the System process (make sure to disable the predefined filter "Process name is System then Exclude"), find some big writes and open the event in the tab called Stack. The only way to guess what's going on is to take the function names as hints. Also it's likely that you'll need to download and install Debugging tools for Windows from https://developer.microsoft.com/en-us/windows/hardware/download-windbg, then open Options -> Configure symbols and point it to dbghelp.dll that will be installed (the one that's included with windows doesn't really work).
-
The actual limits are 1MB per file (this is what Marcos mentioned) and 100MB globally.
-
Have you considered moving the temp directory away from SSD? That would solve the problem for all software that might be using it. Assuming D: is your HDD, do the following
- Download the Junction utility: https://technet.microsoft.com/en-us/sysinternals/bb896768.aspx
- Delete the directory c:\windows\temp (go to safe mode if there are locked files)
- Create the directory d:\temp
-
Run
junction c:\windows\temp d:\temp
-
SCR: Is there a chance you need to use an HTTP proxy to connect to the internet?
-
itman: That shouldn't be happening regardless of version. Can you try if you can replicate it using an utility that accesses only a single page (e.g. wget or curl) and make sure you connect to the same IP each time? It is possible that each time you reload a page a new server is connected due to load balancing.
-
There are two reasons ekrn.exe might make connections to servers that are not operated by ESET if you have TLS filtering enabled. First when a browser tries to establish a TLS connection, ESET Security needs to decide if it will filter, block or leave the connection untouched. This decision is in part based on the certificate the server would present if the connection was to proceed, which is not available yet. To solve this problem, ekrn.exe opens a separate connection and requests the certificate, which allows it to make the right decision in the main connection. This certificate is then cached so that the extra connection is not needed later which minimizes performance impact. Doing this is necessary to implement certificate exclusions (e.g. F5 -> Web and email -> SSL/TLS -> Exclude communication with trusted domains, or storing a certificate with Access set to Allow or Block).
Second reason is related to the fact that ESET Security needs to verify the validity of the certificate presented by the server. Online communication is a regular part of such verification (see https://en.wikipedia.org/wiki/Online_Certificate_Status_Protocol). Doing this is necessary to make sure that no attacker hijacked the connection in transit since your browser won't see the original certificate.
Please note that in both cases ekrn.exe sends only what a browser would send if ESET Security was not installed. In a way ekrn.exe acts on behalf of the browser. You can verify this by capturing the communication using wireshark and analyzing it (the initial part of TLS connection that ekrn.exe does in case 1 is not encrypted, and OCSP is not encrypted at all since it relies on digital signatures).
-
It would help us a great deal if you could do the following:
1. Download procdump from https://technet.microsoft.com/en-us/sysinternals/dd996900.aspx and Process Monitor from https://technet.microsoft.com/en-us/sysinternals/processmonitor.aspx
2. Update from prerelease update servers, this should reintroduce the problems, possibly after a reboot
3. When that happens, run
procdump -ma ekrn.exe
4. Start Process Monitor, start monitoring, open some pages that end with error, stop monitoring
5. Send me the ekrn dump and process monitor log
Thanks a lot.
-
Thank you for reporting this problem. Eve launcher was unable to communicate because ESS9 has a default configuration that blocks the obsolete SSLv2 protocol. However it was not our intention to filter the communication of anything else than browsers or email clients in automatic SSL filtering mode, so I've made a change that fixes that. It should be available on prerelease update servers next week, and on release servers possibly a week later.
-
Please start the machine that has the Protocol filtering problem and open this link: hxxp://www.eicar.org/download/eicar.com
A red notification informing about an infection should appear, please post a screenshot of it. Also please note that the link in question is not infected, it is perfectly safe and it is only used to test if an antivirus software is working properly.
Just make sure c:\windows\temp points to a valid place before ekrn.exe starts. Guessing the size is tricky. For protocol filtering you'll want it to fit all of your simultaneous downloads (plus files extracted if there are any archives).
Memory Usage
in ESET Internet Security & ESET Smart Security Premium
Posted
There's a CRL on the same URL with about the same size and a newer expiration date. Do you still experience the same issue?