What data Eset writes to SSD, and how to minimize it?

[zamar27 5]

I've latest regularly updated ESS 10 installed and running on a Win10 64-bit PC. Looking at Process Explorer, it writes to system SSD about 3-4GB/day, but reads back only 1-1.5GB/day. An obvious question is, what exactly ESS writes to disk? Why it reads less than writes? Can such writes be done into RAM instead of SSD to prevent the SSD wear for no valid reason?

To add some info, I've PerfectDisk defrag running in "prevent fragmentation" mode. Also ongoing LAN traffic from security cams, which is not saved to disk by CAM software, but processed in RAM to detect motion. Also web browsers - usually around 500MB/day sessions saved, and internet video streaming via browsers. I wonder, may be Eset Real Time Protection catches the video streaming, which portions are temp saved to browser cache on disk, and then later deleted and replaced by the next video fragments? But why Eset needs to write so much data to SSD on ongoing basis, when no threats are usually identified by disk scans or RTP?

My goal is to completely stop Eset writing to SSD unless a threat is found and needs to be saved locally. How to do that? How to find out exactly, what data Eset keeps writing to disk? If I can't do it on my end, I ask the Eset devs to move all web fragment analysis to RAM instead of constantly writing to be analyzed data to disk, and then replacing it on disk with the next data. I didn't subscribe to Eset to kill my SSD, this is the wrong way to dynamically analyze received from network data.

Edited June 28, 2017 by zamar27

[Marcos 5,259]

Obviously you cannot prevent software from writing to SSD unless you store temporary and user profile folders on a HDD. If larger archives or http streams are scanned, the data is saved to temp files on a disk. We cannot allocate additional hundreds of MB of RAM when needed and we have to work with RAM in an optimal way. Also when update is being performed it requires quite a lot of data to be prepared for compilation of modules which is again something that cannot be accomplished solely in RAM. I guess it wouldn't be a problem to not write to a disk at all if it was common to have dozens of GB of RAM installed on users' systems but this is not something that's gonna happen in the near future.

[cyberhash 194]

In addition to the above, running any sort of de fragmentation on a SSD will reduce its lifespan. Its not required on a SSD.
Could well be another reason why you are seeing such a large amount of disk activity.

[zamar27 5]

3 hours ago, cyberhash said:

In addition to the above, running any sort of de fragmentation on a SSD will reduce its lifespan.

Not sure if you understood me, but PerfectDisk prevents data fragments to be written to disk. It defragments data in RAM before its written to disk. Also, I was not talking about "disk activity" in general, but about Eset only write activity in particular.

[zamar27 5]

6 hours ago, Marcos said:

If larger archives or http streams are scanned, the data is saved to temp files on a disk.

Does that mean, Eset keeps saving to disk local webcam stream portions and constantly analyses them before they reach motion detection software? If that's the case, how I can add these streams to exemptions list for Realtime Protection?

Also, why Eset reads from disk only 30% of what it writes as per Process Monitor stats? Daily virus DB updates are small and don't count into this. It appears, stream writes to disk are mostly done "just in case", while Eset mostly analyzes incoming stream in RAM, so it doesn't need to read back data just written to disk? In this case its either a bug, or you should give a user some control to limit amount of data Eset uselessly writes to disk. With proliferation of SSD as system drive, and HDD production gradually stopped, this method to analyze the stream is becoming quickly obsolete.

Edited June 28, 2017 by zamar27

[Marcos 5,259]

If you use an IP camera to stream video and if it causes a lot of data to be written to a disk,you can exclude its IP address from protocol filtering.

I'd bet that Windows writes much more to the swap file than Eset does.

Please provide a Procmon log (ideally a boot log created as per the instructions linked in my signature) so that we can check what operations were performed.

[zamar27 5]

Hi Marcos,

I'll collect the PM log. These are regular activities I see in monitoring software. Note, System for some reasons regularly accessing Eset stats and local.db - is that Eset itself masking as System? Also, Eset reads back only a small portion of what it writes to disk. I can't believe small Eset logs and stats create 4GB write activity a day. This is the largest single daily write contributor to disk after System as per Process Explorer.

And below you can see, Eset continuously writes to disk an internet TV station stream, not IP camera. That's what namely creates such unnecessary wear workload on SSD, given the fact Eset seldom reads back the temp data it writes to disk. However, I can't find any Eset user tools that easily allow to eliminate certain websites or streams from being continuously real time monitored, or switch Eset to monitor them in RAM, where these stream fragments are already continuously loaded by the browser's player. That means, if TV or movies play in the background all day, especially in HD or 4K format now prevalent, your SSD is going to be weared out very fast by Eset, which doesn't even read this temp data it keeps writing to disk.

So my questions are:

1. How I can exclude certain TV and Movie website streams from Eset Real Time Protection? If not possible, can you add this type of control to Eset User Control Panel?
2. Can Eset process these ongoing stream segments in RAM (given small 2MB temp file size it keeps writing to disk and erasing, and because they are already loaded to RAM by the browser player), instead of continuously writing these stream segments to SSD?
3. Why Firefox and its Flash player don't write the web streams to disk, but instead play them from RAM, which is obvious from Process Explorer stats, but Eset keeps writing them to disk instead of using data already loaded to RAM by the browser?

Edited June 30, 2017 by zamar27

[zamar27 5]

4 hours ago, Marcos said:

If you use an IP camera to stream video and if it causes a lot of data to be written to a disk,you can exclude its IP address from protocol filtering.

But your advice seems to be related to Eset Firewall, but not to Real Time Antivirus Protection? How exactly I can do this, including for certain website urls such as TV streaming websites? Can you give a particular example for Eset Smart Security v10 GUI controls instead of a generic suggestion? The TV & Movie streaming is done via MS Edge browser, and it doesn't seem to write streams to disk, but System writes its cache to disk. System Swap file was not updated for 4 days as per File Explorer.

I sent you the requested Process Monitor log by email (Case #46490).

Edited June 28, 2017 by zamar27

[Marcos 5,259]

I was talking about Web access protection which is part of all ESET security products for Windows.

Some remarks:
- I/O operations also include communication with drivers. That said, the amount of data read/written from/to a disk cannot be determined from Process Explorer and you'd need to use Process Monitor instead.

- Browsers do not need to keep data from streams; they read the data, process it and do not keep it any more or ditch it right away, if not needed. However, antivirus programs need to see the whole content in order to be able to evaluate if it's malicious or not and therefore the data must be temporarily stored which is not the case of browsers.

- We keep 1 MB of data in memory and the rest is saved to a disk. Antivirus programs cannot allocate too much memory in order to to store all data they need to scan.

It's possible to exclude a particular url or IP address from protocol filtering in the Web access protection setup - URL management to prevent the http communication from being scanned.

 

Edit: I've just received your Procmon log from colleagues in the US. The log contains information about operations performed in approx. 2 minutes. During this time, ESET read 4,5 MB from the disk and wrote 0 bytes to the disk which appears ok to me. Reading 4,5 MB in 2 minutes is not excessive. It appears that no media was streaming while the log was being created, otherwise ekrn would have likely created htt*.tmp files in a temp. folder.

[MMx 28]

Have you considered moving the temp directory away from SSD? That would solve the problem for all software that might be using it. Assuming D: is your HDD, do the following

1. Download the Junction utility: https://technet.microsoft.com/en-us/sysinternals/bb896768.aspx
2. Delete the directory c:\windows\temp (go to safe mode if there are locked files)
3. Create the directory d:\temp
4. Run

   junction c:\windows\temp d:\temp

[zamar27 5]

5 hours ago, Marcos said:

Browsers do not need to keep data from streams'

We keep 1 MB of data in memory and the rest is saved to a disk. Antivirus programs cannot allocate too much memory in order to to store all data they need to scan.

Thanks Marcus for your help. Sorry for sending you the wrong log, I took several with different filters. I will try to use Web Access Protection functionality the way you suggested. Never had to modify it before. I'm still concerned to use it with movie streaming sites, since you never know what they may package within the movie stream. But for TV stations its probably OK, though some also insert certain hypnosis & brain manipulation segments into the TV stream, like bright light flashes and certain fast pace non recognizable visual sequences targeting subconscious. But this is not filtered by Eset anyway. (May be a good idea for future development?)

As to saving stream to disk for processing, I prefer MS Edge for stream playback due to highly optimized CPU load and codecs smoothness. It doesn't save stream to disk, but instead it seems to be done by System process, which temp saves segments of the stream to:

C:\Users\User\AppData\Local\Microsoft\Windows\WebCache\V01.loglog
C:\Users\User\AppData\Local\Microsoft\Windows\WebCache\WebCacheV01.dat
C:\Users\User\AppData\Local\Microsoft\Windows\WebCache\WebCacheV01.jfm

Of course Google Chrome and Firefox also temp save stream segments cache in their corresponding directories. I noticed, during stream playback Eset keeps saving and deleting 2MB segments of the stream to C:\Windows\Temp\htt58AD.tmp . It appears redundant to me for 2 reasons:
- Eset can instead use stream cache segments temp saved by the browser;
- why not dedicate 10-20MB RAM space for that task instead of 1MB RAM? Current PCs seldom come with less than 8-16GB RAM, so 10MB is really nothing noticeable, unless using a Pentium PC. May be you can allow a user to set this limit in Eset advanced prefs? I think its very important, given the proliferation of SSDs as primary disks nowadays, and their suffering from ongoing writes.

[MMx 28]

The actual limits are 1MB per file (this is what Marcos mentioned) and 100MB globally.

[Marcos 5,259]

Also please follow MMx's advice to redirect the system temp folder and possibly also user's temp folder to a hard drive or virtual RAM drive as there are many applications that write into these folders.

[zamar27 5]

4 hours ago, MMx said:

Have you considered moving the temp directory away from SSD?

I'm using excellent Link Shell Extension for that purpose, mostly to relocate various browsers Profile & Cache folders from the system drive to pollute it less. However, in modern laptops, there is often no way to use SSD as a system drive, and a 2nd internal HDD as a service drive. So, people just create several SSD volumes for that. But this is not going to address the wear issue due to ongoing writes.

I'm considering RAMDisk software like this one or similar free packages. My question is, what RAMDisk size would you consider optimal? Will it work OK for Eset? It should flash its data to SSD when shutting down the PC. While analyzing Eset writes, Marcus suggested to look at system writes as well. I found that System process writes routinely 15-30GB a day to SSD, which was a shocking discovery. Not sure if its related to Eset though, some of it is Windows Telemetry (spying) activity, some are Event logs, but why such a huge amount of data System continuously writes to disk? Can you guys look at the Process Monitor log I will send, to suggest what is the bulk of data System Process writes to disk, what activity its related to?

Edited June 29, 2017 by zamar27

[MMx 28]

8 minutes ago, zamar27 said:

However, in modern laptops, there is often no way to use SSD as a system drive, and a 2nd internal HDD as a service drive.

I'm not sure I follow. Are you saying that you only have the SSD drive?

10 minutes ago, zamar27 said:

My question is, what RAMDisk size would you consider optimal? Will it work OK for Eset?

No one's ever tested that as far as I know, but it should work Just make sure c:\windows\temp points to a valid place before ekrn.exe starts. Guessing the size is tricky. For protocol filtering you'll want it to fit all of your simultaneous downloads (plus files extracted if there are any archives).

13 minutes ago, zamar27 said:

Can you guys look at the Process Monitor log I will send, to suggest what is the bulk of data System Process writes to disk, what activity its related to?

That's fairly easy to do yourself. Filter the procmon log to the System process (make sure to disable the predefined filter "Process name is System then Exclude"), find some big writes and open the event in the tab called Stack. The only way to guess what's going on is to take the function names as hints. Also it's likely that you'll need to download and install Debugging tools for Windows from https://developer.microsoft.com/en-us/windows/hardware/download-windbg, then open Options -> Configure symbols and point it to dbghelp.dll that will be installed (the one that's included with windows doesn't really work).

[zamar27 5]

Thanks MMx,

Yes, I mean there is usually a mounting space for only one disk inside laptops, and lately these are SSD drives only.

I'm not sure why you hesitate to define a RAM disk size? While Eset may process simultaneously multiple downloads, it segments each download into small parts for processing (if I understand it correctly). Then probably I need to ask a different question: is it possible to direct to RAM disk only Eset activity related to web stream processing, but keep it as is all other activity related to various file downloads and web browsing, because it results in much less frequent and smaller in total writes to SSD?

[MMx 28]

33 minutes ago, zamar27 said:

I'm not sure why you hesitate to define a RAM disk size?

It's user dependant. Any constant I might give you is bound not to be enough for someone.

33 minutes ago, zamar27 said:

While Eset may process simultaneously multiple downloads, it segments each download into small parts for processing (if I understand it correctly).

Not really, every htt???.tmp file is a separate download, but not every download creates such file.

33 minutes ago, zamar27 said:

is it possible to direct to RAM disk only Eset activity related to web stream processing, but keep it as is all other activity related to various file downloads and web browsing?

Not really, because "The service control manager does not support passing custom environment variables to a service at startup." (from https://msdn.microsoft.com/en-us/library/windows/desktop/ms685990(v=vs.85).aspx). The closest you can get is by redefining the system-wide environment variables TMP and TEMP (in Control Panel -> System -> Advanced system settings -> Advanced -> Environment variables -> System variables). All of the common applications (like browsers) should be using the per-user temp directory, so that won't change.

Edited June 29, 2017 by MMx

[zamar27 5]

In this case, since Eset saves stream segments in 2MB files in Temp, why simply not increase Eset 1 file RAM limit to 2-3MB, and that would solve the issue with streams checking completely in RAM. All other downloads are variable in time, they may happen once or two a week, but not on ongoing basis like streams for most users. And browser pages content processing doesn't result in regular ongoing writes to Temp either. Pls consider instead of simply denying it. You have to resolve this major SSD wear issue anyway, since streams playback is prevalent now, and almost completely replaced movie torrent downloads. Namely streams with Eset ongoing writes cause primary SSD wear, because any other web content is sporadic for most users.

Edited June 30, 2017 by zamar27

[MMx 28]

That might work in your case, but we try to tune our solutions to work for the majority of around 100 milion of our users. In particular we detect streams and avoid writing them to disk altogether. It's possible that the server in your case is using some less common ways to present the stream to you which our detection doesn't recognize. To investigate further it would help if you could let protocol filtering logging run for a couple of minutes while the temp files are being created:

1. Enable F5 -> Tools -> Diagnostics -> Enable Protocol filtering advanced logging
2. Make sure the temp files are being created, wait a while
3. Disable the logging
4. Send me the files "c:\ProgramData\ESET\ESET Internet Security\Diagnostics\*.pcapng"