Error when applying differential update

[Samoréen 8]

Quote

I also now believe what is happening is not of a malicious nature; i.e. MITM activity, but either a local hardware or a data transmission issue. Possible sources are the ISP transmission network, the router, and receiving device memory or storage media. Other possible sources could be issues with power to the device or within the device power supply unit. The issue manifests with Eset downloads primarily since almost all are signed and occur frequently.

I checked all my disks more deeply and they are all healthy, especially my system disk (C: - SSD) and E: (disk where all temp and scratch files are stored - also a [recycled] SSD). Also, it's hard to believe that a hardware or network failure would only affect ESET files. Never had problems when downloading Windows updates (which are also signed), software updates from Adobe and other software vendors  which are signed as well and lots of big image files which are not signed but where any corruption either in the header or in the image part can be immediately spotted. Not talking about other updates like those of Malwarebytes and the regular checking made by my backup software (Acronis). When I say signed, I mean either actually signed or submitted to a CRC check or similar. Also, I'm running a lot a Microsoft .Net software using signed units that are regularly updated.

Also, I had a look at the statistics of my ISP's router and the error ratio is extremely low and when errors occur, they are of the auto-corrected kind. They are fixed before my PC sees them.

OK. Still waiting for the next update. The last one was at 14:34 and it didn't fail.

Edited June 12, 2018 by Samoréen

[Samoréen 8]

New Detection Engine update at 18:50. No error.

[Samoréen 8]

No new error since 06-09-2018 18:24. I'll be on the road for a while. I guess I'll get a lot of updates when I'm back.

[Peter Randziak 1,130]

Hello guys,

@Samoréen 

1. The advanced update engine logging does not use much of the resources and might run for a long time.

2. When it comes to the Processmonitor, it can be filtered with "Drop filtered events" option and than it might be used for hours / days as well. Just set filter to include paths you are interrested in in this case %programfiles%\ESET\ESET Security\Modules and %programfiles%\ESET\ESET Security\Modules %programdata%\ESET\ESET Security\updfiles

3. The most complex part is backing up the contents of the two folders mentioned above :-( 

It's content is changed only after successful update (update.ver downloaded on each update attempt is not important here), but when you do not know, when the error is going to happen, you cannot backup it in advance :-(.

Please keep us posted if the error starts to occur again, we will try to find something out.

@itman yes the fast boot might cause some issues as parts of memory are being reused. We saw that recently on 32 bit versions of Win10 when dll loading sometime fails, and only full reboot resolves it.

Regards, P.R.

[Phil_S 3]

I have just found this topic as the result of a search on error notifications that I have just started receiving from my daughter's laptop:

24/06/2018 15:29:48 - During execution of Update on the computer xxx-LAPTOP, the following event occurred: Error occurred when applying differential update to base file.

 

This has only started today, but so far I have received several notifications by email from her Eset AV. Unfortunately she is halfway around the world from me (hence the time discrepancy between the pasted notification text and my posting time) and not computer savvy, so I have no way of attempting to diagnose or obtain any logs from her machine, but I thought it worth mentioning in any case.

[Samoréen 8]

Thanks Phil,

I'm feeling less alone.

I had the problem again today and as usual, the update succeeded after the second manual attempt. Unfortunately, I was just back to my office and Process monitor was not active.

[Peter Randziak 1,130]

Hello guys,

@Samoréen the logs have been checked and we find following entries in your log files:

"Entry" = "A corruption was discovered in the file system structure on volume E:. The exact nature of the corruption is unknown. The file system structures need to be scanned online. " 11/05/2018 10:07:12 ;

And temp path (used for modules compilation) is redirected to E: drive using environment variables:

TEMP" = "E:\TEMP"  ;
"TMP" = "E:\TEMP"  ;

it seems it might be a cause of previous BSOD as well

"Entry" = "The computer has rebooted from a bugcheck.  The bugcheck was: 0x0000012c (0x00000000001101a3, 0x0000000000000000, 0x0000000000000000, 0x0000000000000000). A dump was saved in: C:\WINDOWS\MEMORY.DMP. Report Id: 3dc2426f-dfb6-462b-96a1-2b105e56c962.
" 11/05/2018 14:02:15 ;
"Entry" = "The system has rebooted without cleanly shutting down first. This error could be caused if the system stopped responding, crashed, or lost power unexpectedly." 11/05/2018 14:01:17 ;
"Entry" = "The previous system shutdown at 3:59:57 PM on ‎5/‎11/‎2018 was unexpected." 11/05/2018 14:01:21 ;

The EXFAT_FILE_SYSTEM bug check has a value of 0x0000012C. This bug check indicates that a problem occurred in the Extended File Allocation Table (exFAT) file system.
 

So it really seems that the issue with module compilation is caused by some issues with your E drive,...

You can either try to get that files system to a correct state or cancel . redirect elsewhere your Temp locations.

Regards, P.R.

[Samoréen 8]

Thanks Peter,

Strange. I'll check again this disk and try to relocate the TEMP/TMP folder. It's a NTFS formatted SSD which doesn't have exFAT, though. I'll let you know if I detect some problem.

Thanks again.

[Samoréen 8]

OK. I have checked the disk and no problem was reported. I have reorganized this SSD so that the whole TEMP folder was moved to another location (partition). That partition is now dedicated to the TEMP folder. Other folders have been left in the E: disk and the pagination file (which was in the same partition as the TEMP folder) also has now its own partition.

We'll see if this changes something. I'll keep you informed.

[itman 1,659]

1 hour ago, Peter Randziak said:

And temp path (used for modules compilation) is redirected to E: drive using environment variables:

TEMP" = "E:\TEMP"  ;
"TMP" = "E:\TEMP"  ;

I also believe that relocating this temp directory, i.e.:

TEMP=C:\Users\xxxxxx\AppData\Local\Temp
TMP=C:\Users\xxxxxx\AppData\Local\Temp

to a non-OS resident drive can cause a multitude of other issues with both OS and non-Eset app software. One obvious one would be permission settings.

Edited June 25, 2018 by itman

[Samoréen 8]

18 hours ago, itman said:

I also believe that relocating this temp directory, i.e.:

TEMP=C:\Users\xxxxxx\AppData\Local\Temp
TMP=C:\Users\xxxxxx\AppData\Local\Temp

to a non-OS resident drive can cause a multitude of other issues with both OS and non-Eset app software. One obvious one would be permission settings.

Hi itman,

You're correct about the permission settings. But if a user is able to manage the location of the TEMP folder, he should also be able to manage the access permissions accordingly.

I know that some applications, instead of using the Windows API in order to retrieve this kind of information, are using hard-coded strings. It's just nonsense . Such applications are written by incompetent developers and should not be used because if these programmers are able to do that, they are likely to make similar mistakes in other programming areas.

As for the OS, I'm not aware of any problem with TEMP folders redirection. I'm doing this since years and never noticed any issue. Trying to move the Users\xxxxxx folders to a non-system partition is another problem, though, and I agree that this should be avoided.

[Samoréen 8]

Peter,

Despite the massive reorganization and re-partitioning of my E: disk, the problem occurred again this afternoon. This time, I could collect all the necessary information. Here is how I proceeded...

These updates always fail the same way :

1. I'm notified of the error.
2. The first manual update attempt always fails.
3. The second manual update attempt always succeeds.

So I collected the requested files before the first attempt, ran it while Process Monitor was running, and then collected again the requested folders and logs after that first manual attempt failed. Then I ran the manual update again (which, as expected, succeeded) and collected again everything. The result is an enormous ZIP file (1,2 GB) that is currently uploading to Dropbox : https://www.dropbox.com/s/q7813a2jh0q5uj3/EAVLogs.zip?dl=0

The file shouldn't be ready for download before this night (19:30-20:30 french time). I'll notify you if it is uploaded faster than expected.

Now a remark : if the problem is actually due to a random failure of my E: disk, I can't understand why it always occurs the same way, as described above. This is too repetitive to be random. From the beginning, the first manual attempt always failed and the second one always succeeded (at least they were reported as such). As you can see from the attached screen capture, the Advanced logging switch generated additional messages. It seems that after the second manual attempt (which I considered as successful) nothing was actually installed : retval = 00005007 [NOT NEED] . Not needed ? So why try to update ? Or was the apparently failed first update attempt actually successful ? I think that there is more about this issue than a mere disk failure (which may actually exist, of course).

Edited June 26, 2018 by Samoréen

[itman 1,659]

I have seen enough to state that Eset's PICO updates are dependent upon files being downloaded to C:\Users\xxxxxx\AppData\Local\Temp directory.

I have a similar situation in IE11 SmartScreen downloads that has plagued me for some time. The temp files it uses are supposed to be written to this directory, C:\Users\Don-Test\AppData\Local\Packages\windows_ie_ac_001\AC\Microsoft\Internet Explorer\UrlBlock, and then deleted after the equivalent .bin version has been created. Instead, the temp files, URLxxxx.tmp, are redirected to C:\Users\xxxxxx\AppData\Local\Temp and never deleted from this directory as they should be. As best as I can determine, this is some type of permission issue with the originally targeted download directory and a "fallback" mechanism is provided internally by either the OS or IE11 to reroute to the C:\Users\xxxxxx\AppData\Local\Temp directory. My best efforts to correct the issue by "fooling around" with permission settings and reinstalling IE11have proved futile.

Edited June 26, 2018 by itman

[Samoréen 8]

5 hours ago, Samoréen said:

The file shouldn't be ready for download before this night (19:30-20:30 french time). I'll notify you if it is uploaded faster than expected.

The archive file is now ready for download.

[Peter Randziak 1,130]

Hello @Samoréen ,

thank you for the log files, I will have them checked.

Having pagefile on non-system partition can cause issues as well, if I recall correctly there is an issue with obtaining the full memory dumps.

P.S. regarding the retval = 00005007 [NOT NEED] 

14:38 -update failed, 14:42 update successful 14:47 retval = 00005007 [NOT NEED] is quite expected as the the modules has been successfully updated 5 minutes ago.

Update starts with update.ver download and processing - it is a metadata file containing info about offered module versions and update files, this is compared to list of modules you currently have. In case the versions of offered modules are same or lower, than your current ones, the updater returns Update not needed,...

Regards, P.R.

Tracking note for us: ESSW-5796

[itman 1,659]

38 minutes ago, Peter Randziak said:

Having pagefile on non-system partition can cause issues as well, if I recall correctly there is an issue with obtaining the full memory dumps.

On this regard, the recommended procedure when moving the page file to another disk is to also create a small page file on the boot drive. Recommended minimum size of the boot drive page file is 300 MB but I would make it 1 GB.

Additional if the device has 8 GB or more of memory, the use of the page file is negligible.

[itman 1,659]

@Samoréenone other thing I am a bit curious about.

You stated you have two drives; a SDD and a HDD. The SDD has the OS loaded and I assume is assigned drive letter C. Noted is the HDD is assigned drive letter E. Missing is any assignment to drive letter D. Does the SDD have two partitions?

The normal drive letter assignment in Windows is in sequential alphabetical order with hard drives assigned prior to optical drives then any remaining storage drives such as memory cards and external drives. Don't know if this is a factor in this issue. 

-EDIT-

Actually, there is an easy way to isolate this issue. Change your TMP and TEMP environment variables back to their default values. If the Eset differential updates issue disappears, the issue can then be positively identified as a problem in Eset's use of same in none default OS file allocation.

 

Edited July 4, 2018 by itman

[Samoréen 8]

7 hours ago, itman said:

@Samoréenone other thing I am a bit curious about.

You stated you have two drives; a SDD and a HDD. The SDD has the OS loaded and I assume is assigned drive letter C. Noted is the HDD is assigned drive letter E. Missing is any assignment to drive letter D. Does the SDD have two partitions?

Hi,

No. I have 2 SSDs. C: is a SSD with a single partition. D: is a HDD and E: is an older SSD dedicated to temporary folders and files and to scratch files for my photo applications. It haf only one partition but it has 3 partitions : one for the pagination file, one for the temporary files and folders and one for the scratch files. I made this to make sure that the temporary files and the pagination file were moved to another physical location.

I'll try to add a small pagination file to C: but moving the temporary files and folders back to C: is a problem because the disk will not be big enough, I'm afraid. I'll try.

[itman 1,659]

I have a new theory on what might be going on. Did you physical delete this directory, C:\Users\xxxxxx\AppData\Local\Temp ?

[Samoréen 8]

1 hour ago, itman said:

I have a new theory on what might be going on. Did you physical delete this directory, C:\Users\xxxxxx\AppData\Local\Temp ?

No. I have one application where this path is hard-coded and that re-creates it when it is deleted.

[Phil_S 3]

Well, my daughter appears not to have had any further reports on her laptop since first experiencing the issue in a flurry over 2 days of attempted Eset updates. She has a standard W10 laptop with a single SSD partition and a default W10 installation, so I find the discussion relating to moved swap files and temp folders a bit of a red herring, at least in her case.

@Samoreen are you still experiencing the issue? It seems to have stopped in my daughter's case, so the circumstances suggest some outside influence to me?

[Samoréen 8]

52 minutes ago, Phil_S said:

@Samoreen are you still experiencing the issue? It seems to have stopped in my daughter's case, so the circumstances suggest some outside influence to me?

Yes. But the problem can disappear during 2 or 3 days (or more) and then show up again.

[Marcos 5,071]

Maybe this is the reason:

"Entry" = "A corruption was discovered in the file system structure on volume E:.
The exact nature of the corruption is unknown.  The file system structures need to be scanned online.
" 11/05/2018 10:07:12 ;

The system temp/tmp variables point to e:\temp.

Replace disk e: with a new 100% working hdd/sdd or at least try pointing the system temp/tmp variables to c:\windows\temp and carry out a full reboot (e.g. by running "shutdown -r -t 0").

Also run a scan with HD Tune Error Scan on drives e:, s: and t:

[itman 1,659]

Also, see if the manufacturer of your E:, SSD drive, has a utility to scan the drive for errors and overall drive status. The manufacturer utilities tend to be the best to use in this situation.

[Samoréen 8]

OK. This SSD is an OCZ one, so I downloaded the corresponding utility which reports that everything is OK. No block removed, spare blocks still at 100% available, health is at 100%, temperature is low, etc. After a full scan, HD Tune has the same opinion about this disk. No error reported, scan window green from first to last block.