Error when applying differential update

[Samoréen 8]

Hi itman,

I'm using the firewall of my DSL modem. If this is the cause of the problem, updates should fail all the time. Which is not the case.

[itman 1,659]

31 minutes ago, Samoréen said:

Hi itman,

I'm using the firewall of my DSL modem. If this is the cause of the problem, updates should fail all the time. Which is not the case.

Referring to @Marcos previously comments, I would temporarily switch your DNS server IP address from your ISP's one to a third party one such as Google, Verisign, Norton, etc.. If this resolves the Eset module update certificate errors, it is prove that Eset's module update downloads are somehow being hijacked via your ISP servers or backbone network servers it connects to.

-EDIT- Also prior to doing the above, open a command prompt window. Then enter:

ipconfig /flushdns

It is possible you have corrupted DNS cache entries for Eset download servers.

Edited June 9, 2018 by itman

[Samoréen 8]

I can test this but again, this would not explain what's happening with the "Check for updates..." button. When an update failure occurs, if I click once on this button, the update fails again. If I click a second time or a third time, the update succeeds.

I don't see how the DNS Server could intervene in this process.  The DNS Server is just here to translate a domain name into an IP address. Unless I have missed something, the DNS Server doesn't determine the route. Maybe I should update my knowledge about TCP/IP.

[Samoréen 8]

And, as a reminder, I never had such problems before version 11.1.54 was installed.

[itman 1,659]

23 minutes ago, Samoréen said:

And, as a reminder, I never had such problems before version 11.1.54 was installed.

Did you install 11.1.54 from scratch? You might want to run Eset's UnInstaller to clear out all remnants of previous Eset installations. Then install 11.1.54 and see if the issue persists.

[Samoréen 8]

I flushed the DNS cache and just a few minute ago a new update took place. Same error again. I clicked on "Check for updates...", same error. I clicked again, update succedeed.

 

[Samoréen 8]

2 minutes ago, itman said:

Did you install 11.1.54 from scratch? You might want to run Eset's UnInstaller to clear out all remnants of previous Eset installations. Then install 11.1.54 and see if the issue persists.

I can't remember how many times I have uninstalled and re-installed ESET Nod32 when having problems because this is what I had been told to do. This never helped. Now, if a product has a repeating bug, I prefer that the bug be fixed instead of masking it by restarting from scratch. I have sent all the requested logs and information one month ago. Waiting for the feedback...

[itman 1,659]

Since you are using your router's firewall as your only network protection, you might want to check out this article in regards to if your router is one of the affected products: https://www.bleepingcomputer.com/news/security/vpnfilter-can-also-infect-asus-d-link-huawei-ubiquiti-upvel-and-zte-devices/ . Of note:

Quote

ssler - plugin for intercepting and modifying web traffic on port 80 via man-in-the-middle attacks. Plugin also supports downgrading HTTPS to HTTP.

 

[itman 1,659]

If your router is one of the listed affected ones, here are instructions on how to reset the router which is the only sure method to insure you are not infected: https://www.bleepingcomputer.com/news/security/reboot-your-router-to-remove-vpnfilter-why-its-not-enough/ .

[Samoréen 8]

3 hours ago, itman said:

If your router is one of the listed affected ones...

Thanks. My router is specific to my ISP. It doesn't belong to the affected routers.

[itman 1,659]

At this point, all I can say is something is intercepting your Eset module updates; the invalid certificate signature Eset log messages attest to that. Whereas an external MITM can do that, it is rare. One other possibility is malware has established a hidden localhost proxy on your PC. To tamper with signed downloads, it would also have to install a Windows root CA certificate to do so. If this is the case, it is also intercepting other HTTPS traffic other than Eset module updates.

[Samoréen 8]

Quote

One other possibility is malware has established a hidden localhost proxy on your PC

If this really happened, then both ESET Nod32 and Malwarebytes missed it. So should I drop these products and look for something more effective ? Moreover, the invalid certificate warning occurred only once (could be a mere transmission error). All other update failures since the problem appeared were compiler errors.

[itman 1,659]

1. 5 hours ago, Samoréen said:

If this really happened, then both ESET Nod32 and Malwarebytes missed it. So should I drop these products and look for something more effective ? Moreover, the invalid certificate warning occurred only once (could be a mere transmission error). All other update failures since the problem appeared were compiler errors.

According to your prior screen shots, you received Eset module certificate errors on 5/26, 5/29, and 6/9. So it has happened more than once.

The Windows firewall does not monitor localhost connections; i.e. to/from 127.0.0.x. 

I would contact your in-country Eset Customer Service. You need someone to take an in depth look at your system.

[Samoréen 8]

48 minutes ago, itman said:

According to your prior screen shots, you received Eset module certificate errors on 5/26, 5/29, and 6/9. So it has happened more than once.

Correct. I forgot these.

Quote

I would contact your in-country Eset Customer Service. You need someone to take an in depth look at your system.

Not an option. My system contains sensitive data. However, I'm a former software/system engineer, so I can certainly understand directions from Eset in order to detect any anomaly. Just waiting for them. As mentioned above, I provided all the requested information one month ago.

I certainly value your comments but :

1. I'm able to monitor what's happening on my network and I couldn't find anything unusual or suspect.

2. My router has been fully reinitialized recently. I checked the settings. Nothing wrong. There's something I can check, though. I have the possibility to bypass my ISP's router and be connected to Internet via a 4G device. I can live with such a connection one or two days. Let's see if that changes anything...

3. If some malware is at work somewhere on my system, why should it target only ESET Nod32 ? No other problem during the past month.

4. Why does the update always eventually succeed when I use the Check for updates button ? If the updated modules are tampered with only from time to time and are always eventually installed, what's the actual purpose of this malware ? Too lazy to be effective .

5. Why did this problem only appear just after version 11.1.54 was installed ?

6 Why do I get compiler errors anyway even if there's no digital signature problem ?

[itman 1,659]

5 hours ago, Samoréen said:

5. Why did this problem only appear just after version 11.1.54 was installed ?

6 Why do I get compiler errors anyway even if there's no digital signature problem ?

Since you won't let anyone remotely examine your system, do as @Marcos previously suggested:

Quote

You can capture such communication with Wireshark so that we can check what modifications were made to the update files.

As least this will resolve if external MITM is occurring.

[Samoréen 8]

12 hours ago, itman said:

You can capture such communication with Wireshark so that we can check what modifications were made to the update files.

As least this will resolve if external MITM is occurring.

OK, thanks. But this assumes that I'm able to anticipate when an update will take place. Otherwise, I'll have to keep Whireshark active all the time. This will generate huge logs, I guess.

[itman 1,659]

I am going to take a different approach to these certificate errors.

Eset modules are coded in assembler language. Updates are downloaded and compiled on your device. Assumed is after successful compilation, the modules are then code signed using Eset's SSL Windows trusted root CA store certificate. This certificate is authorized to perform code signing.

Referring to your prior log screen shots, it is observed that in every instance where a certificate error occurred, it was preceded by a module compilation error. So the question to be resolved is if the certificate error was caused by the module compilation error, or was the cause of module compilation error the certificate error? Appears to me, the former is the case. It is reasonable to assume that the Eset module code update downloads are unsigned and are only locally code signed after successful compilation. It is then copied to the Eset module sub-directory in the C:\Program Files directory.

Since not all module updates on your PC are failing and the issue appears to be intermittent, the best conclusion I can arrive at is something is periodically corrupting your Eset module downloads. Since you state you are not having other download corruption issues, at least that you are aware of, this leads to the conclusion that there is an issue with your PC connecting to the Eset servers hosting the module updates and the connection issues are intermittent. How and why this occurring I frankly have no clue since I have never heard of a similar like issue. If this was due to man-in-the-middle activity, I would expect the activity to be constant and affect other download activity besides one particular type of Eset download activity. It appears to be some type of network connection "glitch" at this point.

Edited June 11, 2018 by itman

[Samoréen 8]

Interesting comments.

So I should now be looking for intermittent and undetected problems on the network OR on my system disk (I just checked it, no apparent problem). After all, if your assumptions are correct, this could also be a file corruption occurring when the source files are written, not necessarily during the download. I guess that these assembly source files are not maintained on my system ? Otherwise I could read them and at least be able to determine if they are actually corrupted.

Re : module updates
The Event log reads as follows : update module - Compiler error...

Actually, if I understand well, we are not talking about "updating modules" but about the "module that is in charge of running updates". It's always that particular code that fails. The updated module appears to be the "Detection Engine" module. If I look at the component list, I can see that module updates are not that frequent. The "Rapid Response" module was updated today, the "Network Protection" module was updated on 06-08-2018 and other modules were updated in May. Otherwise, the updates occurring almost daily are the Detection Engine updates. When I re-launch a failed update manually, it's always a Detection Engine update that is installed and listed in the Event log. So only that module would be corrupted ?

By the way, this problem is not related to 11.1.54. I just discovered that it also occurred with 11.1.42 and 11.0.159.9, although not that frequently.

 

[itman 1,659]

Verify that the following services are running and set to their default startup values:

• WinHttpAutoProxySvc - manual
• iphlpsvc - automatic

Also that you haven't disabled IPv6 IP-HTTPS tunneling if you are using IPv4 or both IPv4 and IPv6 preferring IPv4 over IPv6.

[Samoréen 8]

Everything OK.

[itman 1,659]

1 hour ago, Samoréen said:

By the way, this problem is not related to 11.1.54. I just discovered that it also occurred with 11.1.42 and 11.0.159.9, although not that frequently.

Since the problem is intermittent is also possible that the download is hitting "bad areas" on your HDD. The detection engine module is probably one of the most frequently updated modules.

[itman 1,659]

Also one thing we might all have missed to date is given in the error message:

Error occurred when applying differential update to base file

I strongly suspect this is related to the new Pico updating processing Eset implemented a while back. These are streamed on demand to Eset installations and are stored in this directory:

One possibility is your router's firewall is somehow blocking these or interfering with the download of them. I would take a look at your router's firewall log for block connections and the like relating to these updates.

[Peter Randziak 1,130]

Hello guys,

@Samoréen I'm sorry for my delayed response, I'm somehow not able to answer anything I should in a timely manner, in case I won't respond you in this case in 2 business days, feel free to send me a private message with a reference to this thread as a wake up call for me :-)

The logs I downloaded weeks ago are not consistent i.e. not from the same failed update attempt.

As it seems you are able to reproduce it nearly on demand please do following:

1. back up the modules folder and updfiles by default located in C:\Program Files\ESET\ESET Security\Modules and C:\ProgramData\ESET\ESET Security\updfiles 

2. enable the Update engine advanced logging (ESET main GUI -> Advanced settings (F5)-> Tools -> Diagnostics )

3. Reproduce the issue 

4. stop the Update engine advanced logging - this will flush the log to the updater_$time_spamp.etl files to Diagnostics folder (by default C:\ProgramData\ESET\ESET Security\Diagnostics)

5. pack the contents of the two folders in step 1 and the updater_$time_spamp.etl and send it to me

note: they all need to be consistent i.e. from the same update attempt 

In case you would be able to capture the update event with Process monitor, it would be a great addition for us to be able to analyze it properly.

You can also send the files only to me as the logs might contain data, you would not like to share publicly,..

 

@itman the pico updates does not use the standard update mechanism, that is quite complex.

The update files are a binary difference of the actual module from the one you are about to update, they are signed to ensure the code integrity.

The compilation happens in memory it takes the old data, appends new one, sort in case the module is a database like, appends the signature, writes it to disk, loads the new module, yes a bit of a magic :-)    

The new compiled module is digitally signed by us as well - this digital signature can be easily verified as it is the standard one for signing .dll files.

P.S. most parts of the modules are coded in C++

Regards, P.R.

[Samoréen 8]

Peter,

I cannot reproduce at will since this problem randomly occurs when the Detection Engine is updated. How could I know when these updates occur and which one will fail ? This means that I should leave Advanced logging enabled all the time. OK but as I already mentioned in one of my messages in answer to the same recommendation made by Marcos,

Please note that Advanced logging disabled itself multiple times after I enabled it. It also disabled itself just after the last module update.

Quote

In case you would be able to capture the update event with Process monitor, it would be a great addition for us to be able to analyze it properly.

Which process ? Should I capture all events for all processes ? For the reasons explained above, this will generate a huge log until the problem occurs again.

[itman 1,659]

4 hours ago, Samoréen said:

I cannot reproduce at will since this problem randomly occurs when the Detection Engine is updated. How could I know when these updates occur and which one will fail ?

This again leads me to believe that some type of file corruption is occurring; either to the download in transit or within the receiving device at the network or file storage level. As little as a one byte change will alter a downloaded file hash value. Further credence is given to this theory in that its occurrence is of a random and intermittent nature. I also now believe what is happening is not of a malicious nature; i.e. MITM activity, but either a local hardware or a data transmission issue. Possible sources are the ISP transmission network, the router, and receiving device memory or storage media. Other possible sources could be issues with power to the device or within the device power supply unit. The issue manifests with Eset downloads primarily since almost all are signed and occur frequently.

Reviewing like Eset update download issues, they had one common characteristic. That is, they were of a consistent nature and could be easily duplicated. 

-EDIT- An example of the above is I was having serious and intermittent Windows issues after upgrading to Win 10 1803. After a bit of research, I traced the problem to Win's "Fast Boot" feature. I disable that and have not had a problem since then.

Edited June 12, 2018 by itman