Changed LiveGrid Behavior Under Ver. 11.1.42

[Marcos 5,072]

Please refer to http://www.serverframework.com/asynchronousevents/2011/01/time-wait-and-its-design-implications-for-protocols-and-scalable-servers.html.

TIME_WAIT is a normal state after finishing a TCP connection and in this state the system monitors already closed connections for possible further packets. This time is controlled by the value TcpTimedWaitDelay which is set to 4 minutes by default. It is also not true that you would run out of free local ports because Windows takes into account the source and destination IP address and port, not only the port itself when establishing new connections.

As for RpcSS, it's unrelated to the ekrn communication. It's just listed below ESET's connections and that's it, there's no correlation to ESET.

[itman 1,659]

50 minutes ago, Marcos said:

It is also not true that you would run out of free local ports because Windows takes into account the source and destination IP address and port, not only the port itself when establishing new connections.

I was being a bit dramatic ..............

50 minutes ago, Marcos said:

This time is controlled by the value TcpTimedWaitDelay which is set to 4 minutes by default

I haven't altered this parameter on my Win 10 build. As previously posted, the port reallocation activity is occurring in under a minute.

50 minutes ago, Marcos said:

As for RpcSS, it's unrelated to the ekrn communication. It's just listed below ESET's connections and that's it, there's no correlation to ESET.

Do this. Create a new Eset firewall rule copying rule details for the existing default ports 137-139 and 445 outbound rule but restricting it to Trusted Network connections as it should be coded. Enabling logging for the rule. Place the rule prior to the existing like Eset firewall rule and disable that rule.

What is observed is a slew of ekrn.exe outbound connections to ports 137-139 being blocked. This is so because of the new rule being executed prior to the existing ekrn.exe default rule that allows all inbound and outbound communication.

Edited April 4, 2018 by itman

[itman 1,659]

I finally had enough of the way Eset is performing these PICO updates. Since I have tried every way possible to disable them to no avail, I went back to ver. 10.

I also have no intention of upgrading to ver. 11 until Eset changes this.

[Marcos 5,072]

TSM*.eset.com servers have nothing to do with updates, neither with standard updates nor with streamed (pico) updates which are downloaded from update servers as well and not from ThreatSense servers. I will talk to the developer who is in charge of LiveGrid for information about the mechanism of communication with servers and will update you then.

With Endpoint 6.6 that doesn't use the latest technologies, I have 10 connections with TS servers in TIME_WAIT state after an attempt to update.

[itman 1,659]

@Marcos, a few other comments that need to be discussed with the LiveGrid developer.

The first thing I did after installing ver. 11.1.42 and observing these multiple port connections was to disable LiveGrid. Doing so did not stop this behavior. Multiple ports were still being allocated and data being transmitted back and forth. Hence my previous comment that this activity must be related to the pico updating changes.

My primary concern was not the port allocation but the frequency of the connections, approx. ever minute, and the amount of data being transmitted back and forth during each connection, 140K sent and 104K bytes received per Process Explorer recording. Also the consistency of the data transmission; the noted amounts were uncannily the same for each transmission.

Here's my best professional guess what is happening. The current activity LiveGrid performs is very similar to when DHCP cannot establish a connection with one's ISP server. The connection will be repeated multiple times in rapid succession until it times out. Then the same activity will again be repeated a later predetermined interval. Bottom line - there is a problem with data transmission to/from the LiveGrid servers in this new release. This also explains the high data transmission byte count. It's actually 3K approx. data being transmitted each attempt times 30+ attempts equals the counts I previously posted. If my theory proves correct, LiveGrid is actually more insecure in this current release since the frequent update activity is not successfully being performed.

Edited April 5, 2018 by itman

[Marcos 5,072]

7 hours ago, itman said:

The first thing I did after installing ver. 11.1.42 and observing these multiple port connections was to disable LiveGrid. Doing so did not stop this behavior. Multiple ports were still being allocated and data being transmitted back and forth. Hence my previous comment that this activity must be related to the pico updating changes.

He didn't have a clue since with the LiveGrid feedback system is disabled, there should be no communication with TSM*.eset.com servers. Please check if you have some files in the C:\ProgramData\ESET\ESET Security\Charon folder besides cache.ndb.

[itman 1,659]

5 minutes ago, Marcos said:

Please check if you have some files in the C:\ProgramData\ESET\ESET Security\Charon folder besides cache.ndb.

Can't help with that since I fully uninstalled ver. 11.1.42 prior to installing 10.1.235.1. As such, only thing in that folder now is cache.ndb.

8 minutes ago, Marcos said:

He didn't have a clue since with the LiveGrid feedback system is disabled, there should be no communication with TSM*.eset.com servers.

I have enabled diagnostic logging in ver. 10.1.235.1. Is the above the reason I am now seeing the following events? They have been occurring all day when LiveGrid attempts to update.

Time;Module;Event;User
4/5/2018 4:48:36 PM;ESET Kernel;DNS LiveGrid request to c.eset.com ended up with error 19008 (type=0:11, duration=5ms, attempts=1);

"My gut" is telling me that there is a problem with DNS request submission to the LiveGrid servers. Which could be at least a partial explaination for the loop behavior manifesting under ver. 11.1.42.
 

[itman 1,659]

Also I am getting conflicting data from RIPE in regards to c.eset.com. Note the different IP address assignments:

[itman 1,659]

This is what I receive when I ping c.eset.com:

[itman 1,659]

[itman 1,659]

Below is what my dnscache is showing. Something is definite screwed up with LiveGrid DNS resolution:

Edited April 5, 2018 by itman

[itman 1,659]

Believe I resolved the LiveGrid DNS connection issue as far as ver. 10.1.235.4 goes. I state "believe" because I am still monitoring the event log diagnostic entries. So far after fixing the issue, I have not seen any further LiveGrid DNS errors in the event log. Also I visually observed via TCPView, one confirmed download after connection to LiveGrid DNS server.

I tried a number of fixes; reset network connection, flush DNS cache, etc. all to no avail. I finally did a hard reset to default Eset Network Connection settings which appears to have fixed the issue. Of course, I lost all my custom block firewall rules which will have to be manually reentered. This is the second recent occurrence of this; I did the same when I did a hard reset to Network Connection settings when ver. 11.1.42 was installed to no avail and manually reentered all those custom rules only to lose them when I uninstalled it.

What I theorize was the issue with LiveGrid DNS was one or more of Eset's localhost proxy settings got corrupted. I suspect it was one of the default firewall rules; i.e. allow all connections within the computer, etc.. It also could have been the localhost/s connection profile in that 127.0.0.1 was not properly registered under Eset's firewall Local Connections setting. In any case, I am reaching my tolerance point in regards to Eset's use of a localhost proxy. Whereas this is necessary to filter SSL/TLS traffic on ports 80/443 for the browser, Eset has no business using it to connect to LiveGrid by ekrn.exe on port 53 or to use NetBIOS ports 137-139 when NetBIOS is disabled on the IPv4/6 network adapter connection.

What is a major issue is that there currently is no alert provided by Eset to indicate there is a problem with LiveGrid Internet connections. I strongly suspect I have not been receiving LiveGrid updates for some time.

Edited April 6, 2018 by itman

[itman 1,659]

Appears this is the log entry you receive when LiveGrid does actually have an issue connecting to its servers:

Time;Module;Event;User
4/6/2018 10:18:55 AM;ESET Kernel;DNS requests to ESET Live Grid doesn't seem to work in actual network.;

My PC was in the process of entering sleep mode at this time.

Suspect this was the "culprit" in past bogus Eset alerts people were receiving in regards to LiveGrid not being able to communicate. Again, "my gut" tells me those might have been related to network adapter sleep/power settings that will power it down when no activity was received for a certain period of time.

The issue appears to be that Eset firewall can't detect when LiveGrid internal localhost connections are borked. 

Edited April 6, 2018 by itman

[djcz 0]

I had same problem with multiple connections to TSM*.eset.com servers after upgrade to 11.1.42.

Before upgrade my settings were:  LiveGrid enabled , submition of samples disabled + submition of statistics disabled. After upgrade settings were the same in 11.1.42.  
But LiveGrid was trying to send something every single minute. I was one step from unistalling 11.1.42. Then I have checked C:\ProgramData\ESET\ESET Security\Charon folder and there were two files cache.ndb and one small .nfi file. I have deleted nfi file and all connections to TSM*.eset.com servers stopped.
I think that something happened during upgrade process. File was created and LiveGrid was trying to send it to tsm servers with no success. I assume that it wasn't successful because of disabled submission. My LiveGrid event logging was disabled but I checked creation date of nfi file and I think it was trying to send only this one file.

I see another problem after upgrade to 11.1.42. Every 5 minutes Eset is making connetion and exchanging some data with 13.91.57.145 (bal-edf-pcs-app-vmss-02.westus.cloudapp.azure.com). I don't know why, but I haven't seen such connections in previous version of ESET Internet Security.

[itman 1,659]

14 minutes ago, djcz said:

Then I have checked C:\ProgramData\ESET\ESET Security\Charon folder and there were two files cache.ndb and one small .nfi file.

Great find!

15 minutes ago, djcz said:

I see another problem after upgrade to 11.1.42. Every 5 minutes Eset is making connetion and exchanging some data with 13.91.57.145 (bal-edf-pcs-app-vmss-02.westus.cloudapp.azure.com). I don't know why, but I haven't seen such connections in previous version of ESET Internet Security.

I see those same IP refs.. Speculate that Eset is actually now performing cloud scanning using Microsoft's Azure AI servers? In reality, Eset is just hosting files on them depending on the location where you reside.

[itman 1,659]

Dang! Looks like I "jinxed" LiveGrid by posting about it.

The DNS server error is back again. At least this time, I know its not related to my Eset firewall configuration:

Time;Module;Event;User
4/6/2018 11:18:58 AM;ESET Kernel;DNS LiveGrid request to c.eset.com ended up with error 19008 (type=0:11, duration=6ms, attempts=1);

[djcz 0]

36 minutes ago, itman said:

Great find!

Luckily I searched forum before uninstaling and saw your posts so I knew which folder to check. 

Edited April 6, 2018 by djcz

[itman 1,659]

I now have a theory on what is causing LiveGrid DNS connections to fail; as least as far as ver. 10.1.235.4 goes.

After the last failed ekrn DNS event log entry, I rebooted. I then waited for over an hour - LiveGrid is updating hourly on this ver - and have not had any further failed ekrn DNS event log entries. What appears to be occurring is when LiveGrid senses isomething is amok with the network connection, it must internally reset a switch or modify default network rules that will result in any further LiveGrid DNS connections to fail. Why this is happening is up to Eset to find out. It may very well happen in the OS state transition to sleep mode and the later transitioning to the active mode as I posted previously.

Now here is where it gets very weird ............... I was observing network activity during the last failed LiveGrid DNS server request. Immediately after the ekrn DNS request, the Win 10 background task for Windows store started from the suspended state and sent a very small data transmission; a few KBs. No outbound transmission at all. I strongly suspect this was the LiveGrid update. If so it implies that there is some arrangement with MS to use its servers as LiveGrid backup? If this is the case, people need to know about this. What happens if the firewall option to "also use Windows inbound firewall rules" is disabled? Will the Eset firewall allow un-statefull Win store inbound traffic?

Edited April 6, 2018 by itman

[itman 1,659]

Ok. I am done debugging LiveGrid for ver. 10.1.235.4.

Based on recent 11.1.42 LiveGrid comments, decided to give it a try again. What a nightmare installation! In fact, everything so far about 11.1.42 has been nightmarish.

Tried to do an in-program upgrade from  ver. 10.1.235.4. That crapped out with a message that Eset installer could not access Program Data\Eset\Eset Security\Logs\eScan directory. Yeah, no one could since it was marked read only with access by no one! The problem was that when I cancelled Eset installation at that point since it was the only thing allowed, Eset was uninstalled for the most part but Window Defender was not auto activated. In fact, Windows Defender Security Center still showed Eset as the primary AV installed! Anyway got WD activated. Manually deleted eScan directory which interestingly had at that time no permission issues. Then deleted any residual Eset directories and files.

Next, went to the Eset web site and downloaded ver 11.1.42. Then the fun started. After installation, most Eset statuses flagged red, modules started updating by themselves, WD Security Center started going crazy, and for the grand finale Eset started the first time AV scan. Also, Eset indicated that a restart was required. Now I know what forum posters are experiencing in their installation posting woes. Anyway, I said the hell with all this and re-booted. After re-boot, at least everything got back to normal. Perhaps all this install chaos was due to the fact I didn't clean out any residual Eset registry entries?

Then I started reviewing the network connections Eset created for 11.1.42. Two of them; one for my Ethernet connection and one a VPN connection for Wi-Fi for the same network name? Problem is I am on a Ethernet connection. The Wi-Fi connection is not active? Anyway, I  had exported my previous ver. 10.1.235.4 public network setup and imported that. At this point, all is OK.

As far as LiveGrid goes, one connection pops-up every 5 mins. and goes away after about a minute. Great! Well, maybe because I haven't seen any network activity for the connection. But with this localhost proxy baloney, who really knows what is going on ............. I have also seen these tidbits in the Eset event log. Don't know what they are about:

Time;Module;Event;User
4/6/2018 4:39:18 PM;ESET Kernel;Message not handled from untrusted source: ID=010000AC;

Time;Module;Event;User
4/6/2018 4:39:18 PM;ESET Kernel;Problematic Message Source: ID=0001F01D [01000f00];

-EDIT- Guess what? The LiveGrid DNS connection issue is back. Why not ...........

Time;Module;Event;User
4/6/2018 5:19:27 PM;ESET Kernel;DNS LiveGrid request to c.eset.com ended up with error 19008 (type=0:11, duration=5ms, attempts=1);SYSTEM

Edited April 6, 2018 by itman

[cyberhash 188]

@itman
@djcz

The lists of the ip addresses(including the azure & c.eset ones you have mentioned) and an explanation of what they are used for is available at

https://support.eset.com/kb332/

[itman 1,659]

The LiveGrid never ending saga continues .............

After cold boot this morning, LiveGrid connection was not happening every 5 mins. like it did yesterday after installing ver. 11.1.42. Did a repair install. Afterwards, diagnostic event logging showed same details as previously noted; 

ESET Kernel;DNS requests to ESET Live Grid doesn't seem to work in actual network.

ESET Kernel;DNS LiveGrid request to c.eset.com ended up with error 19008 (type=0:11, duration=5ms, attempts=1);SYSTEM

Plus a new one -

Time;Module;Event;User
4/7/2018 10:07:31 AM;ESET Kernel;Update credentials are not trusted by LiveGrid.;SYSTEM

Really .......... enough is enough. Uninstalled ver. 11.1.42. Installed ver. 11.0.159 which did install w/o issue.

What I can now confirm inequitably is that there is a problem with LiveGrid when standby mode is enabled. The same behavior I previously described for ver. 10.1.235.4 manifests on ver. 11.0.159. This can be duplicated by simply setting your PC manually to standby mode. I consider this to be a very serious issue since a user has no way of knowing that LiveGrid is no longer performing updates. The only mitigation is not to allow your PC to enter standby mode or to reboot after it does so.  

[Marcos 5,072]

LiveGrid connections do not occur every 5 minutes, the interval is much longer. Also the product queries LiveGrid servers on a need basis. Normally if a product is activated and updates alright, there's no reason for reporting "Update credentials are not trusted by LiveGrid". This could be theoretically caused by conflicting licenses, however, in that case every attempt to authenticate against LiveGrid servers would fail. Do you mean that in stand-by mode all services are running, network connection is established and communication through all protocols is working? My understanding is that in stand-by mode at least some services enter a power-saving state, otherwise no power would be saved.

[itman 1,659]

10 minutes ago, Marcos said:

Do you mean that in stand-by mode all services are running, network connection is established and communication through all protocols is working?

I am referring to normal Win stand-by mode. In Win 10, a new quick start-up mode was introduced. It only works if your PC supports hibernate mode. Mine does not. So my PC is performing the old Win stand-by processing. In that mode, your PC is powered down. As such, no network connections or anything else for that matter are active.

Based on the Eset event log settings I have posted, LiveGrid appears to know when the network connection is lost as part of the stand-by powering down process. It appears that LiveGrid has a problem with reestablishing its connection once the PC resumes normal operation from standby mode.

[itman 1,659]

@Marcos, check with the developers if the below error is really not an error. For example, it may be generated when the LiveGrid servers are down or there is no update data to transmit.

Or another possibility, my DNS third party provider, Cloudflare, doesn't like the DNS request ekrn.exe is initiating. Don't believe this is the case since all like DNS request would fail which appears not to be the case.

Time;Module;Event;User
4/7/2018 4:42:32 PM;ESET Kernel;DNS LiveGrid request to c.eset.com ended up with error 19008 (type=0:11, duration=5ms, attempts=1);SYSTEM
 

[Marcos 5,072]

If it takes some time to re-establish Internet connection after wakening the system from stand-by mode, the product will try repeatedly to connect to LiveGrid servers several times in certain intervals.

Don't know what error 19008 means, will ask developers.