steingat 1 Posted December 17, 2021 Posted December 17, 2021 We get this error on a number of our endpoints who are working from home. There are all personal internet connections the we do not control. Is there a way to route this connection though the ESMC server? This never use to be a problem in older versions of eset until they changed the way that this attempt to reach the server.
Administrators Marcos 5,461 Posted December 17, 2021 Administrators Posted December 17, 2021 For a start I'd enable advanced direct cloud logging as well as change the logging severity to Diagnostic. After reproducing the issue (e.g. after several days), disable logging and provide logs collected with ESET Log Collector. Also provide times when the machine was connected to a different network.
steingat 1 Posted December 17, 2021 Author Posted December 17, 2021 Its not just one machine, its several, is there a policy option that can be enabled to re-route this connection through the ESMC Server? Its possible that there are a few machines that change wifi networks or connect/disconnect from the VPN as well. If a policy based option is not available, then I would be able to gather the requested logs from some of the worst offenders, however, like i said, we do not control the network connections of most of these endpoints so attempting to reach out to the ISP or modifying router settings is not a realistic option in most cases.
Administrators Marcos 5,461 Posted December 18, 2021 Administrators Posted December 18, 2021 Endpoint connects either directly or through an http proxy server, if configured. If there are changes in network, we detect them and use the current DNS server for address resolution. If a machine sometimes connects through a proxy and sometimes directly, you may need to enable this setting in the proxy setup: W-S-K 1
kingoftheworld 10 Posted December 31, 2021 Posted December 31, 2021 I think there is something more going on here with the LiveGrid issue that many seem to report. We had a large number of clients (1000+) running EEA 7.4.2041 for about 1 year without any issue. I upgraded everything to 8.1.2037.2 and most seem to be reporting that it is unable to connect to LiveGrid. I checked each of the domains listed on the KB to ensure they are reachable and that correct DNS resolutions are happening and everything was good. Going off of the knowledge shared on the forum of the communication change with 8.1, I went ahead and upgraded one of the clients with the issue to v9. This fixed the issue. Assuming v9 uses the same communication mechanism as v8.1, I lean toward there being something wrong with v8.1 rather than a local network issue. I have clients running v8.1 with the issue and without the issue that are running the exact same image, behind the same firewalls, configured exactly the same.
steingat 1 Posted December 31, 2021 Author Posted December 31, 2021 Ya, What I found is the following: V9 Seems to help with this issue Some of the endpoints I was able to fix by changing their Wifi DNS to Google public DNS Some of the endpoints connect/disconnect from a VPN on a regular basis, These endpoint are still problematic with this error and suspect that the network switching to be the issue. As for logs, I will be enabling logging on a few of the endpoints to narrow this down.
Administrators Marcos 5,461 Posted December 31, 2021 Administrators Posted December 31, 2021 Couldn't it be that the troublesome machines are connected both via wi-fi and wire at the same time?
kingoftheworld 10 Posted December 31, 2021 Posted December 31, 2021 2 minutes ago, Marcos said: Couldn't it be that the troublesome machines are connected both via wi-fi and wire at the same time? Not for me. Nearly all of these machines are desktops with only a wired NIC.
itman 1,806 Posted December 31, 2021 Posted December 31, 2021 (edited) 14 minutes ago, kingoftheworld said: I have clients running v8.1 with the issue and without the issue that are running the exact same image, behind the same firewalls, configured exactly the same. My own opinion is the only way to get to the bottom of this is by using tracert. The ideal test scenerio is on an installation using v8.1 where some connect OK and some do not. "My gut is telling me" Internet backbone relay server/s are the culprit. Edited December 31, 2021 by itman
kingoftheworld 10 Posted December 31, 2021 Posted December 31, 2021 2 minutes ago, itman said: My own opinion is the only way to get to the bottom of this is by using tracert. The ideal test scenerio is on an installation using v8.1 where some connect OK and some do not. "My gut is telling me" Internet backbone relay server/s are the culprit. That is a good possibility. Given that most (or all) of the IPs go to Europe which is a long way for at least me in the US.
steingat 1 Posted December 31, 2021 Author Posted December 31, 2021 4 minutes ago, Marcos said: Couldn't it be that the troublesome machines are connected both via wi-fi and wire at the same time? I have ruled this out as a possibility and verified that the computers are only on one internet connection (wifi), enabled google public DNS on the wifi connection, and disabled IPv6 on the Wifi network controller. At this point, the most problematic machines are ones that connect/disconnect from our VPN while on wifi. The vast majority of our systems that connect to this VPN do not have any issues, its just a small handful of PCs
kingoftheworld 10 Posted January 1, 2022 Posted January 1, 2022 20 hours ago, itman said: My own opinion is the only way to get to the bottom of this is by using tracert. The ideal test scenerio is on an installation using v8.1 where some connect OK and some do not. "My gut is telling me" Internet backbone relay server/s are the culprit. Further testing says your theory may be correct. On some machines (working machines) the routes to the LiveGrid servers leave the US, go through Denmark and onto ESET. On the ones that aren't working, the route appears to leave the US, head through Russia and some other locations but don't appear to make it to ESET. Curious if anyone else is seeing similar.
itman 1,806 Posted January 1, 2022 Posted January 1, 2022 2 hours ago, kingoftheworld said: Further testing says your theory may be correct. Yesterday I did a tracert on a domain Eset uses for its mPico updates, etc., h1-epnsbroker03.eset.com. My ISP is AT&T and I use their DNS servers. Routing in the U.S. stayed pretty much through AT&T servers. Then to servers in Germany, Slovakia, and finally an Eset server. BTW - I never once have had an issue with LiveGrid connectivity.
kingoftheworld 10 Posted January 1, 2022 Posted January 1, 2022 (edited) 34 minutes ago, itman said: Yesterday I did a tracert on a domain Eset uses for its mPico updates, etc., h1-epnsbroker03.eset.com. My ISP is AT&T and I use their DNS servers. Routing in the U.S. stayed pretty much through AT&T servers. Then to servers in Germany, Slovakia, and finally an Eset server. BTW - I never once have had an issue with LiveGrid connectivity. Maybe ESET will bring up some servers in the US to handle some of this communication given their large number of customers in the States. Edited January 1, 2022 by kingoftheworld
itman 1,806 Posted January 1, 2022 Posted January 1, 2022 (edited) Below is actual tracert output to an Eset LiveGrid server. I would say any routing through Russian servers would be suspect: Quote Tracing route to h1-c06-b.eset.com [91.228.166.52] over a maximum of 30 hops: 1 4 ms 4 ms 4 ms homeportal [192.168.1.XXX] 2 7 ms 6 ms 7 ms XXX-XXX-XXX-X.lightspeed.bcvloh.sbcglobal.net [XXX.XXX.XXX.X] =====> AT&T 3 7 ms 7 ms 7 ms 71.151.85.132 ==============> AT&T 4 * * * Request timed out. 5 23 ms 22 ms 22 ms 32.130.17.83 ==============> AT&T 6 25 ms 26 ms 26 ms cl2oh22crs.ip.att.net [12.122.2.126] 7 27 ms 23 ms 20 ms phlpa22crs.ip.att.net [12.122.2.230] 8 28 ms 26 ms 26 ms phlpa21crs.ip.att.net [12.122.3.225] 9 23 ms 20 ms 19 ms n54ny21crs.ip.att.net [12.122.2.149] 10 22 ms 19 ms 21 ms n54ny402igs.ip.att.net [12.122.131.101] 11 20 ms 20 ms 20 ms 192.205.34.182 =========> AT&T 12 140 ms 124 ms 122 ms 217.239.41.158 =========> The IP number is in Germany. It is hosted by Deutsche Telekom AG 13 129 ms 128 ms 128 ms 80.150.170.82 =========> The IP number is in Germany. It is hosted by Deutsche Telekom AG 14 124 ms 124 ms 124 ms st-static-bckb-22.213-81-252.telecom.sk [213.81.252.22] ====> The IP number is in Slovak Republic 15 * * * Request timed out. 16 * * * Request timed out. 17 124 ms 124 ms 124 ms h1-c06-b.eset.com [91.228.166.52] Trace complete. Edited January 1, 2022 by itman
kingoftheworld 10 Posted January 2, 2022 Posted January 2, 2022 17 hours ago, itman said: Below is actual tracert output to an Eset LiveGrid server. I would say any routing through Russian servers would be suspect: Thanks for posting. I am not seeing the same hops through Russia on my test machine that I previously saw. LiveGrid still isn't happy, but I will deal with it on Monday.
LesRMed 26 Posted January 3, 2022 Posted January 3, 2022 On 1/1/2022 at 4:13 PM, itman said: Below is actual tracert output to an Eset LiveGrid server. I would say any routing through Russian servers would be suspect: I get basically the same thing: simEon1111 1
Administrators Marcos 5,461 Posted January 3, 2022 Administrators Posted January 3, 2022 The correct way is to tracert c.eset.com. Do not use a particular server which is located in EU.
kingoftheworld 10 Posted January 3, 2022 Posted January 3, 2022 33 minutes ago, Marcos said: The correct way is to tracert c.eset.com. Do not use a particular server which is located in EU. Thank you. Just ran on one of my impacted machines and it successfully makes it to h1-c04-s.eset.com [91.228.165.44]. I am leaning back toward the problem being with this version as another machine with the issue running 8.1 was upgraded to v9 and the problem goes away.
itman 1,806 Posted January 3, 2022 Posted January 3, 2022 (edited) 43 minutes ago, Marcos said: The correct way is to tracert c.eset.com Parallels my previous posted tracert output with the exception of connection to Slovakia Telecom routing to Eset server: Quote C:\Users\xxxxxx>tracert c.eset.com Tracing route to c.cwip.eset.com [91.228.166.45] over a maximum of 30 hops: 1 4 ms 4 ms 4 ms homeportal [192.168.1.XXX] 2 7 ms 7 ms 7 ms XXX-XXX-XXX-1.lightspeed.bcvloh.sbcglobal.net [XXX.XXX.XXX.1] 3 7 ms 7 ms 8 ms 71.151.85.132 4 * * * Request timed out. 5 29 ms 27 ms 26 ms 32.130.17.83 6 26 ms 25 ms 27 ms cl2oh22crs.ip.att.net [12.122.2.126] 7 26 ms 26 ms 27 ms phlpa22crs.ip.att.net [12.122.2.230] 8 20 ms 21 ms 21 ms phlpa21crs.ip.att.net [12.122.3.225] 9 22 ms 27 ms 24 ms n54ny21crs.ip.att.net [12.122.2.149] 10 22 ms 21 ms 20 ms n54ny402igs.ip.att.net [12.122.131.101] 11 21 ms 20 ms 20 ms 192.205.34.182 12 116 ms 116 ms 116 ms pd9ef315a.dip0.t-ipconnect.de [217.239.49.90] ==> The IP number is in Germany. It is hosted by Deutsche Telekom AG 13 121 ms 121 ms 123 ms 80.150.170.82 ==> The IP number is in Germany. It is hosted by Deutsche Telekom AG 14 * * * Request timed out. 15 * * * Request timed out. 16 * * * Request timed out. 17 117 ms 116 ms 116 ms h1-c01-b.eset.com [91.228.166.45] Trace complete. Edited January 3, 2022 by itman
LesRMed 26 Posted January 3, 2022 Posted January 3, 2022 Mine is similar as well. Tracing route to c.cwip.eset.com [91.228.166.45] over a maximum of 30 hops: 1 3 ms 1 ms <1 ms ip63-247-101-97.Houston.iapc.net [63.247.101.97] 2 13 ms 12 ms 9 ms ip63-247-104-101.Houston.iapc.net [63.247.104.101] 3 23 ms 11 ms 19 ms core1-v2-0.Houston.iapc.net [63.247.97.23] 4 9 ms 12 ms 11 ms 63.247.97.14 5 11 ms 18 ms 9 ms 6-1-5.ear1.Houston1.Level3.net [4.30.110.49] 6 10 ms 12 ms 15 ms ae-5.bear2.Houston1.Level3.net [4.69.216.142] 7 * * * Request timed out. 8 35 ms 35 ms 35 ms 80.150.170.13 9 584 ms 144 ms 138 ms pd9ef315a.dip0.t-ipconnect.de [217.239.49.90] 10 256 ms 136 ms 136 ms 80.150.170.82 11 136 ms 135 ms 139 ms st-static-bckb-112.213-81-254.telecom.sk [213.81.254.112] 12 * * * Request timed out. 13 * * * Request timed out. 14 137 ms 136 ms 136 ms h1-c01-b.eset.com [91.228.166.45] Trace complete.
itman 1,806 Posted January 3, 2022 Posted January 3, 2022 41 minutes ago, LesRMed said: Mine is similar as well. The interesting part of the tracert output is we are both traversing through the same Deutsche Telekom AG server from the U.S.. Given is we are using different ISPs. This leads me to speculate that for those in the U.S. having LiveGrid connectivity issue, the source of the issue lies with their Internet provider or the DNS servers they are using. LesRMed 1
itman 1,806 Posted January 3, 2022 Posted January 3, 2022 Another point to note is the AT&T server used for forwarding traffic to Europe is one shared by Deutsche Telekom AG:
steingat 1 Posted January 3, 2022 Author Posted January 3, 2022 Here are my traceroutes along with some logs. It seems that the connection to 91.228.165.44 is much worst. Endpoint 1 C:\temp>tracert c.eset.com Tracing route to c.cwip.eset.com [38.90.226.12] over a maximum of 30 hops: 1 * * * Request timed out. 2 * * * Request timed out. 3 26 ms 20 ms 29 ms nwblwihed11-lag11-433.network.tds.net [184.61.161.97] 4 30 ms 29 ms 28 ms nwblwidst52-tg0-0-2-0.network.tds.net [64.50.229.61] 5 36 ms 36 ms 30 ms chi-b23-link.ip.twelve99.net [80.239.135.66] 6 * * * Request timed out. 7 32 ms 28 ms 24 ms be2766.ccr42.ord01.atlas.cogentco.com [154.54.46.177] 8 51 ms 45 ms 46 ms be2832.ccr22.mci01.atlas.cogentco.com [154.54.44.169] 9 54 ms 47 ms 46 ms be3036.ccr22.den01.atlas.cogentco.com [154.54.31.89] 10 76 ms 73 ms 68 ms be3047.ccr21.elp01.atlas.cogentco.com [154.54.1.125] 11 67 ms 69 ms 68 ms be2930.ccr32.phx01.atlas.cogentco.com [154.54.42.77] 12 85 ms 80 ms 82 ms be2941.rcr52.san01.atlas.cogentco.com [154.54.41.33] 13 82 ms 82 ms 91 ms te0-0-0-35.nr61.b036483-1.san01.atlas.cogentco.com [154.24.24.186] 14 85 ms 80 ms 84 ms 38.88.58.18 15 84 ms 83 ms 96 ms 38-90-226-12.ptr.eset.com [38.90.226.12] Trace complete. C:\temp>tracert c.eset.com Tracing route to c.cwip.eset.com [91.228.165.44] over a maximum of 30 hops: 1 * * * Request timed out. 2 * * * Request timed out. 3 21 ms 21 ms 21 ms nwblwihed11-lag11-433.network.tds.net [184.61.161.97] 4 31 ms 30 ms 31 ms nwblwidst52-tg0-0-2-0.network.tds.net [64.50.229.61] 5 29 ms 26 ms 30 ms chi-b23-link.ip.twelve99.net [80.239.135.66] 6 42 ms 48 ms 44 ms nyk-bb2-link.ip.twelve99.net [62.115.137.58] 7 120 ms 118 ms 120 ms ldn-bb1-link.ip.twelve99.net [62.115.113.21] 8 * 140 ms * prs-bb1-link.ip.twelve99.net [62.115.135.25] 9 142 ms 140 ms 138 ms ffm-bb1-link.ip.twelve99.net [62.115.123.12] 10 139 ms 146 ms 143 ms win-bb3-link.ip.twelve99.net [62.115.137.203] 11 145 ms 142 ms 141 ms win-b2-link.ip.twelve99.net [62.115.114.185] 12 165 ms 145 ms 143 ms 87.128.239.252 13 141 ms 222 ms 141 ms 80.150.170.82 14 150 ms 146 ms 177 ms st-static-bckb-22.213-81-252.telecom.sk [213.81.252.22] 15 * * * Request timed out. 16 * * * Request timed out. 17 146 ms 148 ms 146 ms h1-c04-s.eset.com [91.228.165.44] Trace complete. Endpoint 2: C:\temp>tracert c.eset.com Tracing route to c.cwip.eset.com [38.90.226.12] over a maximum of 30 hops: 1 * * * Request timed out. 2 * * * Request timed out. 3 41 ms 42 ms 55 ms nwblwihed11-lag11-433.network.tds.net [184.61.161.97] 4 40 ms 41 ms 33 ms nwblwidst52-tg0-0-2-0.network.tds.net [64.50.229.61] 5 35 ms 36 ms 34 ms chi-b23-link.ip.twelve99.net [80.239.135.66] 6 * * * Request timed out. 7 34 ms 34 ms 34 ms be2766.ccr42.ord01.atlas.cogentco.com [154.54.46.177] 8 56 ms 56 ms 60 ms be2832.ccr22.mci01.atlas.cogentco.com [154.54.44.169] 9 57 ms 58 ms 63 ms be3036.ccr22.den01.atlas.cogentco.com [154.54.31.89] 10 84 ms 85 ms 82 ms be3047.ccr21.elp01.atlas.cogentco.com [154.54.1.125] 11 85 ms 79 ms 82 ms be2930.ccr32.phx01.atlas.cogentco.com [154.54.42.77] 12 92 ms 91 ms 91 ms be2941.rcr52.san01.atlas.cogentco.com [154.54.41.33] 13 91 ms 87 ms 93 ms te0-0-0-35.nr61.b036483-1.san01.atlas.cogentco.com [154.24.24.186] 14 110 ms 105 ms 111 ms 38.88.58.18 15 105 ms 88 ms 92 ms 38-90-226-12.ptr.eset.com [38.90.226.12] Trace complete. C:\temp>tracert c.eset.com Tracing route to c.cwip.eset.com [91.228.165.44] over a maximum of 30 hops: 1 * * * Request timed out. 2 * * * Request timed out. 3 40 ms 34 ms 33 ms nwblwihed11-lag11-433.network.tds.net [184.61.161.97] 4 43 ms 41 ms 35 ms nwblwidst52-tg0-0-2-0.network.tds.net [64.50.229.61] 5 43 ms 44 ms 59 ms chi-b23-link.ip.twelve99.net [80.239.135.66] 6 52 ms 60 ms 59 ms nyk-bb2-link.ip.twelve99.net [62.115.137.58] 7 126 ms 132 ms 124 ms ldn-bb1-link.ip.twelve99.net [62.115.113.21] 8 * 150 ms 152 ms prs-bb1-link.ip.twelve99.net [62.115.135.25] 9 149 ms 146 ms 147 ms ffm-bb1-link.ip.twelve99.net [62.115.123.12] 10 149 ms 149 ms 151 ms win-bb3-link.ip.twelve99.net [62.115.137.203] 11 151 ms 148 ms 150 ms win-b2-link.ip.twelve99.net [62.115.114.185] 12 152 ms 146 ms 159 ms 87.128.239.252 13 150 ms 218 ms 153 ms 80.150.170.82 14 152 ms 153 ms 156 ms st-static-bckb-22.213-81-252.telecom.sk [213.81.252.22] 15 * * * Request timed out. 16 * * * Request timed out. 17 156 ms 154 ms 152 ms h1-c04-s.eset.com [91.228.165.44] Trace complete. C:\temp> 2yn0vn2 Joshcollected_eset_logs.zip 399qnq2 collected_eset_logs.zip
Administrators Marcos 5,461 Posted January 3, 2022 Administrators Posted January 3, 2022 It doesn't matter if you are routed to US or EU LG servers since the traffic is very low, typically a few bytes or kB. If that mattered, US users would be constantly getting LG communication errors which is not the case. Most of LG servers are in EU which explains why you are routed there quite often. We'll need advanced logs from time when a LG communication error actually occurs.
Recommended Posts