OS X upgrade to v7 causes Product not Activated for EEI connector

[j-gray 35]

As far as OS X v7, I'm having better luck with product activation, but some of them, once activated become deactivated again and log the following:

2023-01-20 13:38:15 0x70000b130000 Info: ESET Inspect integration with Endpoint has been successfully enabled
2023-01-20 13:39:19 0x70000b130000 Error: License check failed. Try 1 out of 5. Failed to process a request to/from ESET Endpoint Security/Antivirus. RUN_LOOP_ERROR RUN_LOOP_TIMEOUT (2)
2023-01-20 13:40:22 0x70000b130000 Error: License check failed. Try 2 out of 5. Failed to process a request to/from ESET Endpoint Security/Antivirus. RUN_LOOP_ERROR RUN_LOOP_TIMEOUT (2)
2023-01-20 13:41:25 0x70000b130000 Error: License check failed. Try 3 out of 5. Failed to process a request to/from ESET Endpoint Security/Antivirus. RUN_LOOP_ERROR RUN_LOOP_TIMEOUT (2)
2023-01-20 13:42:28 0x70000b130000 Error: License check failed. Try 4 out of 5. Failed to process a request to/from ESET Endpoint Security/Antivirus. RUN_LOOP_ERROR RUN_LOOP_TIMEOUT (2)
2023-01-20 13:43:31 0x70000b130000 Error: License check failed. Try 5 out of 5. Failed to process a request to/from ESET Endpoint Security/Antivirus. RUN_LOOP_ERROR RUN_LOOP_TIMEOUT (2)

2023-01-20 13:36:46 0x70000aea1000 Info: Events sent successfully to xxx.xx.xx.x:8093. Server responded with 200 status code in 0s016ms.
2023-01-20 13:38:15 0x70000b130000 Error: License check failed. License no longer active. Failed to process a request to/from ESET Endpoint Security/Antivirus. RUN_LOOP_ERROR RUN_LOOP_TIMEOUT (2)

[Peter Randziak 1,130]

On 1/20/2023 at 7:07 PM, j-gray said:

@Peter Randziak Sorry about that, too many support cases. It should be #00444283. I'm unable to edit my previous post, however.

In our case, we would see the following logged and EP console would show not activated. Note the 1969-12-31 date:

2022-11-29 00:00:32 00d08 Error: License check failed. License no longer active. Request to ESET Endpoint Security/Antivirus failed. Error PERSEUS_E_EI_NO_LICENSE (21801)
2022-11-29 00:00:32 00d08 Info: ESET Inspect integration with Endpoint has been successfully enabled
2022-11-29 00:01:32 00d08 Error: License check failed. Try 1 out of 5. Request to ESET Endpoint Security/Antivirus failed. Error PERSEUS_E_EI_NO_LICENSE (21801)
2022-11-29 00:02:32 00d08 Error: License check failed. Try 2 out of 5. Request to ESET Endpoint Security/Antivirus failed. Error PERSEUS_E_EI_NO_LICENSE (21801)
2022-11-29 00:03:32 00d08 Error: License check failed. Try 3 out of 5. Request to ESET Endpoint Security/Antivirus failed. Error PERSEUS_E_EI_NO_LICENSE (21801)
2022-11-29 00:04:32 00d08 Error: License check failed. Try 4 out of 5. Request to ESET Endpoint Security/Antivirus failed. Error PERSEUS_E_EI_NO_LICENSE (21801)
2022-11-29 00:04:45 00be8 Info: Events Statistics, From:, 1969-12-31 17:00:00, To:, 1969-12-31 17:00:00, Duration (s):, 0, Events Per Second:, 0.000, Events:, 0, File:, 0, Registry:, 0, TcpIp:, 0, Http:, 0, Dns:, 0, Process:, 0, Injections:, 0, Dll:, 0, Traffic:, 0, Info:, 0, Metadata:, 0, Livegrid:, 0, OriginUrl:, 0, Alarms:, 0, UserActivity:, 0, Wmi:, 0, Scripts:, 0, ExeDrops:, 0, OpenProcess:, 0, TrafficSize:, 0, TrafficInterval:, 0, Executions:, 0, Subprocesses:, 0, Connections:, 0, LoadUnloadDriver:, 0, Batch Size (bytes):, 15
2022-11-29 00:04:50 00938 Info: Events sent successfully to server.ip:8093. Server responded with 200 status code in 0s011ms.
2022-11-29 00:05:32 00d08 Error: License check failed. Try 5 out of 5. Request to ESET Endpoint Security/Antivirus failed. Error PERSEUS_E_EI_NO_LICENSE (21801)
2022-11-29 00:06:32 00d08 Error: License check failed. License no longer active. Request to ESET Endpoint Security/Antivirus failed. Error PERSEUS_E_EI_NO_LICENSE (21801)
2022-11-29 00:06:32 00d08 Info: ESET Inspect integration with Endpoint has been successfully enabled

Hello @j-gray,

not an issue, one may get easily confused. Also editing the posts might lead to an confusion so better to have it as a new post.
I checked it with the EI support specialist and he told me that this should be addressed in the hotfix release, which is about to come...
It should address the activation issues, issues with the stats reporting (showing the year 1969/1970), high RAM usage and EI connector crashes.

Peter

[Peter Randziak 1,130]

Hello @j-gray,

On 1/20/2023 at 10:00 PM, j-gray said:

As far as OS X v7, I'm having better luck with product activation, but some of them, once activated become deactivated again and log the following:

So you have the latest EEA for macOS v7 and EI server and connector on the latest versions too? 
The product specialist is trying to reproduce the issue.
With EEA from macOS 6 latest and EI connector 1.9 everything works without any issues.
If it will stay this way, an upgrade to EEA for macOS to v.7 will be performed to see how it behaves on it...

The hotfix mentioned above is on QA now, hopefully it will address most of the issues reported

Peter

[j-gray 35]

5 hours ago, Peter Randziak said:

Hello @j-gray,

So you have the latest EEA for macOS v7 and EI server and connector on the latest versions too? 
The product specialist is trying to reproduce the issue.
With EEA from macOS 6 latest and EI connector 1.9 everything works without any issues.
If it will stay this way, an upgrade to EEA for macOS to v.7 will be performed to see how it behaves on it...

The hotfix mentioned above is on QA now, hopefully it will address most of the issues reported

Peter

Yes, this is specific to OS X with EP v7 and latest EI server/connector v1.9. EI Connector is successfully activated for some time, then reverts to inactivated with the 'RUN_LOOP_ERROR RUN_LOOP_TIMEOUT' errors logged repeatedly.

[Peter Randziak 1,130]

Hello @j-gray,

15 hours ago, j-gray said:

Yes, this is specific to OS X with EP v7 and latest EI server/connector v1.9. EI Connector is successfully activated for some time, then reverts to inactivated with the 'RUN_LOOP_ERROR RUN_LOOP_TIMEOUT' errors logged repeatedly.

It seems that my colleague reproduced it we probably revealed the root cause.
The licenses are being removed on upgrade of the EEAM. So when it is upgraded to the v7, the loop errors start to appear and after the system reboot it starts to report that the EI connector is not activated.
The solution is to reactivate the EI connector so it will store again the license.
The fix needs to be done on the side of endpoint, the colleague will report it to the team responsible to address it.

Peter on behalf of our EI support specialist
 

[avielc 54]

20 minutes ago, Peter Randziak said:

Hello @j-gray,

It seems that my colleague reproduced it we probably revealed the root cause.
The licenses are being removed on upgrade of the EEAM. So when it is upgraded to the v7, the loop errors start to appear and after the system reboot it starts to report that the EI connector is not activated.
The solution is to reactivate the EI connector so it will store again the license.
The fix needs to be done on the side of endpoint, the colleague will report it to the team responsible to address it.

Peter on behalf of our EI support specialist
 

Hi Peter, 
I might be missing something  here in the logic. 
Basically when removing everything, and installing from scratch, EPv10 (Console) reports everything is working right. 
After a while the "EI not activated" status appears. - Assuming this is the same issue with RUNLOOP
The solution you mentioned doesn't seem to be effective at all, as Sending activation task from EP makes the task get stuck \ fail. Even if it succeeds, the machine still has the "EI not activated" status. 

One more thing, (I think I mentioned that before somewhere else) esets_daemon is missing from EEAv7 - which the support couldn't pull --ecp logs from there. 
Could you please provide some KB \ info on how to pull ecp logs from the machine in EEAv7?

Thanks

[j-gray 35]

6 hours ago, avielc said:

Hi Peter, 
I might be missing something  here in the logic. 
Basically when removing everything, and installing from scratch, EPv10 (Console) reports everything is working right. 
After a while the "EI not activated" status appears. - Assuming this is the same issue with RUNLOOP
The solution you mentioned doesn't seem to be effective at all, as Sending activation task from EP makes the task get stuck \ fail. Even if it succeeds, the machine still has the "EI not activated" status.

I'm seeing the same; activation task has been running for 10 hours.

Interestingly, a reboot clears the error/activation state for a number of hours. It looks as if it's good/activated until another license check is triggered, at which point it reverts to the RUN_LOOP_ERROR cycle and becomes inactivated again.

[Peter Randziak 1,130]

Hello @avielc and @j-gray,

It suggests that the activation task failed 😞 ,the EI license check is being performed every few minutes
Before we proceed with further logs, can you please
1. Check if the EI is activated (does not report the not activated status) and sends data to the server OR in the EI console - when it connected and send the events last time.

2. check if you have the license (i.e. files license_322 and license_cfg_322.json) for EI in /Library/Application\ Support/ESET/Security/var/licensed   ?

3. When it starts to report the not activated status, check the license files as mentioned in #2

Thank you, Peter

[j-gray 35]

@Peter Randziak Thanks for your help with this -I greatly appreciate your time and efforts.

1) EI connector shows Inactivated in EP Console. Both EP and EI consoles show connected within the last minute, however, EI Console shows last event over a day ago when it started throwing the RUN_LOOP errors. Just FYI; it does show events sent successfully to the EI server, it's just that there are zero events listed.

2) In the 'licensed' folder I have license_cfg_112.json from the reboot on 01/25 and license_cfg_322.json from yesterday (12:02 a.m.) when the errors began occurring. The files are slightly different (see attachment).

3) The '322' file is the one created when the RUN_LOOP errors began.

[Peter Randziak 1,130]

Hello @j-gray,

On 1/27/2023 at 5:41 PM, j-gray said:

@Peter Randziak Thanks for your help with this -I greatly appreciate your time and efforts.

You are welcome, I'm trying to assist as it is a long polling issue :-(, I must admit that I'm just a proxy here the hanks goes to our EI support specialist, who is doing the real job.

Thank you for trying it out and for the screenshots with SeatID data.
We checked the requests for it server side, but haven't found any meaningful requests in that time frame (after my previous post and before your last post).

Can you please 

1. enable debug EI connector logging > via policy (make sure that the policy was applied to the endpoint in question)

2. sudo launchctl stop com.eset.protection

3. open license_cfg_112.json a license_cfg_322.json a change the logging from false to true

4. kill licensed service > find its PID via ps aux | grep licensed and kill -9 PID

5. sudo launchctl start com.eset.protection

6. reactivate the EI connector

7. wait until the EI connector is reported as not activated

8. Provide us with the content of the ECP folder > /Library/Application\ Support/ESET/Security/var/licensed/ecp

9. collect ESET Log Collector output logs from the mac + EI connector logs

 

Just to confirm you activate the Endpoint and EI by means of the license key i.e. offline license file is NOT used, correct?

 

Thank you,
Peter on behalf of the EI support specialist

[avielc 54]

6 hours ago, Peter Randziak said:

Hello @j-gray,

You are welcome, I'm trying to assist as it is a long polling issue :-(, I must admit that I'm just a proxy here the hanks goes to our EI support specialist, who is doing the real job.

Thank you for trying it out and for the screenshots with SeatID data.
We checked the requests for it server side, but haven't found any meaningful requests in that time frame (after my previous post and before your last post).

Can you please 

1. enable debug EI connector logging > via policy (make sure that the policy was applied to the endpoint in question)

2. sudo launchctl stop com.eset.protection

3. open license_cfg_112.json a license_cfg_322.json a change the logging from false to true

4. kill licensed service > find its PID via ps aux | grep licensed and kill -9 PID

5. sudo launchctl start com.eset.protection

6. reactivate the EI connector

7. wait until the EI connector is reported as not activated

8. Provide us with the content of the ECP folder > /Library/Application\ Support/ESET/Security/var/licensed/ecp

9. collect ESET Log Collector output logs from the mac + EI connector logs

 

Just to confirm you activate the Endpoint and EI by means of the license key i.e. offline license file is NOT used, correct?

 

Thank you,
Peter on behalf of the EI support specialist

Hi Peter, 
I have received a similar request from our local support (assuming that's what they were asked by HQ-support) 
Want me to share some results with you as well? 

[j-gray 35]

Hi @Peter Randziak I'm afraid I didn't have much luck. I followed the steps outlined. After 30+ minutes, the EI Connector does not report as inactivated in the EP Console. However, log files indicate that check 4 of 5 failed, then I see this repeated:

2023-01-31 14:36:13 0x700002f43000 Debug: LicenseStateManager: license state check started
2023-01-31 14:36:21 0x7000031d2000 Debug: Control Checker received HTTP status response (408).
2023-01-31 14:36:31 0x700002819000 Debug: Periodic task FullDiskAccessCheckerTask is starting
2023-01-31 14:36:31 0x700002819000 Debug: Periodic task FullDiskAccessCheckerTask has finished in 0.0 seconds
2023-01-31 14:37:14 0x700002f43000 Debug: LicenseStateManager: license state check started
2023-01-31 14:37:15 0x70000314f000 Debug: Unknown connection id received (28850) for event EVENT_PROCESS_IP_CLOSE
2023-01-31 14:37:31 0x700002819000 Debug: Periodic task FullDiskAccessCheckerTask is starting
2023-01-31 14:37:31 0x700002819000 Debug: Periodic task FullDiskAccessCheckerTask has finished in 0.0 seconds
2023-01-31 14:37:32 0x70000314f000 Debug: Unknown connection id received (28233) for event EVENT_PROCESS_IP_CLOSE
2023-01-31 14:37:51 0x7000031d2000 Debug: Control Checker received HTTP status response (408).
2023-01-31 14:38:14 0x700002f43000 Debug: LicenseStateManager: license state check started
2023-01-31 14:38:14 0x70000314f000 Debug: Removing stale connections: 1/393 removed
2023-01-31 14:38:27 0x70000314f000 Debug: Unknown connection id received (29658) for event EVENT_PROCESS_IP_CLOSE
2023-01-31 14:38:31 0x700002819000 Debug: Periodic task FullDiskAccessCheckerTask is starting
2023-01-31 14:38:31 0x700002819000 Debug: Periodic task FullDiskAccessCheckerTask has finished in 0.0 seconds
2023-01-31 14:38:37 0x70000314f000 Debug: Unknown connection id received (28716) for event EVENT_PROCESS_IP_CLOSE
2023-01-31 14:39:14 0x700002f43000 Debug: LicenseStateManager: license state check started
2023-01-31 14:39:21 0x7000031d2000 Debug: Control Checker received HTTP status response (408).
 

In addition, I do not find an 'ecp' folder under /licensed. Just the two .json files along with two corresponding .lf files.

And to confirm, all products are activated via network, nothing offline.

[j-gray 35]

...in addition; problem with the license files. After changing the logging to 'true', the files are recreated (I assume when the ESET process is restarted) and new file are generated with logging set to 'false'.

In the debug logs, I can see now that the licensing attempts are failing again with the RUN_LOOP errors, but EP console still does not show error/inactivated.

[Peter Randziak 1,130]

11 hours ago, avielc said:

Hi Peter, 
I have received a similar request from our local support (assuming that's what they were asked by HQ-support) 
Want me to share some results with you as well? 

if they were asked by the HQ support I prefer to use that channel so we won't fork it and to prevent checking them twice.
I checked the queue and I found the ticket, which your local support has opened with the HQ support so it should arrive there shortly.

[Peter Randziak 1,130]

Hello @j-gray,

in the part of the log, there are no meaningful errors, the 408 return codes are for "control checker connections", which are supposed to time out after 90 seconds. Those are being written due to the debug logging enabled.

12 hours ago, j-gray said:

...in addition; problem with the license files. After changing the logging to 'true', the files are recreated (I assume when the ESET process is restarted) and new file are generated with logging set to 'false'.

In the debug logs, I can see now that the licensing attempts are failing again with the RUN_LOOP errors, but EP console still does not show error/inactivated.

I checked it with the support guy and the assumes, that the licensed hasn't been killed.
Can you please check if you have killed it as stated at the line 4. "kill licensed service > find its PID via ps aux | grep licensed and kill -9 PID" ? 

It seems that the steps to create the ECP log files are working for @avielc as he was able to obtain them.

Peter

[avielc 54]

1 hour ago, Peter Randziak said:

It seems that the steps to create the ECP log files are working for @avielc as he was able to obtain them.

Hi Peter, 
I didn't say that ECP logs work for me, 
I said that the local support gave me the same instructions as you did, and "when" I'll reach the point where I can diagnose it, I can send it over to you as well if you wish for it.
 

[j-gray 35]

4 hours ago, Peter Randziak said:

I checked it with the support guy and the assumes, that the licensed hasn't been killed.
Can you please check if you have killed it as stated at the line 4. "kill licensed service > find its PID via ps aux | grep licensed and kill -9 PID" ? 

It seems that the steps to create the ECP log files are working for @avielc as he was able to obtain them.

@Peter Randziak I ran through the steps again, paying close attention to step #4; grep returns two licensed processes. I kill both and rerun the grep command to ensure no other licensed processes are spawned. None are returned. As soon as I issue the 'sudo launchctl start com.eset.protection', two new .json files are generated with logging set to false (and of course read/write permissions removed).

And, same as before, the Activation task reports completed successfully within seconds. The log files look the same as above with the "license state check started" (but no reported results) and the same "unknown connection id received" errors.

EP console still does not report an EI activation error state, however.

[Peter Randziak 1,130]

On 2/1/2023 at 2:07 PM, avielc said:

Hi Peter, 
I didn't say that ECP logs work for me, 
I said that the local support gave me the same instructions as you did, and "when" I'll reach the point where I can diagnose it, I can send it over to you as well if you wish for it.
 

Sorry about that, my reading between the lines was apparently incorrect 😞 
Please let us know how it went.

On 2/1/2023 at 5:28 PM, j-gray said:

@Peter Randziak I ran through the steps again, paying close attention to step #4; grep returns two licensed processes. I kill both and rerun the grep command to ensure no other licensed processes are spawned. None are returned. As soon as I issue the 'sudo launchctl start com.eset.protection', two new .json files are generated with logging set to false (and of course read/write permissions removed).

And, same as before, the Activation task reports completed successfully within seconds. The log files look the same as above with the "license state check started" (but no reported results) and the same "unknown connection id received" errors.

EP console still does not report an EI activation error state, however.

Thank you for trying it again and for describing the details. Strange that it works here for us.

Can you please try it once more with the steps described by our dev team?

Quote

• stop product: sudo launchctl stop com.eset.protection
• activate via dummy license: sudo /Applications/ESET...app/Contents/MacOS/lic -k XXXX-XXXX-XXXX-XXXX-XXXX
  • After this, /Library/Application Support/ESET/Security/var/licensed/license_cfg.json file is generated.
• the contents of the license_cfg.json file:

{
"State":0,
"Type":0,
"SeatId":"...",
"SeatName":"focal",
"ERA":false,
"Logging":false
}

• in a text editor, change Logging to 'true' and save the file:

{
"State":0,
"Type":0,
"SeatId":"...",
"SeatName":"focal",
"ERA":false,
"Logging":true
}

• kill licensed service
• start product: sudo launchctl start com.eset.protection
• activate with your real product license
• /Library/Application Support/ESET/Security/var/licensed/ecp folder is generated with ecp logs inside

If it won't work the EI support specialist offered a remote session to check it with you.
Would it be an acceptable for you, so we can move this forward?

Thank you, Peter