ESET Endpoint Security for Mac OS X - Startup Issue

[kingoftheworld 7]

On 1/12/2017 at 2:31 PM, itghelp said:

Hi all,

I've been following this thread for a long time now as I've experienced the same issues (startup hanging with ESET being the culprit) on a variety of macOS machines since about August. I did open a ticket with ESET support back then but they just kept asking me to send more logs and we weren't getting anywhere so I gave up with them. 

One thing we have done recently which has seemed to stop the startup problems is to remove ESET from "Login Items" and then write a script that delays ESET to launch 2 minutes after login. I am not the one who wrote the script but I would be happy to share details with you.

It's not a perfect solution, but in the meantime it seems to work well.

Hope that helps!

We also have been experimenting with a similar solution of changing the egui path in the PLIST file so that it will automatically fail and not attempt to launch. All of the core functions of the software continue to function including communication with ERA. I think we can safely say that it is related to the GUI.  We have provided numerous sets of logs, but can anyone confirm if they have received a resolution from support?

 

[jrioux 2]

1 hour ago, kingoftheworld said:

We also have been experimenting with a similar solution of changing the egui path in the PLIST file so that it will automatically fail and not attempt to launch. All of the core functions of the software continue to function including communication with ERA. I think we can safely say that it is related to the GUI.  We have provided numerous sets of logs, but can anyone confirm if they have received a resolution from support?

 

Still nothing from my end. I will try and poke the support again.

[j-gray 13]

I last heard from support on Jan-12. They said there was a new revision but it hadn't been approved for testing, yet. Still waiting...

[kingoftheworld 7]

Any ESET employees able to give any insight to this issue? @Peter Randziak @MichalJ  Are we close to a fix on this?

[Peter Randziak 557]

Hello guys,

we are trying to reproduce the issue in-house to be able to debug it and verify the fix.

So what would help us the most, would be the exact steps (step by step) how to reproduce the issue.

One of my colleagues from the HQ tech support spend few days with it, but the issue happens only rarely and on unknown circumstances so that was not much helpful :-( 

Regards, P.R.

[kingoftheworld 7]

5 hours ago, Peter Randziak said:

Hello guys,

we are trying to reproduce the issue in-house to be able to debug it and verify the fix.

So what would help us the most, would be the exact steps (step by step) how to reproduce the issue.

One of my colleagues from the HQ tech support spend few days with it, but the issue happens only rarely and on unknown circumstances so that was not much helpful :-( 

Regards, P.R.

We have been able to replicate it with a vanilla install of OSX Sierra and El Capitan joined to a Windows AD environment using the built in Directory Utility.  Then the only thing that is needed to replicate it is an install of ESET Endpoint Antivirus and log into the machine using a domain account.  We are seeing it happen nearly 100% of the time.

[j-gray 13]

On 2/6/2017 at 6:37 AM, kingoftheworld said:

We have been able to replicate it with a vanilla install of OSX Sierra and El Capitan joined to a Windows AD environment using the built in Directory Utility.  Then the only thing that is needed to replicate it is an install of ESET Endpoint Antivirus and log into the machine using a domain account.  We are seeing it happen nearly 100% of the time.

Pretty much the same here. We've provided exact steps and collected and sent numerous log files for analysis, as well. It seems that a number of other customers with the same issue(s) have been providing the same.

Given that this issue was reported 8 months ago (our case has been open for 6 months now) what else can we do to work towards a resolution?

[Peter Randziak 557]

Hello guys,

can you please if it happens with this version?

http://ftp.nod.sk/~mego/RC/business/6.4.161/

Please let us know how it went.

Thank you, P.R.

[plex 2]

We have been having this same issue.  We have 700+ Macs and many have complained about the system unresponsiveness at login using ESET Endpoint Security 6.3.85.  All are joined to our Active Directory (2012R2 schema).  We recently deployed ESET throughout the company and only finished recently.  As we deployed to more users, the support calls increased.

After testing, we are fairly confident every user has this problem, but not all have it to the same degree.  The range is from noticeable only if you know what to look for up, all the way to 45 minutes until the desktop was usable after login.

In order to see what might be happening, we configured a freshly imaged Mac set to have Activity Monitor load at startup by placing it in the Login Items.  Activity Monitor is configure to view All Processes, the Update Frequency is set to Very often, and % CPU sorted to show the highest processes at the top of the list.

After login, we noticed that opendirectoryd had very high CPU usage during login.  Typically, Activity Monitor would not respond during the unresponsive period and as soon as the freezing stopped, the graph and data would just flood in all of the data from during the freezing period.  If watched closely, we noticed opendirectoryd would be at the top of the list most of the time.

In order to make sure this wasn't normal, we removed EES and we did not see opendirectyd spike the CPU usage.  Next we unjoined the machine from AD and reinstalled EES.  When not joined to the domain, there was no freezing at login or spike in opendirectoryd.

We next looked to Console for clues.  We couldn't find anything conclusive, but on one machine we did find 600+ entries like this:

Feb  6 12:47:09 <machine name here> esets_gui[1171]: error[04930000]: Cannot retrieve information about user "<user ID here>" (uid 630748767): Permission denied

All 600+ entries were logged over a 2 second period of time.  There were user and service accounts listed.  But there are more than 3000+ user/service accounts in our AD, so the list was not comprehensive.

Next we joined the machine to a test domain with only a few computers and about 3 user accounts.  No issues with opendirectoryd.  Our guess is the number of user accounts maybe the catalyst to why some domain joined Macs are reporting this issue and other don't.  It depends on the size of the company.  Also most of our Mac users rarely reboot and are only sleeping there computers most of the time.  So they don't encounter the login issue unless the shutdown or restart.

Other interesting finds.

• No issue if the computer cannot connect to the domain (this could be a workaround to get someone logged in faster, just prevent connection to the domain network)
• The first time you open EES and go to Setup > Real-time file system protection > Setup… there is a long period until the setup window opens.  During this time opendirectoryd spikes the CPU.  Opening Setup… a subsequent time has no delay or spike.
• Clicking around the EES GUI you often see opendirectoryd spike.
• During login and after the unresponsive period, there are two EES processes listed, one will be red (not responding) until the EES splash screen displays.
• We tried 6.4.88, .132, and .161.  All have the same issue.  Also, the latest .161 nearly made the machine unusable and had to uninstall it in safemode.

We are guessing EES is enumerating the user objects in AD which might be causing this problem.  Not sure why some users barely notice and others have 45 minute freezing periods.  There's probably some other component involved.

Hopefully this will point ESET in a direction to get this resolved.  At this point, we are considering  dropping down to NOD32 for Business v4 until we have a version of ESS6 that no longer has this issue.

If we discover anything else that may be useful, we'll post it here.

[plex 2]

Just in case there is a problem reproducing the issue, we tried to determine what it would take to create the issue on a new test domain.

We first created 5000 user accounts only: no issue
Next we created 3000 user accounts and 3000 computer accounts: issue occurs
Finally we tried 3000 computer accounts only: no issue

So, not sure if its the combination of user and computer accounts or just total number of objects (in which case it would be between 5000-6000).  I'll leave it to ESET to figure out that number or combination that is required.  I only wanted to make sure an environment could be created from scratch that allow the issue to occur.

For completeness, here are the powershell scripts used to create the accounts (edit as needed):

Create-TestAcounts.ps1

$objcount = 3000
$objprefix = "TEMP"
$password = ConvertTo-SecureString "Password1" -AsPlainText -Force

New-ADOrganizationalUnit "Temp" -ProtectedFromAccidentalDeletion $false
New-ADOrganizationalUnit `
    -Name "Users" `
    -Path "OU=Temp,DC=example,DC=com" `
    -ProtectedFromAccidentalDeletion $false
New-ADOrganizationalUnit `
    -Name "Computers" `
    -Path "OU=Temp,DC=example,DC=com" `
    -ProtectedFromAccidentalDeletion $false

for($i=1; $i -le $objcount; $i++) {

    $newuser = "$objprefix$i"
    $newcomputer = "PC-$objprefix$i"
    [decimal] $pct = ($i/$objcount)*100
    $progress = [Math]::Floor($pct)

    Write-Progress `
        -Activity "Creating new AD objects..." `
        -Status "Creating $newuser and $newcomputer" `
        -PercentComplete $progress;
    New-ADUser `
        -SamAccountName $newuser `
        -Name $newuser `
        -GivenName $newuser `
        -Surname "Test" `
        -DisplayName "$newuser Test" `
        -AccountPassword $password `
        -Path "OU=Users,OU=Temp,DC=example,DC=com" `
        -Enabled $true
    New-ADComputer `
        -Name $newcomputer `
        -SamAccountName $newcomputer `
        -Path "OU=Computers,OU=Temp,DC=example,DC=com" `
        -Enabled $true
}

Remove-TestAccounts.ps1

$objcount = 3000
$objprefix = "TEMP"

for($i=1; $i -le $objcount; $i++) {

    $removeuser = "$objprefix$i"
    $removecomputer = "PC-$objprefix$i"
    [decimal] $pct = ($i/$objcount)*100
    $progress = [Math]::Floor($pct)

    Write-Progress `
        -Activity "Removing AD objects..." `
        -Status "Removing $removeuser and $removecomputer" `
        -PercentComplete $progress
    
    Remove-ADUser -Identity $removeuser -Confirm:$false
    Remove-ADComputer -Identity $removecomputer -Confirm:$false
}

Remove-ADOrganizationalUnit -Identity "OU=Temp,DC=example,DC=com" -Recursive -Confirm:$false

 

[Warcholek 0]

Personally I don't think that domain is a problem. At the moment we don't have Mac's joined to AD domain and problem still exists( less computers affected , but still). For us problem started happening after deployment of 802.1x in our company. Issue happens when internet connection takes a bit longer time. When affected Mac tries to connect to normal network ( lets say WPA 2) - problem almost never happens. When it is connecting to 802.1x and authentication takes few seconds to happen - ESET problem appears. Also affected mac's work better via ethernet  ( connected faster). Bigger AD takes more time to obtain internet connection on your mac , thats why you get ESET problem almost every time.

 

[plex 2]

ESET has stated they are having trouble replicating the issue.  Using a domain joined Mac helps reproduce this problem.  Not that it's the ONLY way (or reason) the problem exists.

We aren't purporting the domain is the issue.  What does appear to be the issue is how EES is interacting with opendirectoryd.  But having a domain joined Mac will cause opendirectoryd to be used and therefore allow ESET to replicate the issue.

It would not surprise me at all if opendirectoryd was used by macOS to provide 802.1X support.  I would suggest trying to use Activity Monitor to see if, in fact, opendirectoryd is the culprit for your situation as well.

[j-gray 13]

On 2/8/2017 at 8:44 AM, plex said:

We have been having this same issue.  We have 700+ Macs and many have complained about the system unresponsiveness at login using ESET Endpoint Security 6.3.85.

Curious if you have a case open with support. I'm still being told we're the only ones having this issue.

Also wondering if you've had a chance to try the latest RC linked above? I have not had a chance, yet.

[plex 2]

No ticket.  We have just finished our deployment and are communicating through our ESET contact that helped with our deployment.

I am posting here to "crowd-source" testing so that others with this issue might confirm, discount, or even amend our findings.  I think it would be more useful if others tried some of the things we have tried.  Maybe someone else will uncover one more clue to help get this resolved more quickly.  We would be more than happy to try or look for anything else someone might stumble across.

As for testing other RCs:

Quote

• We tried 6.4.88, .132, and .161.  All have the same issue.  Also, the latest .161 nearly made the machine unusable and had to uninstall it in safemode.

 

[Peter Randziak 557]

Hello,

thank you for the detailed analysis, good job indeed!

We will try to reproduce it as you described.

Regards, P.R.

[plex 2]

Saw that 6.4.168 was available.  Tested it out, still does not prevent high opendirectoryd CPU usage or system unresponsiveness issue.

[Warcholek 0]

19 hours ago, plex said:

Saw that 6.4.168 was available.  Tested it out, still does not prevent high opendirectoryd CPU usage or system unresponsiveness issue.

I've checked CPU usage from opendirectoryd with 802.1x connection and it was 0,1 % CPU while Mac was unresponsive.

Always almost same scenario - user had many applications turned on ,mac was in sleep mode and after waking up - ESET strikes again. Problem occurs when connection takes longer than ( x? sec time)

Also recently i've deployed Newest version of ESET Via ERA to users with slower Mac's (C2D for example), and slower internet connection - instant problem.

Wierd thing is , that some of our macs have this problem like 5-10 times and then ESET Endpoint Antivirus disappear from a bar , and user is unable to start AV - like it would not be installed. ( Visible in Applications, Finder etc.)

 

[Peter Randziak 557]

Hello guys,

our QA guys tried to reproduce the issue as described, but they did not succeed :-(, which makes this issue even more mysterious to me.

Can please someone, who has the issue with the 6.4.168 build collect the logs and send me a private message with a download link so we can check it for any clues?

Thank you in advance, P.R.

[TomasP 267]

Hello all,

I was just informed that after several various attemps, our developers have finally managed to reproduce the issue on one specific hardware configuration, so we may start working on a fix.

Our special thanks go to @plex for providing the detailed description and the reproduction steps!

Thank you for your patience so far.

Regards,
T.

[j-gray 13]

FWIW, to reduce the variables, ESET recommended performing a custom install and install only the real-time component, leaving off the agent, as well.

We're having better luck with the latest build, meaning it now sometimes loads very quickly after logon. But random times it still hangs at logon. Same as before; one ESET process running, the second process takes some time to appear, then appears red/not responding. Once that service resolves itself everything works fine.

Still experimenting trying to determine what causes the hang --seems totally random now whereas before it was easily reproducible.

[kingoftheworld 7]

On 2/14/2017 at 10:17 AM, TomasP said:

Hello all,

I was just informed that after several various attemps, our developers have finally managed to reproduce the issue on one specific hardware configuration, so we may start working on a fix.

Our special thanks go to @plex for providing the detailed description and the reproduction steps!

Thank you for your patience so far.

Regards,
T.

Now that the issue has been reproduced, do we have an ETA on a fix? With the information from this post, our OSX engineer also confirmed that ESET through opendirectoryd is enumerating pretty much our entire directory including users and group memberships.  We currently have licenses purchased that we are unable to deploy to our environment because of this issue.

[Peter Randziak 557]

Hello guys,

so we have a build, which fixes the issue we reproduced internally as described by Plex. 

@plex We would like to thank you for your detailed steps to reproduce, without which we wouldn't be able to reproduce the issue reliably and prepare a fix for it.

The builds are available at: http://ftp.nod.sk/~mego/RC/business/6.4.178/ 

note: you may receive notification attached (for privileged user accounts).

It is just a cosmetic flaw and does not have any affect on the products functionality.

 

Please let us know if the build resolved the issue for you or if it persists.

Thank you once again for cooperation on this.

Regards, P.R.

[mfichera 2]

4 hours ago, Peter Randziak said:

Hello guys,

so we have a build, which fixes the issue we reproduced internally as described by Plex. 

@plex We would like to thank you for your detailed steps to reproduce, without which we wouldn't be able to reproduce the issue reliably and prepare a fix for it.

The builds are available at: hxxp://ftp.nod.sk/~mego/RC/business/6.4.178/ 

note: you may receive notification attached (for privileged user accounts).

It is just a cosmetic flaw and does not have any affect on the products functionality.

 

Please let us know if the build resolved the issue for you or if it persists.

Thank you once again for cooperation on this.

Regards, P.R.

Thanks for the update PR. Are the changes in this build reflected in the new 6.4.168 clients?

[Peter Randziak 557]

Hello Matthew,

20 hours ago, mfichera said:

Thanks for the update PR. Are the changes in this build reflected in the new 6.4.168 clients?

sadly no as it is a freshly baked build, but if the fix will work correctly, it will get to the next released version.

Regards, P.R.

[kingoftheworld 7]

32 minutes ago, Peter Randziak said:

Hello Matthew,

sadly no as it is a freshly baked build, but if the fix will work correctly, it will get to the next released version.

Regards, P.R.

We are still working on testing to see if this correct the issue we are experiencing.  However, when you mention the next released version, will this be soon or will it have to wait until the next major release?