Problems since definition 8766 with 3ware Controller

[socha 1]

Hello,

 

since 05.09.2013 and definition 8766 there are big trouble with ESET File Security and Mail Security on many Server with 3ware 9650 Controller. Event log said for > 1000 times: 

 

  Source 3wareDrv

Event ID 3

06.09.2013 20:56

1.134 *

PostCmd> Error building SGL (READ10)

 

There seems to be no big trouble on the Small Business Servers (2003, 2008), but on Hyper-V (2008r2) the VMs running in a BlueScreen.

Only uninstall ESET helps ...

 

Any one see this fresh error with this Hardware?

[Rolls 0]

Та же самая беда с Файл Секьюрити!  Началась также 05,09,2013. (единственное отличие в ошибке PostCmd> Error building SGL (READ16)

Убил его, поставил обычную серверную версию, пока полёт нормальный.

Edited September 9, 2013 by Rolls

[Peter Randziak 1,130]

Hello Socha,

 

in case you experience BSOD please provide us with full memory dump and SysInspector log for further analysis.

[Willtech 0]

Hi Peter and All,

 

Reporting same issue as OP, with EAV BE, EEA and EFS on all Hyper-V machines running both 2008 r2 and XP .

 

Hello,

 

since 05.09.2013 and definition 8766 there are big trouble with ESET File Security and Mail Security on many Server with 3ware 9650 Controller. ...

... but on Hyper-V (2008r2) the VMs running in a BlueScreen.

Only uninstall ESET helps ...

 

Any one see this fresh error with this Hardware?

 

Also have 3ware (9650?) Controller installed and the issue is Hyper-V machines die with BSOD a few minutes after bootup / login.

 

In Safe Mode disabling the ekrn service for troubleshooting resolves the BSOD issue even after normal reboot until ekrn service is re-started. Even uninstalling, meticulously cleaning all traces of EAV BE / eea / efs from file system and registry and then re-installing does not resolve the issue once updates are collected again from ESET using choose automatically.

 

Hello Socha,

 

in case you experience BSOD please provide us with full memory dump and SysInspector log for further analysis.

 

 

I will endeavour also to collect the logs that you have requested in the morning, it is two hours past knock-off time here already and I'm late for beer.

 

If there is anything else I should know before then please let me know.

 

Cheers

Edited September 9, 2013 by Willtech

[socha 1]

Hi all,

 

my German Support told me there is a solution from 3ware for this problem:

 

https://www.3ware.com/3warekb/article.aspx?id=14869&cNode=4F4F0V

 

But I can not play with my productive Servers ... this solution is from year 2010, the error is a few days ago. 

 

Anyone tried this solution ?

 

What changes on 05.09.2013 in product- or definition update in eset?

[socha 1]

@ Peter Randizak: the log files was sended to german support

 

[Ticket#2013090910000077]

[fips 0]

since 05.09.2013 and definition 8766 there are big trouble with ESET File Security and Mail Security on many Server with 3ware 9650 Controller. Event log said for > 1000 times: 

PostCmd> Error building SGL (READ10)

 

Any one see this fresh error with this Hardware?

 

Hello, i have the same problem.

Running a terminalserver 2008 with eset and a 9650SE controller. A start of egui.exe (user logon) generates pretty much SQL errors. Until every user logged on, there are ten thousand errors.

 

I tried this workaround you linked above, but enabling double buffer will put a brake on the systems performance.

For the moment living with the erros is the better way, i think. Or do not start egui.exe.

 

Best regards from germany.

 

...hoping for a solution...

[c3direct 0]

We have experienced identical issues on four Hyper-V guest machines (all Windows 2008) across two physical server (both have 3ware RAID controllers).

 

This did occur on Sep 5, as others have pointed out.  One host machine (Windows 2008 R2) has ESET installed and does not seem affected by this problem.

 

The BSOD we saw was consistently coming up with 0x000000F4, although I did not gather much more information as it was a work day, so I needed to get the servers stable by uninstalling ESET.

 

This is a pretty big deal, so hopefully developers can look at what was introduced on Sep 5, 2013 that could impact servers in such a way.  If the suggested workaround above works (I am also hesitant to try a 2 year old solution on a new problem) perhaps ESET introduced something that impacted DWORD alignment of buffers.

 

Socha - did you uninstall on the host or the guests (or both)?  If I could keep the guests happy and protected and uninstall on the host, that might be okay for a short-term workaround but I'd rather not play roulette with the servers if I don't need to.  Blue screens and Exchange don't get along from time to time.

[c3direct 0]

UPDATE:  I have received some information from our supplier that ESET may know what is causing this issue.  I am not 100% sure if this is a direct correlation, and I have not yet had a chance to try the solution, but there may be a problem with the eamonm.sys driver file.  There is a development release out that he sent me to try, but it may be a day or two since I can only do this after hours.  There is not currently an installation package fix for this issue.

 

I did try enabling the DoubleBuffer registry key on the host machines this evening, and ESET was able to install and run without the crash.  I am running with that as the workaround until implementing a complete fix.  Thank you, socha, for the suggestion!

[Willtech 0]

UPDATE:  I have received some information from our supplier that ESET may know what is causing this issue.  I am not 100% sure if this is a direct correlation, and I have not yet had a chance to try the solution, but there may be a problem with the eamonm.sys driver file.  There is a development release out that he sent me to try, but it may be a day or two since I can only do this after hours.  There is not currently an installation package fix for this issue.

 

I did try enabling the DoubleBuffer registry key on the host machines this evening, and ESET was able to install and run without the crash.  I am running with that as the workaround until implementing a complete fix.  Thank you, socha, for the suggestion!

 

 

I am able to try a dev release immediately if ESET will make it available for me to do so, please let me know?

 

- have been stuck on my own support calls, requested logs incoming via PM shortly.

... No crash dump is available but after Sysinspector is finished performing the loading sequence I will send that.

Edited September 10, 2013 by Willtech

[socha 1]

@c3direct:

 

first I uninstall eset on the guests, but after 2 more bluescreen over night I uninstall it also on the host. At this time runing MS Secutity Essentials on VMs and host ... 

The other physical Server with SBS2003 and SBS2008 have the same Errors in the system event (>1000 times), but everything seems to work.

 

Please let us know if there is a solution. I only get this link to the old 3ware knowledgebase from German support and don't want to make the servers slow.

 

[c3direct 0]

I can report that the double buffer workaround seems to be working.  Anecdotally, I do not believe the performance has been significantly affected for the end users.

 

@Willtech - I'm afraid I'm not at liberty to distribute a development file.  I'm not 100% sure the file correlates to this problem, so I would hate to distribute something that causes problems.  Perhaps if ESET believes they have a fix in place, a representative could post the file or a link to the file.

[fips 0]

I can report that the double buffer workaround seems to be working.  Anecdotally, I do not believe the performance has been significantly affected for the end users.

 

The systems overall speed is ok, but after copying a file the windows explorer lags many many seconds. I tested it 2 times with buffer on/off, and on my machine it is only when the double buffering is enabled. But it's the host system. I don't use VMs / HyperV

 

Nice that it works for you.

 

Greets

Edited September 11, 2013 by fips

[drhex2000 0]

Same issue here, s2k8r2 with hyper-v and four  s2012 guests. Guests are crashing on initial scan, host shows the errors in the log. Need a fix for this urgently.

Ciao

[Peter Randziak 1,130]

Hello drhex2000,

 

please provide us with full memory dump and SysInspector log and we will look into it as soon as possible.

 

Have you tried suggested workaround with DoubleBuffer registry key?

[socha 1]

@ Peter Randziak

 

do you know a contact to technical support who can give us some information about the work for a solution from eset?

At this time I get no answer from german support, no solution here ... dont know what to do? changing the product?

 

@all: please post here if you have same problem. think eset do not solve the problem for 5 user ... 

[drhex2000 0]

Hi Peter,

 

have added the registry hack as per 3ware and that "fixes" it, the guests are no longer crashing on login. The actual error is happening on the host though. It writes the errors as stated by the thread opener into the syslog on login into one of the guests, either through Hyper-V or RDP. Login crashes the guest on initial ESET scan and writes the errors on the host. I assume this happens when ESET is trying to access the VHD in some manner it doesn't like. As the registry change works around it this must be related to unaligned buffers. The guest then bluescreens with "critical process died". As the host isn't crashing I can't send you meaningful logs. If you have any ideas on how to debug this I am game to help.

Thanks

[Peter Randziak 1,130]

Hello,

 

@socha do you have latest 3ware firmware installed? 

WE will fix it for even one customer as BSOD is critical issue, but we need to know why it crashes and we are not able to find it out without the memory dump/

 

@drhex2000 so please provide us with memory dump from the guest when it is crashing for the beginning.

[drhex2000 0]

Can do, where do you want it?

[c3direct 0]

Hello Peter,

 

I'll see if I can find a dump file, but it sounds as if none of us are really sure how to provide this to you in such a way that it will be meaningful.  Can you provide or link to instructions?

 

Also - we have pretty thoroughly narrowed things down to the point that it should be relatively easy to replicate in your own environment.  3ware controllers should be readily available from eBay for a reasonable amount if you don't already have one available.  Then you could take all the blue screen dumps you'd like.  It's a bit of a challenge to ask your customers to let their production servers blue screen a few times in order to collect data, so a test system in a controlled environment would be a win-win.

 

However, I'm not even sure that much in the way of dump data is necessary.  I assume your developers keep track of changes and updates that are released.  This change was released on Sep. 5, 2013 and almost certainly has to do with aligning one or more buffers to a DWORD boundary.  That information alone should be enough to start tracking down what changes were made that may have created this issue.

 

Let us know how to get information to you, and I'm sure we will do our best to provide you with additional data as we are able.

[Peter Randziak 1,130]

Hello,

 

@drhex2000 just send me download link to packed memory dump. 

@c3direct so what are the steps to reproduce the BSOD, please describe them step-by-stem with as many details as possible.

 

Thank you.

[c3direct 0]

Hi Peter,

 

From what I can gather from my own situation and the descriptions of others on this forum, all you need is a combination of 3ware controller, Hyper-V host, and Hyper-V guest.

 

In our own environment, we are running two different versions of Hyper-V (Windows 2008 R2 on one and Hyper-V Server 2008 on the other) on two different 3ware controllers that are at two different firmware revisions and driver revisions, so I don't believe you need to be too picky about the specifics.  All of our Hyper-V guests are Windows 2008, so I can't be 100% sure whether or not that's requred, but from the other posts I don't believe it to be the case.

 

So, if you install W2k8 or W2k8 R2 (or even Hyper-V Server, although it lacks a GUI making things a bit more challenging to troubleshoot and manage in a test environment) on a physical machine using a 3ware 9650 RAID controller and create a Windows guest and then install ESET into that guest (and possibly the host as well), you should be able to recreate the issue.  The virtual should blue screen within a matter of minutes of boot time.  Again, the time frame during which this was introduced is pretty well-defined, so I don't think you'll have a lot of code changes to look through to find this issue.

[Willtech 0]

Hello,

 

@drhex2000 just send me download link to packed memory dump. 

@c3direct so what are the steps to reproduce the BSOD, please describe them step-by-stem with as many details as possible.

 

Thank you.

 

Hi Peter,

 

I have created Memory Dumps for one Server 2008 R2 x64 VM and one Windows XP x86 VM.

 

It is not so straight forward to create a memory dump for the Hyper-V VM, the BSOD goes through the motions but without the .dmp file actually being created.

 

For my future reference and the use of others if it helps, the procedure is basically as follows;

• Install "Debugging Tools for Windows" (as a Standalone component if you like) on the HOST machine:

  hxxp://bit.ly/157iH5d

   

• Add the "Microsoft Hyper-V VM State to Memory Dump Converter" to the folder on the HOST machine where symsrv.dll for your platform is located (e.g. C:\Program Files (x86)\Windows Kits\8.0\Debuggers\x64\ on a 64-bit system):

  hxxp://bit.ly/17Ky0Uw

   

• Run the VM and then Save State (in this instance you must trigger BSOD first).

   

• Execute the vm2dmp.exe on the HOST machine to create the dump:

  C:\Path>vm2dmp.exe -vm "{vm name}" -dmp C:\VM\Memory.dmp

 

Zipped up the files that I have are around 600MB so I will upload them on an FTP for you and PM the details to collect when that is done.

 

If you would like a fresh SysInspector log also then please let me know?

 

Cheers 

Edited September 19, 2013 by Willtech

[socha 1]

Hi all,

 

think this problem is sleeping here, this is my solution: 

 

Install 3Ware Patch on Hyper-V Host or physical Server and restart: https://www.3ware.com/3warekb/article.aspx?id=14869&cNode=4F4F0V

After this install newest ESET File Security on all Server and VM.

 

Thanks all for posting here and big thanks to german support ... it seems to work (if not, I will post it the next days. 

 

Greetings,

Peter 

[Willtech 0]

Hi all,

 

think this problem is sleeping here, this is my solution: 

 

Install 3Ware Patch on Hyper-V Host or physical Server and restart: https://www.3ware.com/3warekb/article.aspx?id=14869&cNode=4F4F0V

After this install newest ESET File Security on all Server and VM.

 

Thanks all for posting here and big thanks to german support ... it seems to work (if not, I will post it the next days. 

 

Greetings,

Peter 

 

Hi Peter,

 

If that is the solution then the problem is why all of a sudden after a particular signature update on the VM does it cause the buffer on the VM Host to be not aligned?

 

I would rather work to resolve the cause of the problem than implement a workaround.

 

Cheers