Upgrade to 6.4 fixed a lot of issues, but then....

[BDeep 7]

I'm getting this after day three of running ERA 6.4. MOD note: I'm already working with support on this. Just reaching out to the community to see if these are severe problems carrying over from 6.3.

 

Performance details 2016-Jul-22 19:11:00 Detailed performance statistics:

• I/O reads: 413 KB/s
• I/O writes: 1193 KB/s
• I/O others: 0 KB/s
• Logs latency: 8 s
• Pending logs: 5563

Received logs 2016-Jul-22 19:05:37 Received logs statistics:

• Received in last minute: 1445 (24 /s)
• Received in last hour: 42392 (12 /s)

Server performance 2016-Jul-22 19:11:00 ERROR: Overal performance status is LIMITED

• Server can be overloaded due to large amount of received data

Static objects 2016-Jul-22 19:01:18 Cached static objects statistics:

• Total number of cached objects is 9992 (+ removed 13239)
• Computers count is 1451 (+ removed 5621)
• Client tasks count is 54 (+ removed 210)
• Client triggers count is 8236 (+ removed 6943)
• Policies count is 65 (+ removed 15)
• Policy relations count is 52 (+ removed 94)
• Users count is 0 (+ removed 0)
• Users relations count is 0 (+ removed 0)

 

[MartinK 378]

Any chance you have the same statistics from previous days?

Also could you please provide database tables sizes for tables starting with tbl_log_ as it seems you are receiving quite too many logs, especially in case there is no ongoing security incident in your network. In case you are using MS SQL database, there is pre-defined report in Management studio for table sizes.

[BDeep 7]

Any chance you have the same statistics from previous days?

Also could you please provide database tables sizes for tables starting with tbl_log_ as it seems you are receiving quite too many logs, especially in case there is no ongoing security incident in your network. In case you are using MS SQL database, there is pre-defined report in Management studio for table sizes.

CentOS 6 appliance with MySQL. No stats from previous days unless you have access to case notes and uploads, otherwise see the uploads on Case #1441554.

 

Straight from Terminal. Copy/pasted see attached. Opens properly in Notepad++.

TBL.txt

[BDeep 7]

Additional attachments. The MySQL memory, and related, errors have been plaguing me for months. This is the first ERAServer error message I've seen in the console. Most of it has been MySQL>

Edited July 22, 2016 by BDeep

[MartinK 378]

There seems to be only one table (tbl_log_apps_installed_status) with quite too much data .. are you intentionally reporting all installed applications from AGENTs?

It seems database is not able to handle receiving logs even when received logs statistics are not even near limits. This slowness and queuing of received logs most probably increased memory consumption of ERAServer so that it was killed by operating system.

 

Does this problem persists after restart? Could you check memory consumption in upcoming hours? Insufficient memory (caused by error/leak) may be reason of this problems.

 

EDIT: just realized there is too many installed application records for your computer's count. Any chance you are still using older AGENTs? (by older I mean especially 6.1 and possibly 6.2).

[BDeep 7]

All of our agents should be 6.2 or newer as that is when we started down the road with ESET. As of this email, a little less than half (about 600 clients) have agents that are below 6.4

1. How can I have the agents update? I have set auto-update program components for agents in the agents policies but, to date, I have never seen agents on clients update automatically.

 

Yes on intentionally reporting installed applications from agents for Macintosh and Windows. We routinely audit installed software using the findings of ESET Agents and take actions appropriately. It also helps us identify clients running near zero-day exploitable software.

1. How can we keep reporting applications in the current method without crashing the database? As of this posting, I have disabled application reporting on Windows and Macintosh via agent policy as a temporary measure.

Finally, additional information uploaded on case #1441554.

 

Edited July 23, 2016 by BDeep

[MartinK 378]

All of our agents should be 6.2 or newer as that is when we started down the road with ESET. As of this email, a little less than half (about 600 clients) have agents that are below 6.4

1. How can I have the agents update? I have set auto-update program components for agents in the agents policies but, to date, I have never seen agents on clients update automatically.

 

This will update only specific modules but not installation itself. In case of AGENT it is mostly only support for configuring new products.

In order to upgrade you may use Component upgrade task.

 

Yes on intentionally reporting installed applications from agents for Macintosh and Windows. We routinely audit installed software using the findings of ESET Agents and take actions appropriately. It also helps us identify clients running near zero-day exploitable software.

1. How can we keep reporting applications in the current method without crashing the database? As of this posting, I have disabled application reporting on Windows and Macintosh via agent policy as a temporary measure.

 

There should be no problem reporting all installed applications, especially in case it worked until now. I am not sure, but it seems something went wrong during database cleanups (they are performed at 00:00 of SERVER's local time -> seems it may be time it started spiking). I would suggest to check what happens after restart and if it won't help - try to stop SERVER and manually erase content of previously mentioned table tbl_log_apps_installed_status, especially in case it's size will be this huge. You will loose installed applications data for some time (it should fully recover in no more than 24 hours for connecting AGENTs).

[BDeep 7]

 

All of our agents should be 6.2 or newer as that is when we started down the road with ESET. As of this email, a little less than half (about 600 clients) have agents that are below 6.4

1. How can I have the agents update? I have set auto-update program components for agents in the agents policies but, to date, I have never seen agents on clients update automatically.

 

This will update only specific modules but not installation itself. In case of AGENT it is mostly only support for configuring new products.

In order to upgrade you may use Component upgrade task.

 

Yes on intentionally reporting installed applications from agents for Macintosh and Windows. We routinely audit installed software using the findings of ESET Agents and take actions appropriately. It also helps us identify clients running near zero-day exploitable software.

1. How can we keep reporting applications in the current method without crashing the database? As of this posting, I have disabled application reporting on Windows and Macintosh via agent policy as a temporary measure.

 

There should be no problem reporting all installed applications, especially in case it worked until now. I am not sure, but it seems something went wrong during database cleanups (they are performed at 00:00 of SERVER's local time -> seems it may be time it started spiking). I would suggest to check what happens after restart and if it won't help - try to stop SERVER and manually erase content of previously mentioned table tbl_log_apps_installed_status, especially in case it's size will be this huge. You will loose installed applications data for some time (it should fully recover in no more than 24 hours for connecting AGENTs).

 

 

1: ERA components upgrade task will update ERA Agents on client OS' like Windows 8, Macintosh El Capitan, and Linux OS?

2: That is not a problem. How do I erase the content (not a DBA)? Drop table or something else?

[MartinK 378]

1: ERA components upgrade task will update ERA Agents on client OS' like Windows 8, Macintosh El Capitan, and Linux OS?

2: That is not a problem. How do I erase the content (not a DBA)? Drop table or something else?

 

1. Yes, it will upgrade AGENTs and also other ERA components (SERVER, PROXY and Webconsole with some limitations) if installed. Upgrade should work regardless of operating system. Recommended way is to create dynamic group matching AGENTS with older version and assign "Remote Administrator Components upgrade" task. Be aware that upgrade to be successful AGENTs must have access to ESET repository or HTTP proxy as installation package for specific platform will be downloaded on client.

 

2. To fast erase data use these two commands:

TRUNCATE TABLE tbl_log_apps_installed_status;
CALL usp_log_apps_installed_status_reset;

First one will do data removal and second one will fix internal counters so that it actually works afterwards. Please make sure SERVER is not running when executing queries. It would be best if you could avoid at all - so please wait & check whether problems will reappear.

[kingoftheworld 10]

Sorry to stir up an old thread, but has their been any resolution to this?  I am having similar issues with ERA 6.4 on a Windows 12R2 install with an external MS SQL server.  The resources on both system not taxed at all (usually around 10-20%).  However the log queue will show usually around 15k for most of the day.  At random times it will go back down to normal before shooting back up.  I worked with support a couple weeks back and basically was told that it was because I had third party application reporting enabled was likely the cause.  However, the issue came back.  Wanted to check here before I went round and round with support again.

[MartinK 378]

Technically it means that database is not able to handle amount of received data. There are multiple possibilities what could be wrong:

• it may be HW limitation -> write speed is sensitive to DB connection delay (especially if DB is on different machine) and also disk write speed that hosts DB server
• your AGENTs are sending too much data .. for example automatically collected exported configuration in case computer reported problems could flood server with too much data. Also reporting non-ESET applications may be generating big data sets. We have also seen huge quarantines that overloaded database.
• pending logs count can be increased during DB cleanups execution, which are executed at 00:00 of SERVER's local time.
• errors during logs processing (writing into database) may result in overall slowness of processing: any chance your SERVER's trace log is full of DB-related errors?

Please provide basic statistics of your ERA as in previous posts, especially basic information from status.html and DB table sizes (count of rows). In case value "Received in last hour" in log statistics will show higher number, there should be tool-tip showing more detailed information (not sure whether in your version), which may help us find out what type of logs is causing problems.

Log processing has been significantly improved for upcoming version (6.5) to be released soon. Most of changes were performance-related.

Edited January 26, 2017 by MartinK

[kingoftheworld 10]

3 hours ago, MartinK said:

Technically it means that database is not able to handle amount of received data. There are multiple possibilities what could be wrong:

• it may be HW limitation -> write speed is sensitive to DB connection delay (especially if DB is on different machine) and also disk write speed that hosts DB server
• your AGENTs are sending too much data .. for example automatically collected exported configuration in case computer reported problems could flood server with too much data. Also reporting non-ESET applications may be generating big data sets. We have also seen huge quarantines that overloaded database.
• pending logs count can be increased during DB cleanups execution, which are executed at 00:00 of SERVER's local time.
• errors during logs processing (writing into database) may result in overall slowness of processing: any chance your SERVER's trace log is full of DB-related errors?

Please provide basic statistics of your ERA as in previous posts, especially basic information from status.html and DB table sizes (count of rows). In case value "Received in last hour" in log statistics will show higher number, there should be tool-tip showing more detailed information (not sure whether in your version), which may help us find out what type of logs is causing problems.

Log processing has been significantly improved for upcoming version (6.5) to be released soon. Most of changes were performance-related.

Thanks for the reply.  For database specific issues, what should I recommend to our DBA to check for?  SQL isn't showing any errors and there is plenty of available resources on the server.  The database does reside on a separate server from the ERA server. 

The pending logs don't seem to have any patterns around 00:00 localtime. When I am watching this happen, it appears the ERA server gets roughly 15-16k pending logs and then goes into some type of frozen state.  The machines don't appear to be checking and the status.html page doesn't seem to update.  Eventually it starts working again.    Also, the last hour statistic may be a little high now.  I restarted the service after it seemed to be hung for several hours in order to get it back operational and likely is the backlog.

 

 

The trace log seems to quickly fill with:

Edited January 26, 2017 by kingoftheworld

[MartinK 378]

There is definitely too much data, but from trace log or status.html it is hard to find out what type of log it is.

Please check database table sizes (especially size of tables tbl_log_*): it is standard report in case you are using SQLServer with Management studio.