Number of pending logs increased after ESMC upgrade to 7.2

[FerdinandG 3]

Hi,

We've recently upgraded ESMC server from 7.1 to 7.2. However, we've noticed that the number of pending logs increased around three times since then. Is there any way to increase the ESMC performance in order to reduce the number of pending logs?

We've also noticed from the ESMC server's status.html, there is Replication Throttling scope that showed the following settings:

•Max logs count is : 14000
•Max logs KB is : 1048576 KB
•Max agents is : 280

Are these settings limit the number of logs that are processed by the server? If so, can we change the limits? Because we often see that the number of logs count or agents are more than the maximum number and the replication throttling state is throttled.

ESMC is running on Windows Server 2012 OS with 4 CPU core and 16GB RAM. The average CPU utilization is around 25% and the average memory usage is 8GB

 

Thank you

[MartinK 378]

Indeed ESMC 7.2 introduced mechanisms for throttling connections and received data -> its purpose is to limit load and prevent service exhaustion for temporary peaks, mostly detected during work time hours start. This change was definitely not supposed to increase number of pending logs, but during development, it was discovered that counters were previously not accurate, which might explain increase you are seeing.

Regarding performance, most crucial is performance of database, which is connected to performance of underlying storage. I would recommend to check whether storage performance is not hitting its limits. In case of cloud, I would recommend to check IOPS limits on storage and database.

Could you also provide number of managed / actively connected endpoints just for statistical purpose? We are interested in such numbers as it would enable us to adapt mentioned settings.

[FerdinandG 3]

Hi @MartinK,

Currently we are upgrading from ESET Endpoint Antivirus 5 to ESET Endpoint Antivirus 7.3 in all of our endpoints. The number of actively connected endpoints right now are 37.000, but at the end they should be around 46.000 endpoints. All endpoints connected to ESMC through two Apache HTTP Proxy servers, but we have more local Apache HTTP Proxy in our 300-ish branches that chaining to those two Apache HTTP Proxy servers at the HQ.  We have three different timezone so the temporary peaks might be divided into three groups based on the timezone.

I suppose we have adequate storage/IO performance because the database disks are in our enterprise storage. The thing that I'm concerned is we are using MySQL 5.7 on Linux for the database. Is there any configuration to increase the MySQL database performance? FYI, we have separated ESMC and database servers.

 

Edited June 28, 2020 by FerdinandG
Added the separated ESMC and database servers information

[MartinK 378]

11 hours ago, FerdinandG said:

I suppose we have adequate storage/IO performance because the database disks are in our enterprise storage. The thing that I'm concerned is we are using MySQL 5.7 on Linux for the database. Is there any configuration to increase the MySQL database performance? FYI, we have separated ESMC and database servers.

Thanks for sizing details. In case of such "enterprise-grade" network, it is normal that there might be pending logs for some time, but should be no longer than your replication interval, otherwise you might experience some delays when managing clients.

Regarding database, I am quite surprised that you were able to use MySQL in such a big environment. Our experience shows that once environment is larger that ~10000, performance starts to degrade, especially in terms of console responsibility. That is why it is not recommended to use MySQL in such large environments. But it might be mitigated to some extend by employing custom configuration of database, which is extremely hard to achieve without testing in real environment, but it seems you were already successful. That is why I have no hints for improvements .. maybe just increasing all possible buffers or improving tweaks that are probably already applied.

[FerdinandG 3]

2 hours ago, MartinK said:

Thanks for sizing details. In case of such "enterprise-grade" network, it is normal that there might be pending logs for some time, but should be no longer than your replication interval, otherwise you might experience some delays when managing clients.

Regarding database, I am quite surprised that you were able to use MySQL in such a big environment. Our experience shows that once environment is larger that ~10000, performance starts to degrade, especially in terms of console responsibility. That is why it is not recommended to use MySQL in such large environments. But it might be mitigated to some extend by employing custom configuration of database, which is extremely hard to achieve without testing in real environment, but it seems you were already successful. That is why I have no hints for improvements .. maybe just increasing all possible buffers or improving tweaks that are probably already applied.

The replication interval is 50 minutes, at working hours the pending logs are about 4000 to 5000 logs now. While using ESMC 7.1, the pending logs are about 1500 to 2500 as far as I remember.

Actually I would like to say that the performance is not good. Most of the reports cannot be displayed due to 90000ms time out error. Some of the panels in the dashboard fail to show, for instance the Product Version Status. Even the ESET Application tab took ages to show, and needs to retry several times before the tables are shown. We have told this problem to the local support, and they are still want to try to tuning the database first. We need to create reports based on ESMC reports and we are still struggling to do such thing. I'm afraid that the more endpoints are successful to connect to the ESMC server, the more dashboard panels will failed to load.