Jump to content

Simultaneous connection count


Recommended Posts

After upgrading ERA (Ubuntu 14.04) to 6.3 from 6.2 I noticed that "last connection" status of agents are not updating. So I looked over my server and saw enormous count of simultaneous connections but the server is running fine all the other info in network section is normal and all agents execute tasks without any problem.

post-9816-0-45406700-1453735948_thumb.png

Link to comment
Share on other sites

  • ESET Staff

Please ignore those "enormous" numbers. Seems there are caused by broken arithmetic computations, and we will be investigating it.

 

In case last connection time is still not working, could you try to restart ERA SERVER service and check (after few minutes) whether it helped? Also please check whether currently shown times are not "in the future", because current logic is not showing last connection time, but latest time of previous connections (= check whether local time on SERVER is correct).

Link to comment
Share on other sites

Time on server is correct, "last connection" is not updating (in fact some random agent statuses are updating) only when this bug with large numbers happens. When I restart virtual machine with ERA all works normally for some time. For example in picture you can see Jan 25 09:00 with my normal average connection count then at some point it starts to grow and agent statuses are not updating. Today after restart ERA already 3 hours works normally.

Link to comment
Share on other sites

My "last connected" statuses are not updating too. After restarting ERA server they are updating some time, but then stop updating again.

And at about the same time server trace log (/var/log/eset/RemoteAdministrator/Server/trace.log) states:

2016-01-26 10:40:06 Error: CDataMinersModule [Thread 7fec577eb700]: CClientsDataMiner2: Failed with: [MySQL][ODBC 5.3(w) Driver][mysqld-5.6.28]Lock wait timeout exceeded; try restarting transaction (1205)

It seems to be a database issue.

Edited by vadim
Link to comment
Share on other sites

  • ESET Staff

Time on server is correct, "last connection" is not updating (in fact some random agent statuses are updating) only when this bug with large numbers happens. When I restart virtual machine with ERA all works normally for some time. For example in picture you can see Jan 25 09:00 with my normal average connection count then at some point it starts to grow and agent statuses are not updating. Today after restart ERA already 3 hours works normally.

 

When it stops working, please check /var/log/eset/RemoteAdministrator/Server/trace.log for Error entries. Those invalid average number should not be related to this problem. Also we have seen similar behavior in case unsupported MariaDB was used instead of MySQL database - what database/version are you using?

 

Please provide content of /etc/odbcinst.ini or at least check whether multi-threading is enabled for you MySQL driver -> it is parameter Threading=0 as described in ERA documentation: hxxp://help.eset.com/era_install/63/en-US/index.html?odbc_configuration.htm

Link to comment
Share on other sites

There is no 'Threading' parameter (see below). I'll try to add and update.

[PostgreSQL]
Description        = ODBC for PostgreSQL
Driver        = /usr/lib/psqlodbc.so
Setup        = /usr/lib/libodbcpsqlS.so
Driver64        = /usr/lib64/psqlodbc.so
Setup64        = /usr/lib64/libodbcpsqlS.so
FileUsage        = 1

[MySQL ODBC 5.3 Unicode Driver]
Driver        = /usr/lib64/libmyodbc5w.so
UsageCount        = 1

[MySQL ODBC 5.3 ANSI Driver]
Driver        = /usr/lib64/libmyodbc5a.so
UsageCount        = 1

We are using MySQL (supplied with virtual appliance).

Edited by vadim
Link to comment
Share on other sites

Problem still exists. In trace log every time it happens i found this

2016-01-26 18:29:12 Error: NetworkModule [Thread 7f71ff5fe700]: Error reported by JobScheduler[Name:Dns job scheduler for not network operation]. Error message is:resolve: Host not found (non-authoritative), try again later
2016-01-26 18:33:32 Warning: NetworkModule [Thread 7f7218dfa700]: The connection will be closed due to timeout. Resolved endpoint is NULL

Link to comment
Share on other sites

MartinK,

It turns out my problem is not fixed too :(

Last status update is "Jan 26 20:59:51" (UTC) and I have the same MySQL error:

2016-01-26 21:00:59 Error: CDataMinersModule [Thread 7f7b7d1f4700]: CClientsDataMiner2: Failed with: [MySQL][ODBC 5.3(w) Driver][mysqld-5.6.28]Lock wait timeout exceeded; try restarting transaction (1205)
Link to comment
Share on other sites

Please attach content of /etc/odbcinst.ini to be sure it is correctly configured.

[PostgreSQL]
Description        = ODBC for PostgreSQL
Driver        = /usr/lib/psqlodbc.so
Setup        = /usr/lib/libodbcpsqlS.so
Driver64        = /usr/lib64/psqlodbc.so
Setup64        = /usr/lib64/libodbcpsqlS.so
FileUsage        = 1
UsageCount        = 2

[MySQL ODBC 5.3 Unicode Driver]
Driver        = /usr/lib64/libmyodbc5w.so
UsageCount        = 2
Threading        = 0

[MySQL ODBC 5.3 ANSI Driver]
Driver        = /usr/lib64/libmyodbc5a.so
UsageCount        = 2
Threading        = 0

Link to comment
Share on other sites

 

MartinK,

It turns out my problem is not fixed too :(

Last status update is "Jan 26 20:59:51" (UTC) and I have the same MySQL error:

2016-01-26 21:00:59 Error: CDataMinersModule [Thread 7f7b7d1f4700]: CClientsDataMiner2: Failed with: [MySQL][ODBC 5.3(w) Driver][mysqld-5.6.28]Lock wait timeout exceeded; try restarting transaction (1205)

 

Today I have the same error in the same time:

2016-01-27 21:01:04 Error: CDataMinersModule [Thread 7f4a427e3700]: CClientsDataMiner2: Failed with: [MySQL][ODBC 5.3(w) Driver][mysqld-5.6.28]Lock wait timeout exceeded; try restarting transaction (1205)

ERA logs time in UTC(+0) and my local time is UTC+3, so maybe the error occurs every local midnight.

Link to comment
Share on other sites

Restarting just mysqld doesn't fix statuses updating.

How can I restart ERA server / agent services on virtual appliance (without rebooting OS)?

Link to comment
Share on other sites

  • ESET Staff

For those with this problem, could you please add innodb_lock_wait_timeout=600 to your MySQL configuration (/etc/my.cnf) into [mysqld] section and restart database server?

 

We think it is caused by database cleanups that are executed at midnight of local time and also just after startup in case SERVER was not running during midnight. Could you all confirm this?

Link to comment
Share on other sites

For those with this problem, could you please add innodb_lock_wait_timeout=600 to your MySQL configuration (/etc/my.cnf) into [mysqld] section and restart database server?

 

We think it is caused by database cleanups that are executed at midnight of local time and also just after startup in case SERVER was not running during midnight. Could you all confirm this?

 

After more than 4 hours, "last connection status" problem seems solved!!! (or appears)

 

@MartinK, can you please confirm my odbcinst.ini settings???

 

[PostgreSQL]
Description             = ODBC for PostgreSQL
Driver          = /usr/lib/psqlodbc.so
Setup           = /usr/lib/libodbcpsqlS.so
Driver64                = /usr/lib64/psqlodbc.so
Setup64         = /usr/lib64/libodbcpsqlS.so
FileUsage               = 1

[MySQL ODBC 5.3 Unicode Driver]
Driver          = /usr/lib64/libmyodbc5w.so
UsageCount              = 1
Threading = 0

[MySQL ODBC 5.3 ANSI Driver]
Driver          = /usr/lib64/libmyodbc5a.so
UsageCount              = 1
Threading = 0

Now remains only the problem about ERA not showing when task execution occured.

 

Thanks!

Link to comment
Share on other sites

  • ESET Staff

After more than 4 hours, "last connection status" problem seems solved!!! (or appears)

 

As I mentioned in previous posts, critical will be midnight (= local time 00:00:00 on your SERVER machine), but we hope this configuration change will minimize chance it happens.

 

@MartinK, can you please confirm my odbcinst.ini settings???

 

Your ODBC configuration is now the same as in newer ERA Appliances and you may benefit from improved performance (parallel execution and query canceling).

Link to comment
Share on other sites

I hope, but previously the problem appeared only after 2 hours. I will waiting midnight  :D.... Instead something about "task execution" problems?? This remain.

Edited by aquarium
Link to comment
Share on other sites

  • ESET Staff

I hope, will waiting midnight  :D.... Instead something about "task execution" problems?? This remain.

 

Seems client task executions time was not considered as essential and redesign was focused more on better "visualization" of task targets and status. Execution time is available in "History" view of client task, but unfortunately it requires more "mouse clicking" to get there.

 

Investigation whether it is "by design" or bug in currently underway.

Link to comment
Share on other sites

So what about my error?  :huh: This topic is only getting answers for another issue.

 

Problem still exists. In trace log every time it happens i found this

2016-01-26 18:29:12 Error: NetworkModule [Thread 7f71ff5fe700]: Error reported by JobScheduler[Name:Dns job scheduler for not network operation]. Error message is:resolve: Host not found (non-authoritative), try again later
2016-01-26 18:33:32 Warning: NetworkModule [Thread 7f7218dfa700]: The connection will be closed due to timeout. Resolved endpoint is NULL

Link to comment
Share on other sites

For those with this problem, could you please add innodb_lock_wait_timeout=600 to your MySQL configuration (/etc/my.cnf) into [mysqld] section and restart database server?

 

We think it is caused by database cleanups that are executed at midnight of local time and also just after startup in case SERVER was not running during midnight. Could you all confirm this?

This midnight passed without errors. Increasing lock wait timeout helps. Thanks a lot.

Link to comment
Share on other sites

  • ESET Staff

So what about my error?  :huh: This topic is only getting answers for another issue.

 

We have looked into it, and order of events was most probably as this:

 

Received connection -> Reverse DNS resolving failed (maybe DNS server was not available at that moment) -> Connection timeouted after 5 minutes -> Timeout caused invalid numbers in statistics (this is our bug we will fix in next release).

Link to comment
Share on other sites

Guest
This topic is now closed to further replies.
  • Recently Browsing   0 members

    • No registered users viewing this page.
×
×
  • Create New...