Jump to content

Archived

This topic is now archived and is closed to further replies.

girts.u

Simultaneous connection count

Recommended Posts

After upgrading ERA (Ubuntu 14.04) to 6.3 from 6.2 I noticed that "last connection" status of agents are not updating. So I looked over my server and saw enormous count of simultaneous connections but the server is running fine all the other info in network section is normal and all agents execute tasks without any problem.

post-9816-0-45406700-1453735948_thumb.png

Share this post


Link to post
Share on other sites

Please ignore those "enormous" numbers. Seems there are caused by broken arithmetic computations, and we will be investigating it.

 

In case last connection time is still not working, could you try to restart ERA SERVER service and check (after few minutes) whether it helped? Also please check whether currently shown times are not "in the future", because current logic is not showing last connection time, but latest time of previous connections (= check whether local time on SERVER is correct).

Share this post


Link to post
Share on other sites

Time on server is correct, "last connection" is not updating (in fact some random agent statuses are updating) only when this bug with large numbers happens. When I restart virtual machine with ERA all works normally for some time. For example in picture you can see Jan 25 09:00 with my normal average connection count then at some point it starts to grow and agent statuses are not updating. Today after restart ERA already 3 hours works normally.

Share this post


Link to post
Share on other sites

My "last connected" statuses are not updating too. After restarting ERA server they are updating some time, but then stop updating again.

And at about the same time server trace log (/var/log/eset/RemoteAdministrator/Server/trace.log) states:

2016-01-26 10:40:06 Error: CDataMinersModule [Thread 7fec577eb700]: CClientsDataMiner2: Failed with: [MySQL][ODBC 5.3(w) Driver][mysqld-5.6.28]Lock wait timeout exceeded; try restarting transaction (1205)

It seems to be a database issue.

Share this post


Link to post
Share on other sites

Time on server is correct, "last connection" is not updating (in fact some random agent statuses are updating) only when this bug with large numbers happens. When I restart virtual machine with ERA all works normally for some time. For example in picture you can see Jan 25 09:00 with my normal average connection count then at some point it starts to grow and agent statuses are not updating. Today after restart ERA already 3 hours works normally.

 

When it stops working, please check /var/log/eset/RemoteAdministrator/Server/trace.log for Error entries. Those invalid average number should not be related to this problem. Also we have seen similar behavior in case unsupported MariaDB was used instead of MySQL database - what database/version are you using?

 

Please provide content of /etc/odbcinst.ini or at least check whether multi-threading is enabled for you MySQL driver -> it is parameter Threading=0 as described in ERA documentation: hxxp://help.eset.com/era_install/63/en-US/index.html?odbc_configuration.htm

Share this post


Link to post
Share on other sites

There is no 'Threading' parameter (see below). I'll try to add and update.

[PostgreSQL]
Description        = ODBC for PostgreSQL
Driver        = /usr/lib/psqlodbc.so
Setup        = /usr/lib/libodbcpsqlS.so
Driver64        = /usr/lib64/psqlodbc.so
Setup64        = /usr/lib64/libodbcpsqlS.so
FileUsage        = 1

[MySQL ODBC 5.3 Unicode Driver]
Driver        = /usr/lib64/libmyodbc5w.so
UsageCount        = 1

[MySQL ODBC 5.3 ANSI Driver]
Driver        = /usr/lib64/libmyodbc5a.so
UsageCount        = 1

We are using MySQL (supplied with virtual appliance).

Share this post


Link to post
Share on other sites

Will try with config Threading=0 as my version of unixODBC is 2.2.14p2-5ubuntu5. 

Share this post


Link to post
Share on other sites

Statuses updating for an hour and a half. The issue seems to be fixed.

Thanks, MartinK!

!

Share this post


Link to post
Share on other sites

Problem still exists. In trace log every time it happens i found this

2016-01-26 18:29:12 Error: NetworkModule [Thread 7f71ff5fe700]: Error reported by JobScheduler[Name:Dns job scheduler for not network operation]. Error message is:resolve: Host not found (non-authoritative), try again later
2016-01-26 18:33:32 Warning: NetworkModule [Thread 7f7218dfa700]: The connection will be closed due to timeout. Resolved endpoint is NULL

Share this post


Link to post
Share on other sites

MartinK,

It turns out my problem is not fixed too :(

Last status update is "Jan 26 20:59:51" (UTC) and I have the same MySQL error:

2016-01-26 21:00:59 Error: CDataMinersModule [Thread 7f7b7d1f4700]: CClientsDataMiner2: Failed with: [MySQL][ODBC 5.3(w) Driver][mysqld-5.6.28]Lock wait timeout exceeded; try restarting transaction (1205)

Share this post


Link to post
Share on other sites

Please attach content of /etc/odbcinst.ini to be sure it is correctly configured.

Share this post


Link to post
Share on other sites

Please attach content of /etc/odbcinst.ini to be sure it is correctly configured.

[PostgreSQL]
Description        = ODBC for PostgreSQL
Driver        = /usr/lib/psqlodbc.so
Setup        = /usr/lib/libodbcpsqlS.so
Driver64        = /usr/lib64/psqlodbc.so
Setup64        = /usr/lib64/libodbcpsqlS.so
FileUsage        = 1
UsageCount        = 2

[MySQL ODBC 5.3 Unicode Driver]
Driver        = /usr/lib64/libmyodbc5w.so
UsageCount        = 2
Threading        = 0

[MySQL ODBC 5.3 ANSI Driver]
Driver        = /usr/lib64/libmyodbc5a.so
UsageCount        = 2
Threading        = 0

Share this post


Link to post
Share on other sites

I have the same problem, also now ERA not shows when task execution occured.

Share this post


Link to post
Share on other sites

... not shows when task execution occured.

 

+1

I'm shocked.

Share this post


Link to post
Share on other sites

 

MartinK,

It turns out my problem is not fixed too :(

Last status update is "Jan 26 20:59:51" (UTC) and I have the same MySQL error:

2016-01-26 21:00:59 Error: CDataMinersModule [Thread 7f7b7d1f4700]: CClientsDataMiner2: Failed with: [MySQL][ODBC 5.3(w) Driver][mysqld-5.6.28]Lock wait timeout exceeded; try restarting transaction (1205)

 

Today I have the same error in the same time:

2016-01-27 21:01:04 Error: CDataMinersModule [Thread 7f4a427e3700]: CClientsDataMiner2: Failed with: [MySQL][ODBC 5.3(w) Driver][mysqld-5.6.28]Lock wait timeout exceeded; try restarting transaction (1205)

ERA logs time in UTC(+0) and my local time is UTC+3, so maybe the error occurs every local midnight.

Share this post


Link to post
Share on other sites

Restarting just mysqld doesn't fix statuses updating.

How can I restart ERA server / agent services on virtual appliance (without rebooting OS)?

Share this post


Link to post
Share on other sites

How can I restart ERA server / agent services ...

initctl restart eraserver

...is fixing statuses.

Till midnight?..

Share this post


Link to post
Share on other sites

For those with this problem, could you please add innodb_lock_wait_timeout=600 to your MySQL configuration (/etc/my.cnf) into [mysqld] section and restart database server?

 

We think it is caused by database cleanups that are executed at midnight of local time and also just after startup in case SERVER was not running during midnight. Could you all confirm this?

Share this post


Link to post
Share on other sites

For those with this problem, could you please add innodb_lock_wait_timeout=600 to your MySQL configuration (/etc/my.cnf) into [mysqld] section and restart database server?

 

We think it is caused by database cleanups that are executed at midnight of local time and also just after startup in case SERVER was not running during midnight. Could you all confirm this?

 

After more than 4 hours, "last connection status" problem seems solved!!! (or appears)

 

@MartinK, can you please confirm my odbcinst.ini settings???

 

[PostgreSQL]
Description             = ODBC for PostgreSQL
Driver          = /usr/lib/psqlodbc.so
Setup           = /usr/lib/libodbcpsqlS.so
Driver64                = /usr/lib64/psqlodbc.so
Setup64         = /usr/lib64/libodbcpsqlS.so
FileUsage               = 1

[MySQL ODBC 5.3 Unicode Driver]
Driver          = /usr/lib64/libmyodbc5w.so
UsageCount              = 1
Threading = 0

[MySQL ODBC 5.3 ANSI Driver]
Driver          = /usr/lib64/libmyodbc5a.so
UsageCount              = 1
Threading = 0

Now remains only the problem about ERA not showing when task execution occured.

 

Thanks!

Share this post


Link to post
Share on other sites

After more than 4 hours, "last connection status" problem seems solved!!! (or appears)

 

As I mentioned in previous posts, critical will be midnight (= local time 00:00:00 on your SERVER machine), but we hope this configuration change will minimize chance it happens.

 

@MartinK, can you please confirm my odbcinst.ini settings???

 

Your ODBC configuration is now the same as in newer ERA Appliances and you may benefit from improved performance (parallel execution and query canceling).

Share this post


Link to post
Share on other sites

I hope, but previously the problem appeared only after 2 hours. I will waiting midnight  :D.... Instead something about "task execution" problems?? This remain.

Share this post


Link to post
Share on other sites

I hope, will waiting midnight  :D.... Instead something about "task execution" problems?? This remain.

 

Seems client task executions time was not considered as essential and redesign was focused more on better "visualization" of task targets and status. Execution time is available in "History" view of client task, but unfortunately it requires more "mouse clicking" to get there.

 

Investigation whether it is "by design" or bug in currently underway.

Share this post


Link to post
Share on other sites

So what about my error?  :huh: This topic is only getting answers for another issue.

 

Problem still exists. In trace log every time it happens i found this

2016-01-26 18:29:12 Error: NetworkModule [Thread 7f71ff5fe700]: Error reported by JobScheduler[Name:Dns job scheduler for not network operation]. Error message is:resolve: Host not found (non-authoritative), try again later
2016-01-26 18:33:32 Warning: NetworkModule [Thread 7f7218dfa700]: The connection will be closed due to timeout. Resolved endpoint is NULL

Share this post


Link to post
Share on other sites

For those with this problem, could you please add innodb_lock_wait_timeout=600 to your MySQL configuration (/etc/my.cnf) into [mysqld] section and restart database server?

 

We think it is caused by database cleanups that are executed at midnight of local time and also just after startup in case SERVER was not running during midnight. Could you all confirm this?

This midnight passed without errors. Increasing lock wait timeout helps. Thanks a lot.

Share this post


Link to post
Share on other sites

So what about my error?  :huh: This topic is only getting answers for another issue.

 

We have looked into it, and order of events was most probably as this:

 

Received connection -> Reverse DNS resolving failed (maybe DNS server was not available at that moment) -> Connection timeouted after 5 minutes -> Timeout caused invalid numbers in statistics (this is our bug we will fix in next release).

Share this post


Link to post
Share on other sites

  • Recently Browsing   0 members

    No registered users viewing this page.

×
×
  • Create New...