Simultaneous connection count

girts.u · January 25, 2016

After upgrading ERA (Ubuntu 14.04) to 6.3 from 6.2 I noticed that "last connection" status of agents are not updating. So I looked over my server and saw enormous count of simultaneous connections but the server is running fine all the other info in network section is normal and all agents execute tasks without any problem.

MartinK · January 25, 2016

Please ignore those "enormous" numbers. Seems there are caused by broken arithmetic computations, and we will be investigating it.

In case last connection time is still not working, could you try to restart ERA SERVER service and check (after few minutes) whether it helped? Also please check whether currently shown times are not "in the future", because current logic is not showing last connection time, but latest time of previous connections (= check whether local time on SERVER is correct).

girts.u · January 26, 2016

Time on server is correct, "last connection" is not updating (in fact some random agent statuses are updating) only when this bug with large numbers happens. When I restart virtual machine with ERA all works normally for some time. For example in picture you can see Jan 25 09:00 with my normal average connection count then at some point it starts to grow and agent statuses are not updating. Today after restart ERA already 3 hours works normally.

vadim · January 26, 2016

My "last connected" statuses are not updating too. After restarting ERA server they are updating some time, but then stop updating again.

And at about the same time server trace log (/var/log/eset/RemoteAdministrator/Server/trace.log) states:

2016-01-26 10:40:06 Error: CDataMinersModule [Thread 7fec577eb700]: CClientsDataMiner2: Failed with: [MySQL][ODBC 5.3(w) Driver][mysqld-5.6.28]Lock wait timeout exceeded; try restarting transaction (1205)

It seems to be a database issue.

Edited January 26, 2016 by vadim

MartinK · January 26, 2016

Time on server is correct, "last connection" is not updating (in fact some random agent statuses are updating) only when this bug with large numbers happens. When I restart virtual machine with ERA all works normally for some time. For example in picture you can see Jan 25 09:00 with my normal average connection count then at some point it starts to grow and agent statuses are not updating. Today after restart ERA already 3 hours works normally.

When it stops working, please check /var/log/eset/RemoteAdministrator/Server/trace.log for Error entries. Those invalid average number should not be related to this problem. Also we have seen similar behavior in case unsupported MariaDB was used instead of MySQL database - what database/version are you using?

Please provide content of /etc/odbcinst.ini or at least check whether multi-threading is enabled for you MySQL driver -> it is parameter Threading=0 as described in ERA documentation: hxxp://help.eset.com/era_install/63/en-US/index.html?odbc_configuration.htm

vadim · January 26, 2016

There is no 'Threading' parameter (see below). I'll try to add and update.

[PostgreSQL]
Description        = ODBC for PostgreSQL
Driver        = /usr/lib/psqlodbc.so
Setup        = /usr/lib/libodbcpsqlS.so
Driver64        = /usr/lib64/psqlodbc.so
Setup64        = /usr/lib64/libodbcpsqlS.so
FileUsage        = 1

[MySQL ODBC 5.3 Unicode Driver]
Driver        = /usr/lib64/libmyodbc5w.so
UsageCount        = 1

[MySQL ODBC 5.3 ANSI Driver]
Driver        = /usr/lib64/libmyodbc5a.so
UsageCount        = 1

We are using MySQL (supplied with virtual appliance).

Edited January 26, 2016 by vadim

girts.u · January 26, 2016

Will try with config Threading=0 as my version of unixODBC is 2.2.14p2-5ubuntu5.

vadim · January 26, 2016

Statuses updating for an hour and a half. The issue seems to be fixed.

Thanks, MartinK!

!

girts.u · January 27, 2016

Problem still exists. In trace log every time it happens i found this

2016-01-26 18:29:12 Error: NetworkModule [Thread 7f71ff5fe700]: Error reported by JobScheduler[Name:Dns job scheduler for not network operation]. Error message is:resolve: Host not found (non-authoritative), try again later
2016-01-26 18:33:32 Warning: NetworkModule [Thread 7f7218dfa700]: The connection will be closed due to timeout. Resolved endpoint is NULL

vadim · January 27, 2016

MartinK,

It turns out my problem is not fixed too

Last status update is "Jan 26 20:59:51" (UTC) and I have the same MySQL error:

2016-01-26 21:00:59 Error: CDataMinersModule [Thread 7f7b7d1f4700]: CClientsDataMiner2: Failed with: [MySQL][ODBC 5.3(w) Driver][mysqld-5.6.28]Lock wait timeout exceeded; try restarting transaction (1205)

MartinK · January 27, 2016

Please attach content of /etc/odbcinst.ini to be sure it is correctly configured.

vadim · January 27, 2016

Please attach content of /etc/odbcinst.ini to be sure it is correctly configured.

[PostgreSQL]
Description        = ODBC for PostgreSQL
Driver        = /usr/lib/psqlodbc.so
Setup        = /usr/lib/libodbcpsqlS.so
Driver64        = /usr/lib64/psqlodbc.so
Setup64        = /usr/lib64/libodbcpsqlS.so
FileUsage        = 1
UsageCount        = 2

[MySQL ODBC 5.3 Unicode Driver]
Driver        = /usr/lib64/libmyodbc5w.so
UsageCount        = 2
Threading        = 0

[MySQL ODBC 5.3 ANSI Driver]
Driver        = /usr/lib64/libmyodbc5a.so
UsageCount        = 2
Threading        = 0

aquarium · January 27, 2016

I have the same problem, also now ERA not shows when task execution occured.

vadim · January 28, 2016

... not shows when task execution occured.

+1

I'm shocked.

vadim · January 28, 2016

MartinK,

It turns out my problem is not fixed too

Last status update is "Jan 26 20:59:51" (UTC) and I have the same MySQL error:
2016-01-26 21:00:59 Error: CDataMinersModule [Thread 7f7b7d1f4700]: CClientsDataMiner2: Failed with: [MySQL][ODBC 5.3(w) Driver][mysqld-5.6.28]Lock wait timeout exceeded; try restarting transaction (1205)

Today I have the same error in the same time:

2016-01-27 21:01:04 Error: CDataMinersModule [Thread 7f4a427e3700]: CClientsDataMiner2: Failed with: [MySQL][ODBC 5.3(w) Driver][mysqld-5.6.28]Lock wait timeout exceeded; try restarting transaction (1205)

ERA logs time in UTC(+0) and my local time is UTC+3, so maybe the error occurs every local midnight.

vadim · January 28, 2016

Restarting just mysqld doesn't fix statuses updating.

How can I restart ERA server / agent services on virtual appliance (without rebooting OS)?

vadim · January 28, 2016

How can I restart ERA server / agent services ...

initctl restart eraserver

...is fixing statuses.

Till midnight?..

MartinK · January 28, 2016

For those with this problem, could you please add innodb_lock_wait_timeout=600 to your MySQL configuration (/etc/my.cnf) into [mysqld] section and restart database server?

We think it is caused by database cleanups that are executed at midnight of local time and also just after startup in case SERVER was not running during midnight. Could you all confirm this?

aquarium · January 28, 2016

For those with this problem, could you please add innodb_lock_wait_timeout=600 to your MySQL configuration (/etc/my.cnf) into [mysqld] section and restart database server?

We think it is caused by database cleanups that are executed at midnight of local time and also just after startup in case SERVER was not running during midnight. Could you all confirm this?

After more than 4 hours, "last connection status" problem seems solved!!! (or appears)

@MartinK, can you please confirm my odbcinst.ini settings???

[PostgreSQL]
Description             = ODBC for PostgreSQL
Driver          = /usr/lib/psqlodbc.so
Setup           = /usr/lib/libodbcpsqlS.so
Driver64                = /usr/lib64/psqlodbc.so
Setup64         = /usr/lib64/libodbcpsqlS.so
FileUsage               = 1

[MySQL ODBC 5.3 Unicode Driver]
Driver          = /usr/lib64/libmyodbc5w.so
UsageCount              = 1
Threading = 0

[MySQL ODBC 5.3 ANSI Driver]
Driver          = /usr/lib64/libmyodbc5a.so
UsageCount              = 1
Threading = 0

Now remains only the problem about ERA not showing when task execution occured.

Thanks!

MartinK · January 28, 2016

After more than 4 hours, "last connection status" problem seems solved!!! (or appears)

As I mentioned in previous posts, critical will be midnight (= local time 00:00:00 on your SERVER machine), but we hope this configuration change will minimize chance it happens.

@MartinK, can you please confirm my odbcinst.ini settings???

Your ODBC configuration is now the same as in newer ERA Appliances and you may benefit from improved performance (parallel execution and query canceling).

aquarium · January 28, 2016

I hope, but previously the problem appeared only after 2 hours. I will waiting midnight .... Instead something about "task execution" problems?? This remain.

Edited January 28, 2016 by aquarium

MartinK · January 28, 2016

I hope, will waiting midnight .... Instead something about "task execution" problems?? This remain.

Seems client task executions time was not considered as essential and redesign was focused more on better "visualization" of task targets and status. Execution time is available in "History" view of client task, but unfortunately it requires more "mouse clicking" to get there.

Investigation whether it is "by design" or bug in currently underway.

girts.u · January 28, 2016

So what about my error? :huh: This topic is only getting answers for another issue.

Problem still exists. In trace log every time it happens i found this

2016-01-26 18:29:12 Error: NetworkModule [Thread 7f71ff5fe700]: Error reported by JobScheduler[Name:Dns job scheduler for not network operation]. Error message is:resolve: Host not found (non-authoritative), try again later
2016-01-26 18:33:32 Warning: NetworkModule [Thread 7f7218dfa700]: The connection will be closed due to timeout. Resolved endpoint is NULL

vadim · January 29, 2016

For those with this problem, could you please add innodb_lock_wait_timeout=600 to your MySQL configuration (/etc/my.cnf) into [mysqld] section and restart database server?

We think it is caused by database cleanups that are executed at midnight of local time and also just after startup in case SERVER was not running during midnight. Could you all confirm this?

This midnight passed without errors. Increasing lock wait timeout helps. Thanks a lot.

MartinK · January 29, 2016

So what about my error? This topic is only getting answers for another issue.

We have looked into it, and order of events was most probably as this:

Received connection -> Reverse DNS resolving failed (maybe DNS server was not available at that moment) -> Connection timeouted after 5 minutes -> Timeout caused invalid numbers in statistics (this is our bug we will fix in next release).

Sign In

Simultaneous connection count

Recommended Posts

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Recently Browsing 0 members

Quick search

FAQ

Topics