ERA 6 Proxy: The maximum number of open file descriptors is reached

[BDeep 7]

I recently had to deploy a new ERA 6 Proxy for our main US HQ. Since the new deployment, we've been onboarding clients left and right. I am seeing this on the ERA Proxy in ERA web console: "The maximum number of open file descriptors is reached. "

 

In VMware, I see this picture:

 

Not sure if this would fix it, but a 2006 thread from a Google search found this:

hxxp://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open-files/

 

Is this something I should try?

 

Should I spin up a second proxy at the HQ site? We're only talking 400 clients and 500 servers. Replication interval 10 minutes. I can do "DNS round robin" to "load-balance" the connections.

[MartinK 378]

Total limit of file descriptors should be sufficient even for much larger networks. There is most probably something wrong, but hard to tell where exactly without further analysis. What version of appliance and ERA products are you using?

I would suggest to install lsof tool (yum install lsof) and try to find out which process is taking so many file descriptors.

 

For example, command:

lsof | wc -l

should print total number of opened file descriptors.

 

Alternatively you could print descriptors for specific process by:

lsof -p `pgrep ERAProxy`
lsof -p `pgrep ERAProxy` | wc -l

should print list and count of file descriptors being held by ERAProxy process. This can be repeated for ERAAgent and httpd processes, until we know where the problem is.

 

You may also analyze number of opened connections using:

netstat -tp

which will print list of all opened connection with process that is responsible for them.

[BDeep 7]

The issue I was seeing on the main ERA server, which was resolved with ESET support with the specifics from this thread: https://forum.eset.com/topic/7248-client-last-connection-information-not-updated/is also appearing on the ERA Proxy server with the MYSQL DB errors showing in trace.log:

 

2015-12-23 19:51:54 Error: CDatabaseModule [Thread 7fe9f17fb700]: Database connection down. Exception:[unixODBC][MySQL][ODBC 5.3(w) Driver]Can't connect to MySQL server on '127.0.0.1' (111) (2003)
2015-12-23 19:51:54 Error: CReplicationModule [Thread 7fe9f17fb700]: CReplicationManager: Failed to initialize replication session with error: [unixODBC][MySQL][ODBC 5.3(w) Driver]Can't connect to MySQL server on '127.0.0.1' (111) (2003)

 

Unfortunately I cannot troubleshoot at this point. Building, a third time, the ERA Proxy for this site. :-(