[Help/Advice] Full 100% CPU load problems with Tomcat7 and login webpage very slow

[Adhara-CS 1]

Hi all,

 

We are deploying the ERA 6 package on LInux (Debian 7).

The ERA server is working fine (it is installed and runs without errors), to start we would like to install the webconsole on the same server (this will be split later on).

 

As we are talking about security, we wanted to have the webconsole running over HTTPS, therefore this is what we did for the web console:

 

Downloading the era JAVA package:

wget hxxp://download.eset.com/download/ra/v6/standalone-installers/webconsole/era.war

Setup the Java packages:

sudo apt-get install openjdk-6-jdk

Setup the Apache Tomcat web server (version 6 and newer):

sudo apt-get install tomcat7

Copy the era.war file into the tomcat application folder:

sudo cp era.war /var/lib/tomcat7/webapps/

Restart the tomcat service to deploy era java file (autodeploy is active):

sudo /etc/init.d/tomcat7 restart

Install the library libtcnative (used by tomcat APR for the SSL/TLS):

apt-get install libtcnative-1

Then, changed following lines in /etc/tomcat7/server.xml:

<Connector port="8443" protocol="org.apache.coyote.http11.Http11AprProtocol"
          maxThreads="200"
          enableLookups="true" disableUploadTimeout="true"
          acceptCount="100" scheme="https" secure="true"
          SSLEnabled="true"
          SSLProtocol="TLSv1"
          SSLCertificateFile="OURCRT.crt"
          SSLCertificateKeyFile="OURKEY.key"
          SSLCertificateChainFile="OURPEM.pem"/>

Uncommenting following line to enable the SSLEngine:

<Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on" />

Apply changes, restarting tomcat with:

sudo /etc/init.d/tomcat7 restart

Here the webconsole works over SSL/TLS on port 8443 !

 

As we want to use the standard 443 port for https, we did create an IPTABLES rule to redirect port 443 to 8443:

iptables -t nat -A PREROUTING -p tcp --dport 443 -j REDIRECT --to-port 8443

As we want to make the manager as default web app of the server, we need to add following lines to file /etc/tomcat7/server.xml inside the <Host> section:

<Context path="" docBase="era">
    <WatchedResource>WEB-INF/web.xml</WatchedResource>
</Context>

Apply changes, restarting tomcat with:

sudo /etc/init.d/tomcat7 restart

As IPTABLE rules are not persistent and will be deleted when server restarts, we need to install a program that will allow us to save them and make them persistent.
Install iptables-persistent:

sudo apt-get install iptables-persistent

It will ask you if you want to save actual rules, select "Yes" twice and go on. This program will have created the directory /etc/iptables/ where we will store the saved rules.
 

Right here the webconsole is working, answering to web requests on https://OURIP/(that redirects to /webconsole automatically).

The login screens appears (well at least the graphical parts) but the little loading GIF turns for minutes (we usually need to wait 15 to 20 minutes) before showing the fields and buttons !

After the long wait the login screen appears and works fine (the CPU load goes back to nearly nothing) and the webconsole works as it should.

 

Looking to the logs, there seems to be no errors or warnings (not in tomcat catalina, not in eset logs), but CPU load goes up to 100%(long live Java) on one of the cores of the VM...!

The VM on which this is all running has a dual core 2,8GHz CPU with 4GB of RAM, which seems to fit the needs (according to documentation).

 

 

We are really stuck here has we have no idea where the problem comes from... It can come from the era webconsole code as from the tomcat server but we really have no clue !

 

 

If anyone can help us out, any idea is welcome.

[Peter Randziak 944]

Hello,

 

can you please try following:

 

1. Update Java to the latest version

2. Assign more RAM for Java, we found on other forums that high PCU consumption was caused by lack of memory.

3. change the port for connector from 8443 to to save resources for port redirection.

4. try to disable autodeploy and do it manually

5. do you have issues with http traffic as well or just with https?

 

To diagnose the source of the issue (Tomcat or our webconsole app) you may try to deploy it to other servlet container like GlassFish or Jetty.

 

 

If you are not able to troubleshoot it please enable our webserver logs by settings logs_path it <Tomcat>\webapps\era\WEB-INF\classes\sk\eset\era\g2webconsole\server\modules\config\EraWebServerConfig.properties

and send me a private message with them, with reference to this forum topic.

 

Please let us know how it went.

 

P.R.

[Adhara-CS 1]

Hi,

 

We will debug the situation with the points you described and keep you informed over here in this topic.

If needed we will send you the logs in Private Message.

 

 

Thank you,

[Adhara-CS 1]

Hi,

 

We did try following:

 

1. Update Java to the latest version

This didn't affect the CPU or RAM consumption.

 

2. Assign more RAM for Java, we found on other forums that high PCU consumption was caused by lack of memory.

This didn't affect the CPU consumption neither

 

3. change the port for connector from 8443 to to save resources for port redirection.

The CPU consumption for the redirection is so low that it didn't change either

 

4. try to disable autodeploy and do it manually

We couldn't do this as the concerned ERA is allready in production

 

5. do you have issues with http traffic as well or just with https?

As we are working on security, there is just no HTTP at all ! (disabled in tomcat)

 

 

But we found out that regarding the place from which we did connect (and the available bandwith) it did impact the behavior.

• Whith a good bandwith: The page loads in something as 10 to 15 seconds and CPU load doesn't go at 100% or just for the few last seconds
• Whith a bad bandwith: The page takes 1 to 3 minutes to load and CPU consumption goes to 100% (over several cores) after the 10 first seconds (just like a memory leak but with CPU).

 

We don't get why but the longer it takes for the client's side (browser) to load the app, how more CPU is consumes on server...