High CPU usage on ERA Virtual Appliance

[mogobjah 0]

For some reason, our customer's ERA VA is showing high CPU usage. The VA is configured with 4 virtual CPUs and on average, the CPU consumption shows between 60-100%. But at times, i  notice the usage can go up to 386%, which lasts for a few seconds (Refer screenshot). This is caused by a single process, and triggered by the command "su" (Refer screenshot). Any idea what this means, and is it normal to have this usage pattern?

Very high CPU usage

Average CPU usage

[MartinK 378]

Could you please ask customer to provide more information of this problematic process? For example process tree (as can be enabled in htop using F5 key) would help us to diagnose what is the cause of this. All processes used by ERA seems to be working correctly and without extensive CPU usage. It is quite surprising that "su" is using CPU as it is simple helper just to execute other scripts or programs...also it seems that this process is long-running, which is even more suspicious.

[mogobjah 0]

Hi MartinK, as requested.

The memory usage also keeps increasing over time since last reboot (yesterday). Fyi, the no of endpoints currently managed by the server is around 100.

[MartinK 378]

To be honest I hoped that there will be visible script/process that is executed in "su" context, but this does not makes sense to me. There are also other processes (route, ps, ls) that seems to be detached from their parent.

Any chance customer made some custom changes in VA configuration? Scheduling own tasks or using WebMin for configuration? It is also possible that those command are executed through SSH sessions - any chance customer is using some outside connections to this VA?

Just to be sure, but have they tried to reboot it? If so, does this issue with strange "su" is there just after startup? I don't think "su" is spawned by ERA itself, as it is already running with root permissions -> no need to use su.

As a workaround, I would recommend to check what happens when this problematic process is killed - but please ask them to create VM snapshot just to be sure nothing goes wrong.

Edited February 13, 2018 by MartinK

[mogobjah 0]

Any chance customer made some custom changes in VA configuration? No. The VA was setup based on the steps provided in the official guide. No additional configuration was applied to the VA.

 Scheduling own tasks or using WebMin for configuration? WebMin is currently enabled.

any chance customer is using some outside connections to this VA? This may be possible, but im not exactly sure which command was executed. Currently the server is accessible from outside because they have some PCs in remote offices that they want to manage from the ERA console. For the external agents to communicate back to the server, i had requested the customer to open port 2222 in the firewall (at the time of deployment), but today found out that additionally, port 443 and 22 have been open as well all this while. Since SSH and HTTPS are not really required i have instructed them to close these two ports immediately.

but have they tried to reboot it? If so, does this issue with strange "su" is there just after startup? We tried rebooting the server again today. After reboot, the process with "su" command went missing, but strangely another process showed up with command "route -n" with similar (high) CPU usage pattern.

As a workaround, I would recommend to check what happens when this problematic process is killed. The customer has a policy of backing up VMs every month, so as suggested, i tried killing the process. Observed the server for a few hours and noticed that the process with high CPU usage is not recreated, but the memory usage keep increasing slowly. I will monitor this further and share the outcome here.