ESET SMC Appliance disc full with proxy cache files

[CNNS 0]

Hi there,

i got a ESET SMC Virtual Appliance that ran out of disc space.

I managed to find the culprit but have no idea what caused this and how to fix it. Just clearing the directory and restarting the service does not seem to work. Please advice.

 

Server version: 7.0.451.0

Agent Version: 7.0.451.0

 

[root@ESET-SMC ~]# df -h

Filesystem                              Size  Used Avail Use% Mounted on

/dev/mapper/centos_ba--eraappl--v-root   41G   41G   28K 100% /

devtmpfs                                1.9G     0  1.9G   0% /dev

tmpfs                                   1.9G     0  1.9G   0% /dev/shm

tmpfs                                   1.9G  8.6M  1.9G   1% /run

tmpfs                                   1.9G     0  1.9G   0% /sys/fs/cgroup

/dev/sda1                               497M  126M  372M  26% /boot

/dev/mapper/centos_ba--eraappl--v-home   20G   33M   20G   1% /home

tmpfs                                   380M     0  380M   0% /run/user/0

 

Total disc space used:

[root@ESET-SMC ~]# du -hsx /var/cache/httpd/* | sort -rh | head -10

36G     /var/cache/httpd/proxy

0       /var/cache/httpd/ssl

 

Total number of files in Cache directory

[root@ESET-SMC proxy]# cd /var/cache/httpd/proxy/

[root@ESET-SMC proxy]# find . -type f -print | wc -l

273262

 

[MartinK 378]

Indeed it seems that Apache HTTP proxy has taken all of the free space. When configuring appliance and proxy is enabled, service used to cleanup cache regularly should be enabled. It uses htcacheclean utility to clean cache directory. From my point of view it seems this service is not working or your proxy is heavily used and all those ~40GB of cached files were downloaded recently.

I would recommend to check status of mentioned service:

service htcacheclean status

 

Any chance you enabled apache HTTP later, i.e. not during initial appliance configuration? It would explain why cleanups are not enabled.

In order to resolve this issue, easiest would be to clean proxy cache directory. In case it won't help, we will need some trace logs from ESMC services, so that we can check reason why services are not running. It is possible that actually database (MySQL) has to be restarted as it might also stopped working due to insufficient disk space.

[CNNS 0]

Hello MartinK,

with the disk completely full the htcacheclean status didnt report anything. i removed some log files from /var/log/http to free up a little and the service is now running

[root@ESET-SMC httpd]# service htcacheclean status
Redirecting to /bin/systemctl status htcacheclean.service
● htcacheclean.service - Disk Cache Cleaning Daemon for Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/htcacheclean.service; static; vendor preset: disabled)
   Active: active (running) since Tue 2019-02-19 09:57:46 CET; 2min 25s ago
     Docs: man:htcacheclean(8)
  Process: 404 ExecStart=/usr/sbin/htcacheclean -P /run/httpd/htcacheclean/pid -d $INTERVAL -p $CACHE_ROOT -l $LIMIT $OPTIONS (code=exited, status=0/SUCCESS)
 Main PID: 405 (htcacheclean)
   CGroup: /system.slice/htcacheclean.service
           └─405 /usr/sbin/htcacheclean -P /run/httpd/htcacheclean/pid -d 60 -p /var/cache/httpd/proxy -l 10000M -i -n -t -L12000

Feb 19 09:57:45 ESET-SMC systemd[1]: Starting Disk Cache Cleaning Daemon for Apache HTTP Server...
Feb 19 09:57:46 ESET-SMC systemd[1]: Started Disk Cache Cleaning Daemon for Apache HTTP Server.
 

After 10 Minutes of runtime there has been no additional space cleaned.

Checking back after 4 hours there are now 35G free again. So this part worked.

[root@ESET-SMC Server]# df -h
Filesystem                              Size  Used Avail Use% Mounted on
/dev/mapper/centos_ba--eraappl--v-root   41G  5.1G   35G  13% /
 

This Installation has been migrated from a Remote Administrator and iirc the http proxy was indeed enabled after initial setup. Are there any settings that i need to check?

Trace logs seem to not have been written recently.

[root@ESET-SMC Server]# ls -l
total 300
-rw-r-----. 1 root root  31976 Feb  9 00:16 status.html
-rw-r-----. 1 root root    256 Feb 12 13:39 trace.log
-rw-r-----. 1 root root  73728 Feb  8 21:22 trace.log.0
-rw-r-----. 1 root root 192512 Feb  8 17:22 trace.log.1

Are there any fruther steps i have to do or configurations to check for the future?

Edited February 19, 2019 by CNNS
typos, tracelog, update

[MartinK 378]

Not sure what went wrong but I would expect that this service is started regularly and automatic, so that cache content is cleaned. There are more details of this cleaning mechanism described in: https://www.digitalocean.com/community/tutorials/how-to-configure-apache-content-caching-on-centos-7

What is crucial is that mentioned htcacheclean service has to be in enabled state, i.e. enabled using command systemctl enable htcacheclean