CNNS 0 Posted February 18, 2019 Share Posted February 18, 2019 Hi there, i got a ESET SMC Virtual Appliance that ran out of disc space. I managed to find the culprit but have no idea what caused this and how to fix it. Just clearing the directory and restarting the service does not seem to work. Please advice. Server version: 7.0.451.0 Agent Version: 7.0.451.0 [root@ESET-SMC ~]# df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/centos_ba--eraappl--v-root 41G 41G 28K 100% / devtmpfs 1.9G 0 1.9G 0% /dev tmpfs 1.9G 0 1.9G 0% /dev/shm tmpfs 1.9G 8.6M 1.9G 1% /run tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup /dev/sda1 497M 126M 372M 26% /boot /dev/mapper/centos_ba--eraappl--v-home 20G 33M 20G 1% /home tmpfs 380M 0 380M 0% /run/user/0 Total disc space used: [root@ESET-SMC ~]# du -hsx /var/cache/httpd/* | sort -rh | head -10 36G /var/cache/httpd/proxy 0 /var/cache/httpd/ssl Total number of files in Cache directory [root@ESET-SMC proxy]# cd /var/cache/httpd/proxy/ [root@ESET-SMC proxy]# find . -type f -print | wc -l 273262 Link to comment Share on other sites More sharing options...
ESET Staff MartinK 384 Posted February 18, 2019 ESET Staff Share Posted February 18, 2019 Indeed it seems that Apache HTTP proxy has taken all of the free space. When configuring appliance and proxy is enabled, service used to cleanup cache regularly should be enabled. It uses htcacheclean utility to clean cache directory. From my point of view it seems this service is not working or your proxy is heavily used and all those ~40GB of cached files were downloaded recently. I would recommend to check status of mentioned service: service htcacheclean status Any chance you enabled apache HTTP later, i.e. not during initial appliance configuration? It would explain why cleanups are not enabled. In order to resolve this issue, easiest would be to clean proxy cache directory. In case it won't help, we will need some trace logs from ESMC services, so that we can check reason why services are not running. It is possible that actually database (MySQL) has to be restarted as it might also stopped working due to insufficient disk space. Peter Randziak 1 Link to comment Share on other sites More sharing options...
CNNS 0 Posted February 19, 2019 Author Share Posted February 19, 2019 (edited) Hello MartinK, with the disk completely full the htcacheclean status didnt report anything. i removed some log files from /var/log/http to free up a little and the service is now running [root@ESET-SMC httpd]# service htcacheclean status Redirecting to /bin/systemctl status htcacheclean.service ● htcacheclean.service - Disk Cache Cleaning Daemon for Apache HTTP Server Loaded: loaded (/usr/lib/systemd/system/htcacheclean.service; static; vendor preset: disabled) Active: active (running) since Tue 2019-02-19 09:57:46 CET; 2min 25s ago Docs: man:htcacheclean(8) Process: 404 ExecStart=/usr/sbin/htcacheclean -P /run/httpd/htcacheclean/pid -d $INTERVAL -p $CACHE_ROOT -l $LIMIT $OPTIONS (code=exited, status=0/SUCCESS) Main PID: 405 (htcacheclean) CGroup: /system.slice/htcacheclean.service └─405 /usr/sbin/htcacheclean -P /run/httpd/htcacheclean/pid -d 60 -p /var/cache/httpd/proxy -l 10000M -i -n -t -L12000Feb 19 09:57:45 ESET-SMC systemd[1]: Starting Disk Cache Cleaning Daemon for Apache HTTP Server... Feb 19 09:57:46 ESET-SMC systemd[1]: Started Disk Cache Cleaning Daemon for Apache HTTP Server. After 10 Minutes of runtime there has been no additional space cleaned. Checking back after 4 hours there are now 35G free again. So this part worked. [root@ESET-SMC Server]# df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/centos_ba--eraappl--v-root 41G 5.1G 35G 13% / This Installation has been migrated from a Remote Administrator and iirc the http proxy was indeed enabled after initial setup. Are there any settings that i need to check? Trace logs seem to not have been written recently. [root@ESET-SMC Server]# ls -l total 300 -rw-r-----. 1 root root 31976 Feb 9 00:16 status.html -rw-r-----. 1 root root 256 Feb 12 13:39 trace.log -rw-r-----. 1 root root 73728 Feb 8 21:22 trace.log.0 -rw-r-----. 1 root root 192512 Feb 8 17:22 trace.log.1 Are there any fruther steps i have to do or configurations to check for the future? Edited February 19, 2019 by CNNS typos, tracelog, update Link to comment Share on other sites More sharing options...
ESET Staff MartinK 384 Posted February 19, 2019 ESET Staff Share Posted February 19, 2019 Not sure what went wrong but I would expect that this service is started regularly and automatic, so that cache content is cleaned. There are more details of this cleaning mechanism described in: https://www.digitalocean.com/community/tutorials/how-to-configure-apache-content-caching-on-centos-7 What is crucial is that mentioned htcacheclean service has to be in enabled state, i.e. enabled using command systemctl enable htcacheclean Link to comment Share on other sites More sharing options...
Recommended Posts