Hi,
One of my ClearOS servers suddenly started generating hundreds of messages like this one:
Low memory; process clamd (65270) killed
Could this be some form of attack or is it something that has upset CLAMAV? I have restarted the server and am watching the processes closely to see if it starts grabbing loads of memory again.
In processes it is listed as this:
Running CPU Memory Command Action
00:00:13 1.0 13.8 clamd Kill
So its RAM usage is higher than most other daemons but seems to have settled at 13.8 (I assume that is MB?) since I restarted the server.
Any advice or other things I should check would be appreciated.
Siv
One of my ClearOS servers suddenly started generating hundreds of messages like this one:
Low memory; process clamd (65270) killed
Could this be some form of attack or is it something that has upset CLAMAV? I have restarted the server and am watching the processes closely to see if it starts grabbing loads of memory again.
In processes it is listed as this:
Running CPU Memory Command Action
00:00:13 1.0 13.8 clamd Kill
So its RAM usage is higher than most other daemons but seems to have settled at 13.8 (I assume that is MB?) since I restarted the server.
Any advice or other things I should check would be appreciated.
Siv
Location [ View Larger Map ]
Share this post:
Accepted Answer
Your log looks awfully like that in this account...
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=824042
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=824042
Responses (18)
-
Accepted Answer
Some basic suggestions...
1) Don't be too fixated on just watching clamd - you haven't provided any proof that it was the culprit for the low memory and not the victim of another runaway process.
2) Would suggest recording quite a few memory usage statistics now while everything is OK - so you have something to compare against when/if the problem reappears... Some of the simpler ones are "free -m" "cat /proc/meminfo" "vmstat -s" "top -b -n 1 | head -n 20"
3) Contact Mr Google for help in this area - https://www.garron.me/en/go2linux/how-find-which-process-eating-ram-memory-linux.html looks interesting - there is much more information available. Spend a little time researching...
4) You have provided no basic system information... How much memory/swap installed and in use eg "free -m" would give us a clue...
5) You wrote "Could this be some form of attack" - what are you doing to monitor whether this is a/the problem?
6) No mention of searching the logs for any clues
I'm sure you could add to the list yourself - maybe you have a similar one already... -
Accepted Answer
Tony,
Because I am more of a Windows guy (from a time served perspective) I naturally assume I don't understand what is going on with Linux (I am getting better, but the curve is steep). So I tend to ask you guys who have clearly been around Linux a lot longer than me where I should be looking. I appreciate I could do all the Googling but when you are very new to a subject there is a danger that following what is out there may not be the wisest thing to do, as quite a lot of what I find out there proves ultimately not to be the best thing to do. Often wiser heads will say don't do that look at this first before assuming that it's X rather than Y.
System is a HP Proliant Gen 8 with 4GB RAM and a Celeron Dual Core Processor. It has run solidly for around 3 months without any issues. Disk is 1TB and plenty of free space.
It is very late and I need some kip, so will run some of your commands tomorrow and post the fimdings back here when I have them.
Siv -
Accepted Answer
Sleep is very good for clearing the head and coming back to the problem with a fresh approach... Sweet dreams
Appreciate and agree with the sentiment while being relatively new to not make system changes without understanding the ramifications - but that is not what was suggested here - it was *monitoring* the system and collecting data to compare against the data if/when the problem returns. All I would be interested in at the moment would be "free -m". The rest of the commands were suggestions for your benefit for you to record data for subsequent evaluation for any changes against the results for the same commands when the problem occurs. The idea was for you to augment that list... you will learn by research.
"So I tend to ask you guys who have clearly been around Linux a lot longer than me " - that applies to an awful lot of people outside the ClearOS community as well. You would be doing yourself a favour by finding the type of places where the real experts hang out and thus have another reliable source for help for 'generic' Linux problems. More arrows in your quiver, not just the one marked ClearOS . ClamAV is not a ClearOS product and is widely used elsewhere.
By the way, Linux is not my speciality either - it is (or 'was' as now retired) large IBM hardware installation/maintenance and some IBM mainframe software such as MVS VSE VM/CMS DB2 REXX ISPF APL etc -
Accepted Answer
Since I restarted the server the ClamD seems to have settled down and it grabbing the most memory of all the daemons but at 14.7MB (i assume the values in the processes part of the dashboard are in MB?) is not overly high. Running free -m gives this:
free -m
total used free shared buff/cache available
Mem: 3783 1400 888 57 1494 1999
Swap: 3839 0 3839
The machine has 4GB physical RAM but is only used as a home server and test bed for me so it doesn't need to be highly specced.
When I was getting the warnings the clamd process was grabbing 10 times that and it appeared the system was closing it as it was grabbing too much memory and then it was restarting it and within a few seconnds it was getting very high and then being terminated again. This was the behaviour that made me wonder if my system was being attacked somehow.
These are top memory grabbing processes at the moment:
00:02:36 0.0 14.4 clamd
00:03:30 0.0 4.7 mysqld
00:00:01 0.0 2.6 /usr/sbin/amavi
00:00:00 0.0 2.6 /usr/sbin/amavi
00:00:00 0.0 2.6 /usr/sbin/amavi
00:00:00 0.0 2.6 /usr/sbin/amavi
00:00:00 0.0 2.6 /usr/sbin/amavi
00:00:00 0.0 2.6 /usr/sbin/amavi
00:00:00 0.0 2.6 /usr/sbin/amavi
00:00:00 0.0 2.6 /usr/sbin/amavi
00:00:00 0.0 2.6 /usr/sbin/amavi
00:03:10 0.0 2.2 mysqld
00:00:04 0.0 2.1 snort
00:00:04 0.0 1.9 slapd
00:00:00 0.0 0.6 webconfig
00:00:01 0.0 0.6 webconfig
00:00:00 0.0 0.5 webconfig
00:00:00 0.1 0.5 webconfig
Siv -
Accepted Answer
Running cat /proc/meminfo gives this:
cat /proc/meminfo
MemTotal: 3874300 kB
MemFree: 895296 kB
MemAvailable: 2037744 kB
Buffers: 2132 kB
Cached: 1151416 kB
SwapCached: 0 kB
Active: 2148820 kB
Inactive: 264952 kB
Active(anon): 1290088 kB
Inactive(anon): 28632 kB
Active(file): 858732 kB
Inactive(file): 236320 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 3932156 kB
SwapFree: 3932156 kB
Dirty: 4 kB
Writeback: 0 kB
AnonPages: 1260072 kB
Mapped: 73492 kB
Shmem: 58496 kB
Slab: 380816 kB
SReclaimable: 337068 kB
SUnreclaim: 43748 kB
KernelStack: 5808 kB
PageTables: 36484 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 5869304 kB
Committed_AS: 4513952 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 103540 kB
VmallocChunk: 34359523324 kB
HardwareCorrupted: 0 kB
AnonHugePages: 886784 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 98188 kB
DirectMap2M: 3930112 kB
Running vmstat -s gives this:
3874300 K total memory
1445308 K used memory
2148964 K active memory
264940 K inactive memory
894536 K free memory
2132 K buffer memory
1532324 K swap cache
3932156 K total swap
0 K used swap
3932156 K free swap
322679 non-nice user cpu ticks
72 nice user cpu ticks
351066 system cpu ticks
63211598 idle cpu ticks
64311 IO-wait cpu ticks
0 IRQ cpu ticks
1201 softirq cpu ticks
0 stolen cpu ticks
969285 pages paged in
5549424 pages paged out
0 pages swapped in
0 pages swapped out
416227125 interrupts
387846469 CPU context switches
1521755983 boot time
825283 forks
top -b -n 1 | head -n 20 gives this:
top - 16:32:54 up 3 days, 17:33, 2 users, load average: 0.00, 0.01, 0.05
Tasks: 184 total, 1 running, 183 sleeping, 0 stopped, 0 zombie
%Cpu(s): 3.0 us, 3.0 sy, 0.0 ni, 90.9 id, 3.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 3874300 total, 890672 free, 1448996 used, 1534632 buff/cache
KiB Swap: 3932156 total, 3932156 free, 0 used. 2033312 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1 root 20 0 272796 3976 2516 S 0.0 0.1 1:58.26 systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:00.06 kthreadd
3 root 20 0 0 0 0 S 0.0 0.0 0:03.88 ksoftirqd/0
5 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:0H
7 root rt 0 0 0 0 S 0.0 0.0 0:01.80 migration/0
8 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcu_bh
9 root 20 0 0 0 0 S 0.0 0.0 21:46.15 rcu_sched
10 root rt 0 0 0 0 S 0.0 0.0 0:01.01 watchdog/0
11 root rt 0 0 0 0 S 0.0 0.0 0:01.03 watchdog/1
12 root rt 0 0 0 0 S 0.0 0.0 0:01.04 migration/1
13 root 20 0 0 0 0 S 0.0 0.0 0:02.30 ksoftirqd/1
15 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/1:0H
17 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kdevtmpfs
If anyone can see any issues in this information please let me know!
Siv -
Accepted Answer
Hi Siv
but at 14.7MB (i assume the values in the processes part of the dashboard are in MB?) is not overly high
This embodies why GUIs are so restrictive - no real idea what you are looking at - and in the this case you don't even have a heading so you know ***NOT*** what the units are
Suspect that Webconfig is using something like the following command to get the process information
# ps -eo pid,user,time,%cpu,%mem,sz,tty,ucomm,command
If we use that and parse out just the clamd line(s) we get (with headings
[root@may ~]# ps -eo pid,user,time,%cpu,%mem,sz,tty,ucomm,command | egrep '%|clam' | egrep -v 'grep|ps'
PID USER TIME %CPU %MEM SZ TT COMMAND COMMAND
4286 clam 00:01:35 0.0 7.3 194427 ? clamd /usr/sbin/clamd
Is that similar to what you see in Webconfig on your system? - this was taken from an up to date ClearOS 7.4 box... (Webconfig is only used here as a 'last resort')
If you want more information on "ps" then reference the 'man' pages or use the 'web
Now to discuss attacks...
The primary use of clamav is to detect Windows viruses and malware with its built-in signatures. So if it was deployed 'in anger' with hundreds of instances it could have been triggered by one or more of the following?
1) Enormous number of incoming mail items
2) Enormous number of outgoing mail items
3) Same for downloads or uploads
4) A process generating a lot of files that clamav was triggered to scan
5) ????
Some of this type of activity could be detected in the logs.... did you check?
As for multiple instances of clamd. They should at least share the executable, and probably most of the data areas too, so they should not be a memory burden, as long as the number is 'reasonable'
It wouldn't hurt, while the system is OK, just to keep a copy of the output of
# ps -eo pid,user,time,%cpu,%mem,sz,tty,ucomm,command -
Accepted Answer
Tony,
I think GUI's are OK as long as the GUI designer thinks about the design and who is going to use it.
Running "ps -eo pid,user,time,%cpu,%mem,sz,tty,ucomm,command" I get this: (more text below)
PID USER TIME %CPU %MEM SZ TT COMMAND COMMAND
1 root 00:02:19 0.0 0.1 68199 ? systemd /usr/lib/system
2 root 00:00:00 0.0 0.0 0 ? kthreadd [kthreadd]
3 root 00:00:04 0.0 0.0 0 ? ksoftirqd/0 [ksoftirqd/0]
5 root 00:00:00 0.0 0.0 0 ? kworker/0:0H [kworker/0:0H]
7 root 00:00:01 0.0 0.0 0 ? migration/0 [migration/0]
8 root 00:00:00 0.0 0.0 0 ? rcu_bh [rcu_bh]
9 root 00:25:33 0.4 0.0 0 ? rcu_sched [rcu_sched]
10 root 00:00:01 0.0 0.0 0 ? watchdog/0 [watchdog/0]
11 root 00:00:01 0.0 0.0 0 ? watchdog/1 [watchdog/1]
12 root 00:00:01 0.0 0.0 0 ? migration/1 [migration/1]
13 root 00:00:02 0.0 0.0 0 ? ksoftirqd/1 [ksoftirqd/1]
15 root 00:00:00 0.0 0.0 0 ? kworker/1:0H [kworker/1:0H]
17 root 00:00:00 0.0 0.0 0 ? kdevtmpfs [kdevtmpfs]
18 root 00:00:00 0.0 0.0 0 ? netns [netns]
19 root 00:00:00 0.0 0.0 0 ? khungtaskd [khungtaskd]
20 root 00:00:00 0.0 0.0 0 ? writeback [writeback]
21 root 00:00:00 0.0 0.0 0 ? kintegrityd [kintegrityd]
22 root 00:00:00 0.0 0.0 0 ? bioset [bioset]
23 root 00:00:00 0.0 0.0 0 ? kblockd [kblockd]
24 root 00:00:00 0.0 0.0 0 ? md [md]
31 root 00:00:00 0.0 0.0 0 ? kswapd0 [kswapd0]
32 root 00:00:00 0.0 0.0 0 ? ksmd [ksmd]
33 root 00:00:04 0.0 0.0 0 ? khugepaged [khugepaged]
34 root 00:00:00 0.0 0.0 0 ? crypto [crypto]
42 root 00:00:00 0.0 0.0 0 ? kthrotld [kthrotld]
45 root 00:00:00 0.0 0.0 0 ? kmpath_rdacd [kmpath_rdacd]
46 root 00:00:00 0.0 0.0 0 ? kpsmoused [kpsmoused]
48 root 00:00:00 0.0 0.0 0 ? ipv6_addrconf [ipv6_addrconf]
67 root 00:00:00 0.0 0.0 0 ? deferwq [deferwq]
100 root 00:00:08 0.0 0.0 0 ? kauditd [kauditd]
287 root 00:00:00 0.0 0.0 0 ? ata_sff [ata_sff]
295 root 00:00:00 0.0 0.0 0 ? ttm_swap [ttm_swap]
296 root 00:00:00 0.0 0.0 0 ? scsi_eh_0 [scsi_eh_0]
297 root 00:00:00 0.0 0.0 0 ? scsi_tmf_0 [scsi_tmf_0]
298 root 00:00:00 0.0 0.0 0 ? scsi_eh_1 [scsi_eh_1]
299 root 00:00:00 0.0 0.0 0 ? scsi_tmf_1 [scsi_tmf_1]
300 root 00:00:00 0.0 0.0 0 ? scsi_eh_2 [scsi_eh_2]
301 root 00:00:00 0.0 0.0 0 ? scsi_tmf_2 [scsi_tmf_2]
302 root 00:00:00 0.0 0.0 0 ? scsi_eh_3 [scsi_eh_3]
303 root 00:00:00 0.0 0.0 0 ? scsi_tmf_3 [scsi_tmf_3]
304 root 00:00:00 0.0 0.0 0 ? scsi_eh_4 [scsi_eh_4]
305 root 00:00:00 0.0 0.0 0 ? scsi_tmf_4 [scsi_tmf_4]
306 root 00:00:00 0.0 0.0 0 ? scsi_eh_5 [scsi_eh_5]
307 root 00:00:00 0.0 0.0 0 ? scsi_tmf_5 [scsi_tmf_5]
319 root 00:00:00 0.0 0.0 0 ? scsi_eh_6 [scsi_eh_6]
320 root 00:00:00 0.0 0.0 0 ? scsi_tmf_6 [scsi_tmf_6]
321 root 00:00:14 0.0 0.0 0 ? usb-storage [usb-storage]
328 root 00:00:01 0.0 0.0 0 ? kworker/0:1H [kworker/0:1H]
388 root 00:00:00 0.0 0.0 0 ? kdmflush [kdmflush]
389 root 00:00:00 0.0 0.0 0 ? bioset [bioset]
399 root 00:00:02 0.0 0.0 0 ? kworker/1:1H [kworker/1:1H]
400 root 00:00:00 0.0 0.0 0 ? kdmflush [kdmflush]
401 root 00:00:00 0.0 0.0 0 ? bioset [bioset]
414 root 00:00:00 0.0 0.0 0 ? bioset [bioset]
415 root 00:00:00 0.0 0.0 0 ? xfsalloc [xfsalloc]
416 root 00:00:00 0.0 0.0 0 ? xfs_mru_cache [xfs_mru_cache]
417 root 00:00:00 0.0 0.0 0 ? xfs-buf/dm-0 [xfs-buf/dm-0]
418 root 00:00:00 0.0 0.0 0 ? xfs-data/dm-0 [xfs-data/dm-0]
419 root 00:00:00 0.0 0.0 0 ? xfs-conv/dm-0 [xfs-conv/dm-0]
420 root 00:00:00 0.0 0.0 0 ? xfs-cil/dm-0 [xfs-cil/dm-0]
421 root 00:00:00 0.0 0.0 0 ? xfs-reclaim/dm- [xfs-reclaim/dm
422 root 00:00:00 0.0 0.0 0 ? xfs-log/dm-0 [xfs-log/dm-0]
423 root 00:00:00 0.0 0.0 0 ? xfs-eofblocks/d [xfs-eofblocks/
424 root 00:02:09 0.0 0.0 0 ? xfsaild/dm-0 [xfsaild/dm-0]
493 root 00:00:14 0.0 0.2 9311 ? systemd-journal /usr/lib/system
514 root 00:00:00 0.0 0.0 48145 ? lvmetad /usr/sbin/lvmet
517 root 00:00:00 0.0 0.0 11120 ? systemd-udevd /usr/lib/system
541 root 00:00:00 0.0 0.0 0 ? edac-poller [edac-poller]
544 root 00:00:00 0.0 0.0 0 ? kipmi0 [kipmi0]
581 root 00:00:00 0.0 0.0 0 ? xfs-buf/sda1 [xfs-buf/sda1]
582 root 00:00:00 0.0 0.0 0 ? xfs-data/sda1 [xfs-data/sda1]
583 root 00:00:00 0.0 0.0 0 ? xfs-conv/sda1 [xfs-conv/sda1]
584 root 00:00:00 0.0 0.0 0 ? xfs-cil/sda1 [xfs-cil/sda1]
585 root 00:00:00 0.0 0.0 0 ? xfs-reclaim/sda [xfs-reclaim/sd
586 root 00:00:00 0.0 0.0 0 ? xfs-log/sda1 [xfs-log/sda1]
587 root 00:00:00 0.0 0.0 0 ? xfs-eofblocks/s [xfs-eofblocks/
589 root 00:00:00 0.0 0.0 0 ? kvm-irqfd-clean [kvm-irqfd-clea
596 root 00:00:00 0.0 0.0 0 ? xfsaild/sda1 [xfsaild/sda1]
617 root 00:00:18 0.0 0.0 13863 ? auditd /sbin/auditd
642 root 00:00:24 0.0 0.0 5369 ? irqbalance /usr/sbin/irqba
643 polkitd 00:00:08 0.0 0.3 134059 ? polkitd /usr/lib/polkit
646 root 00:00:21 0.0 0.3 65834 ? rsyslogd /usr/sbin/rsysl
647 nscd 00:00:32 0.0 0.0 335550 ? nscd /usr/sbin/nscd
652 root 00:00:00 0.0 0.0 1085 ? acpid /usr/sbin/acpid
656 dbus 00:00:34 0.0 0.0 5712 ? dbus-daemon /bin/dbus-daemo
719 root 00:00:18 0.0 0.0 5592 ? systemd-logind /usr/lib/system
765 root 00:00:01 0.0 0.0 31088 ? crond /usr/sbin/crond
767 root 00:00:01 0.0 0.0 19752 ? saslauthd /usr/sbin/sasla
769 root 00:00:01 0.0 0.0 19109 ? saslauthd /usr/sbin/sasla
770 root 00:00:01 0.0 0.0 19109 ? saslauthd /usr/sbin/sasla
771 root 00:00:01 0.0 0.0 19109 ? saslauthd /usr/sbin/sasla
772 root 00:00:00 0.0 0.0 19109 ? saslauthd /usr/sbin/sasla
840 root 00:00:00 0.0 0.0 24735 ? login login -- clearc
845 chrony 00:00:00 0.0 0.0 23627 ? chronyd /usr/sbin/chron
952 root 00:00:00 0.0 0.1 26127 ? sshd /usr/sbin/sshd
954 root 00:00:00 0.0 0.0 1133 ? pptpd /usr/sbin/pptpd
955 root 00:00:15 0.0 0.4 129573 ? httpd /usr/sbin/httpd
956 nobody 00:00:03 0.0 0.0 3647 ? dnsmasq /usr/sbin/dnsma
971 root 00:00:36 0.0 0.4 140120 ? tuned /usr/bin/python
987 clearsy+ 00:08:32 0.1 0.1 182795 ? clearsyncd /usr/sbin/clear
989 suva 00:02:59 0.0 0.0 7040 ? suvad /usr/sbin/suvad
1067 root 00:00:25 0.0 0.0 89575 ? nmbd /usr/sbin/nmbd
1068 root 00:00:00 0.0 0.0 89515 ? nmbd /usr/sbin/nmbd
1074 clearco+ 00:00:00 0.0 0.0 28336 tty1 bash -bash
1099 root 00:00:06 0.0 0.1 35929 ? syswatch syswatch
1165 ldap 00:00:04 0.0 1.9 276707 ? slapd /usr/sbin/slapd
1330 nslcd 00:00:04 0.0 0.0 109167 ? nslcd /usr/sbin/nslcd
1341 root 00:00:02 0.0 0.0 22014 ? master /usr/libexec/po
1351 postfix 00:00:00 0.0 0.1 22031 ? qmgr qmgr -l -t fifo
1515 root 00:05:59 0.0 0.6 303935 ? fail2ban-server /usr/bin/python
1670 ftp 00:00:05 0.0 0.0 45000 ? proftpd proftpd: (accep
1794 mysql 00:00:00 0.0 0.0 28315 ? mysqld_safe /bin/sh /usr/bi
2005 system-+ 00:00:00 0.0 0.0 28316 ? mysqld_safe /bin/sh /usr/cl
2717 postfix 00:00:00 0.0 0.1 21990 ? tlsmgr tlsmgr -l -t un
2862 clearco+ 00:06:48 0.1 0.0 68366 tty1 tconsole /usr/sbin/tcons
3120 mysql 00:03:45 0.0 2.2 225927 ? mysqld /usr/libexec/my
3133 system-+ 00:04:08 0.0 4.9 367436 ? mysqld /usr/clearos/sa
3185 cyrus 00:00:12 0.0 0.1 53788 ? cyrus-master /usr/lib/cyrus-
3259 root 00:00:04 0.0 0.1 102485 ? winbindd /usr/sbin/winbi
3278 cyrus 00:00:00 0.0 0.0 29750 ? imapd imapd
3280 cyrus 00:00:00 0.0 0.0 29750 ? imapd imapd -s
3302 cyrus 00:00:00 0.0 0.0 29750 ? imapd imapd
3304 cyrus 00:00:00 0.0 0.0 29750 ? imapd imapd -s
3312 cyrus 00:00:00 0.0 0.0 29750 ? imapd imapd
3314 cyrus 00:00:00 0.0 0.0 29750 ? imapd imapd -s
3392 root 00:00:00 0.0 0.1 112517 ? smbd /usr/sbin/smbd
3393 root 00:00:00 0.0 0.0 112092 ? smbd-notifyd /usr/sbin/smbd
3394 root 00:00:00 0.0 0.0 112092 ? cleanupd /usr/sbin/smbd
3396 root 00:00:00 0.0 0.1 102540 ? winbindd /usr/sbin/winbi
3398 root 00:00:00 0.0 0.1 102592 ? winbindd /usr/sbin/winbi
3399 root 00:00:00 0.0 0.1 102485 ? winbindd /usr/sbin/winbi
3400 root 00:00:00 0.0 0.1 112519 ? lpqd /usr/sbin/smbd
3603 clam 00:03:02 0.0 14.4 198492 ? clamd /usr/sbin/clamd
5234 amavis 00:00:01 0.0 2.6 90467 ? /usr/sbin/amavi /usr/sbin/amavi
5240 amavis 00:00:00 0.0 2.6 90853 ? /usr/sbin/amavi /usr/sbin/amavi
5241 amavis 00:00:00 0.0 2.6 90853 ? /usr/sbin/amavi /usr/sbin/amavi
5242 amavis 00:00:00 0.0 2.6 90853 ? /usr/sbin/amavi /usr/sbin/amavi
5243 amavis 00:00:00 0.0 2.6 90853 ? /usr/sbin/amavi /usr/sbin/amavi
5244 amavis 00:00:00 0.0 2.6 90853 ? /usr/sbin/amavi /usr/sbin/amavi
5245 amavis 00:00:00 0.0 2.6 90853 ? /usr/sbin/amavi /usr/sbin/amavi
5246 amavis 00:00:00 0.0 2.6 90853 ? /usr/sbin/amavi /usr/sbin/amavi
5247 amavis 00:00:00 0.0 2.6 90853 ? /usr/sbin/amavi /usr/sbin/amavi
10077 cyrus 00:00:00 0.0 0.1 30414 ? imapd imapd -s
18902 cyrus 00:00:00 0.0 0.1 30410 ? imapd imapd -s
18903 cyrus 00:00:00 0.0 0.0 29750 ? imapd imapd -s
19198 root 00:00:00 0.0 0.0 0 ? kworker/1:0 [kworker/1:0]
19607 root 00:00:01 0.0 0.0 0 ? kworker/u128:1 [kworker/u128:1
20748 cyrus 00:00:00 0.0 0.0 29750 ? imapd imapd -s
21365 root 00:00:00 0.0 0.0 0 ? kworker/0:0 [kworker/0:0]
21423 apache 00:00:00 0.0 0.2 129606 ? httpd /usr/sbin/httpd
21424 apache 00:00:00 0.0 0.2 129607 ? httpd /usr/sbin/httpd
21425 apache 00:00:00 0.0 0.2 129607 ? httpd /usr/sbin/httpd
21426 apache 00:00:00 0.0 0.2 129606 ? httpd /usr/sbin/httpd
21427 apache 00:00:00 0.0 0.2 129606 ? httpd /usr/sbin/httpd
21488 snort 00:00:10 0.0 2.2 102813 ? snort snort -i eno1 -
21553 root 00:05:46 0.3 0.0 2104 ? snortsam snortsam /etc/s
21573 root 00:00:04 0.0 0.4 144502 ? webconfig /usr/sbin/webco
21576 webconf+ 00:00:00 0.0 0.4 146038 ? webconfig /usr/sbin/webco
21577 webconf+ 00:00:00 0.0 0.6 148809 ? webconfig /usr/sbin/webco
21579 webconf+ 00:00:01 0.0 0.6 148774 ? webconfig /usr/sbin/webco
21808 root 00:00:00 0.0 0.0 0 ? kworker/1:2 [kworker/1:2]
21973 postfix 00:00:00 0.0 0.0 21987 ? pickup pickup -l -t fi
22150 apache 00:00:00 0.0 0.2 129606 ? httpd /usr/sbin/httpd
22152 apache 00:00:00 0.0 0.2 129606 ? httpd /usr/sbin/httpd
22598 root 00:00:00 0.0 0.0 0 ? kworker/0:2 [kworker/0:2]
23165 root 00:00:00 0.0 0.0 0 ? kworker/0:1 [kworker/0:1]
23510 cyrus 00:00:00 0.0 0.0 29750 ? imapd imapd -s
23621 root 00:00:00 0.0 0.0 0 ? kworker/0:3 [kworker/0:3]
23670 root 00:00:00 0.0 0.0 1661 ? pptpctrl pptpd [109.170.
23671 root 00:00:00 0.0 0.0 28141 ? pppd /usr/sbin/pppd
23710 root 00:00:00 0.1 0.1 38258 ? sshd sshd: root@pts/
23724 root 00:00:00 0.0 0.0 28370 pts/1 bash -bash
23754 root 00:00:00 0.0 0.0 37288 pts/1 ps ps -eo pid,user
32693 apache 00:00:00 0.0 0.2 129607 ? httpd /usr/sbin/httpd
42952 webconf+ 00:00:00 0.0 0.5 146539 ? webconfig /usr/sbin/webco
42953 webconf+ 00:00:00 0.0 0.5 146788 ? webconfig /usr/sbin/webco
42954 webconf+ 00:00:00 0.0 0.3 144858 ? webconfig /usr/sbin/webco
42955 webconf+ 00:00:00 0.0 0.3 144858 ? webconfig /usr/sbin/webco
43304 webconf+ 00:00:00 0.0 0.2 144537 ? webconfig /usr/sbin/webco
51128 apache 00:00:00 0.0 0.2 129606 ? httpd /usr/sbin/httpd
52147 webconf+ 00:00:00 0.0 0.4 146163 ? webconfig /usr/sbin/webco
60171 root 00:00:00 0.0 0.0 0 ? kworker/u128:2 [kworker/u128:2
61878 webconf+ 00:00:00 0.0 0.2 144537 ? webconfig /usr/sbin/webco
62889 apache 00:00:00 0.0 0.2 129606 ? httpd /usr/sbin/httpd
I didn't look at the logs as being a noob I don't know yet what I am looking for, I am starting to get my head around how Linux works, but I am right at the start of the journey and annoyingly don't have as much time as I would like to read up on Linux. I am looking forward to my retirement (will never actually retire just throttle back the amount of work I do on a daily basis) to get deeply into Linux. At the moment 90% of my time is Windows support and Linux is the remainder and more of a hobby than a business. I am wanting to provide Linux servers for my small business clients and ClearOS is a great starting point as you can run it without being a guru though I do want to get to that point with Linux as I don't like not understanding the internals of Linux as well as I know the internals of Windows.
Siv -
Accepted Answer
Just to add, ClamAV can also do a scheduled daily file scan (Webconfig > Server > File > Antimalware File Scanner) if you have it installed, but I've checked and it qoes through the chosen file locations sequentially so you only ever get a couple of processes (parent and sub?) for the file scan.
I've recently hacked my daily scanner to exclude picture files as I have a lot of them and they are not known for their viruses. To do that edit /usr/sbin/file_scan and change line 195 from:
to:$clam_scan_options = '';
I've also raised a feature tracker, #19401 to do this in a slightly nicer way.$clam_scan_options = '--exclude="\.(jpg|jpeg|png|gif|mkv)$"';
-
Accepted Answer
Nick,
Thanks for the useful tip.
I don't have a great lot on my system at the moment so I am curious as to what set it off? I have one flexshare that has about 116GB of files in it but that has been there for ages and I have not had the out of memory errors before. It seems since I have rebooted (4 days 10 hours ago), the problem has gone away?
I am just curious to understand what things I can do with Linux when this happens again to see if I can fathom what is triggering the behabiour.
Siv -
Accepted Answer
Good point Nick about the scanner also scanning the system...
However, not sure skipping "picture" files is wise. For instance there is a technique to hide a virus in a .PNG file so instead of viewing the graphic on a windows system - the virus is launched... There is also the problem of the 'double extension'. GIFs and PDFs can also be carriers of viruses...
For graphics files you create yourself there is probably minimal risk in not scanning - but everything from the 'net should be scanned - including graphics files with an email.
The ability to exclude a directory or directories where self created graphics files reside would be safer... Nick really think this 'feature' change should be more carefully considered...
Edit: fixed some typos - why doesn't this site have a preview? -
Accepted Answer
I agree some of these files could possibly contain viruses. I have not excluded PDF's. It was just a string I found on the internet (to which I added mkv). The reason I exclude them is that I have "security camera" which takes about 4k jpeg's a day and a bunch of mkv's and it makes the scan horrendous. Really I should exclude the camera target folder. Also the filter I used as an example is independent of the double-extension method as it is only looking for the last extension.
In the bug tracker, all I have requested is the ability of the file-scan to read the --exclude and --exclude-dir parameters from /etc/clearos/file_scan.conf in the same way as it can already read max-filesize and max-scansize from the file. Those parameters also don't exist in the default /etc/clearos/file_scan.conf, but would be used if added manually. It would be up to the user to add the parameters and configure them to his wishes. -
Accepted Answer
Siv, you just proved my point about restrictive - as illustrated by extracting sections from data you presented...
I think GUI's are OK as long as the GUI designer thinks about the design and who is going to use it
Not sure this happened here
[code]
PID USER TIME %CPU %MEM SZ TT COMMAND COMMAND
603 clam 00:03:02 0.0 14.4 198492 ? clamd /usr/sbin/clamd
....
So its RAM usage is higher than most other daemons but seems to have settled at 13.8 (I assume that is MB?) since I restarted the server.
[/quote]
On every box here that's running clamav the memory size for clamd was 194xxx, close to your figure - suggest that at no time on your box was it 13.8MB...
a 14x difference.
The other restriction is the number of options offered... As a simple example look at the number of options you can use to tune httpd, then check the number of options Webconfig offers... It just isn't practical to put them all into a GUI - not a failing of Webconfig intrinsically, but of GUIs in general.
Did the problem with clamd/memory errors occur during the time the file scan was running? -
Accepted Answer
-
Accepted Answer
Tony,
I had a look at the clamav/clamd.log and get this:
Thu Mar 22 10:25:58 2018 -> Received 0 file descriptor(s) from systemd.
Thu Mar 22 10:25:58 2018 -> clamd daemon 0.99.3 (OS: linux-gnu, ARCH: x86_64, CPU: x86_64)
Thu Mar 22 10:25:58 2018 -> Running as user clam (UID 990, GID 988)
Thu Mar 22 10:25:58 2018 -> Log file size limited to 4294967295 bytes.
Thu Mar 22 10:25:58 2018 -> Reading databases from /var/lib/clamav
Thu Mar 22 10:25:58 2018 -> Not loading PUA signatures.
Thu Mar 22 10:25:58 2018 -> Bytecode: Security mode set to "TrustSigned".
Thu Mar 22 10:26:23 2018 -> +++ Started at Thu Mar 22 10:26:23 2018
Thu Mar 22 10:26:24 2018 -> Received 0 file descriptor(s) from systemd.
Thu Mar 22 10:26:24 2018 -> clamd daemon 0.99.3 (OS: linux-gnu, ARCH: x86_64, CPU: x86_64)
Thu Mar 22 10:26:24 2018 -> Running as user clam (UID 990, GID 988)
Thu Mar 22 10:26:24 2018 -> Log file size limited to 4294967295 bytes.
Thu Mar 22 10:26:24 2018 -> Reading databases from /var/lib/clamav
Thu Mar 22 10:26:24 2018 -> Not loading PUA signatures.
Thu Mar 22 10:26:24 2018 -> Bytecode: Security mode set to "TrustSigned".
Thu Mar 22 10:26:43 2018 -> +++ Started at Thu Mar 22 10:26:43 2018
Thu Mar 22 10:26:43 2018 -> Received 0 file descriptor(s) from systemd.
Thu Mar 22 10:26:43 2018 -> clamd daemon 0.99.3 (OS: linux-gnu, ARCH: x86_64, CPU: x86_64)
Thu Mar 22 10:26:43 2018 -> Running as user clam (UID 990, GID 988)
Thu Mar 22 10:26:43 2018 -> Log file size limited to 4294967295 bytes.
Thu Mar 22 10:26:43 2018 -> Reading databases from /var/lib/clamav
Thu Mar 22 10:26:43 2018 -> Not loading PUA signatures.
Thu Mar 22 10:26:43 2018 -> Bytecode: Security mode set to "TrustSigned".
Thu Mar 22 10:27:06 2018 -> +++ Started at Thu Mar 22 10:27:06 2018
Thu Mar 22 10:27:06 2018 -> Received 0 file descriptor(s) from systemd.
Thu Mar 22 10:27:06 2018 -> clamd daemon 0.99.3 (OS: linux-gnu, ARCH: x86_64, CPU: x86_64)
Thu Mar 22 10:27:06 2018 -> Running as user clam (UID 990, GID 988)
Thu Mar 22 10:27:06 2018 -> Log file size limited to 4294967295 bytes.
Thu Mar 22 10:27:06 2018 -> Reading databases from /var/lib/clamav
Thu Mar 22 10:27:06 2018 -> Not loading PUA signatures.
Thu Mar 22 10:27:06 2018 -> Bytecode: Security mode set to "TrustSigned".
Thu Mar 22 10:27:21 2018 -> +++ Started at Thu Mar 22 10:27:21 2018
Thu Mar 22 10:27:21 2018 -> Received 0 file descriptor(s) from systemd.
Thu Mar 22 10:27:21 2018 -> clamd daemon 0.99.3 (OS: linux-gnu, ARCH: x86_64, CPU: x86_64)
Thu Mar 22 10:27:21 2018 -> Running as user clam (UID 990, GID 988)
Thu Mar 22 10:27:21 2018 -> Log file size limited to 4294967295 bytes.
Thu Mar 22 10:27:21 2018 -> Reading databases from /var/lib/clamav
Thu Mar 22 10:27:21 2018 -> Not loading PUA signatures.
Thu Mar 22 10:27:21 2018 -> Bytecode: Security mode set to "TrustSigned".
Thu Mar 22 10:27:47 2018 -> +++ Started at Thu Mar 22 10:27:47 2018
Thu Mar 22 10:27:47 2018 -> Received 0 file descriptor(s) from systemd.
Thu Mar 22 10:27:47 2018 -> clamd daemon 0.99.3 (OS: linux-gnu, ARCH: x86_64, CPU: x86_64)
Thu Mar 22 10:27:47 2018 -> Running as user clam (UID 990, GID 988)
Thu Mar 22 10:27:47 2018 -> Log file size limited to 4294967295 bytes.
Thu Mar 22 10:27:47 2018 -> Reading databases from /var/lib/clamav
Thu Mar 22 10:27:47 2018 -> Not loading PUA signatures.
Thu Mar 22 10:27:47 2018 -> Bytecode: Security mode set to "TrustSigned".
Thu Mar 22 10:28:03 2018 -> +++ Started at Thu Mar 22 10:28:03 2018
Thu Mar 22 10:28:03 2018 -> Received 0 file descriptor(s) from systemd.
Thu Mar 22 10:28:03 2018 -> clamd daemon 0.99.3 (OS: linux-gnu, ARCH: x86_64, CPU: x86_64)
Thu Mar 22 10:28:03 2018 -> Running as user clam (UID 990, GID 988)
Thu Mar 22 10:28:03 2018 -> Log file size limited to 4294967295 bytes.
Thu Mar 22 10:28:03 2018 -> Reading databases from /var/lib/clamav
Thu Mar 22 10:28:03 2018 -> Not loading PUA signatures.
Thu Mar 22 10:28:03 2018 -> Bytecode: Security mode set to "TrustSigned".
Thu Mar 22 10:28:19 2018 -> +++ Started at Thu Mar 22 10:28:19 2018
Thu Mar 22 10:28:19 2018 -> Received 0 file descriptor(s) from systemd.
Thu Mar 22 10:28:19 2018 -> clamd daemon 0.99.3 (OS: linux-gnu, ARCH: x86_64, CPU: x86_64)
Thu Mar 22 10:28:19 2018 -> Running as user clam (UID 990, GID 988)
Thu Mar 22 10:28:19 2018 -> Log file size limited to 4294967295 bytes.
Thu Mar 22 10:28:19 2018 -> Reading databases from /var/lib/clamav
Thu Mar 22 10:28:19 2018 -> Not loading PUA signatures.
Thu Mar 22 10:28:19 2018 -> Bytecode: Security mode set to "TrustSigned".
Thu Mar 22 10:28:40 2018 -> +++ Started at Thu Mar 22 10:28:40 2018
Thu Mar 22 10:28:40 2018 -> Received 0 file descriptor(s) from systemd.
Thu Mar 22 10:28:40 2018 -> clamd daemon 0.99.3 (OS: linux-gnu, ARCH: x86_64, CPU: x86_64)
Thu Mar 22 10:28:40 2018 -> Running as user clam (UID 990, GID 988)
Thu Mar 22 10:28:40 2018 -> Log file size limited to 4294967295 bytes.
Thu Mar 22 10:28:40 2018 -> Reading databases from /var/lib/clamav
Thu Mar 22 10:28:40 2018 -> Not loading PUA signatures.
Thu Mar 22 10:28:40 2018 -> Bytecode: Security mode set to "TrustSigned".
Thu Mar 22 10:28:56 2018 -> +++ Started at Thu Mar 22 10:28:56 2018
Thu Mar 22 10:28:56 2018 -> Received 0 file descriptor(s) from systemd.
Thu Mar 22 10:28:56 2018 -> clamd daemon 0.99.3 (OS: linux-gnu, ARCH: x86_64, CPU: x86_64)
Thu Mar 22 10:28:56 2018 -> Running as user clam (UID 990, GID 988)
Thu Mar 22 10:28:56 2018 -> Log file size limited to 4294967295 bytes.
Thu Mar 22 10:28:56 2018 -> Reading databases from /var/lib/clamav
Thu Mar 22 10:28:56 2018 -> Not loading PUA signatures.
Thu Mar 22 10:28:56 2018 -> Bytecode: Security mode set to "TrustSigned".
Thu Mar 22 10:29:20 2018 -> +++ Started at Thu Mar 22 10:29:20 2018
Thu Mar 22 10:29:20 2018 -> Received 0 file descriptor(s) from systemd.
Thu Mar 22 10:29:20 2018 -> clamd daemon 0.99.3 (OS: linux-gnu, ARCH: x86_64, CPU: x86_64)
Thu Mar 22 10:29:20 2018 -> Running as user clam (UID 990, GID 988)
Thu Mar 22 10:29:20 2018 -> Log file size limited to 4294967295 bytes.
Thu Mar 22 10:29:20 2018 -> Reading databases from /var/lib/clamav
Thu Mar 22 10:29:20 2018 -> Not loading PUA signatures.
Thu Mar 22 10:29:20 2018 -> Bytecode: Security mode set to "TrustSigned".
Thu Mar 22 10:29:44 2018 -> +++ Started at Thu Mar 22 10:29:44 2018
Thu Mar 22 10:29:44 2018 -> Received 0 file descriptor(s) from systemd.
Thu Mar 22 10:29:44 2018 -> clamd daemon 0.99.3 (OS: linux-gnu, ARCH: x86_64, CPU: x86_64)
Thu Mar 22 10:29:44 2018 -> Running as user clam (UID 990, GID 988)
Thu Mar 22 10:29:44 2018 -> Log file size limited to 4294967295 bytes.
Thu Mar 22 10:29:44 2018 -> Reading databases from /var/lib/clamav
Thu Mar 22 10:29:44 2018 -> Not loading PUA signatures.
Thu Mar 22 10:29:44 2018 -> Bytecode: Security mode set to "TrustSigned".
Thu Mar 22 10:30:04 2018 -> +++ Started at Thu Mar 22 10:30:04 2018
Thu Mar 22 10:30:04 2018 -> Received 0 file descriptor(s) from systemd.
Thu Mar 22 10:30:04 2018 -> clamd daemon 0.99.3 (OS: linux-gnu, ARCH: x86_64, CPU: x86_64)
Thu Mar 22 10:30:04 2018 -> Running as user clam (UID 990, GID 988)
Thu Mar 22 10:30:04 2018 -> Log file size limited to 4294967295 bytes.
Thu Mar 22 10:30:04 2018 -> Reading databases from /var/lib/clamav
Thu Mar 22 10:30:04 2018 -> Not loading PUA signatures.
Thu Mar 22 10:30:04 2018 -> Bytecode: Security mode set to "TrustSigned".
Thu Mar 22 10:30:23 2018 -> +++ Started at Thu Mar 22 10:30:23 2018
Thu Mar 22 10:30:23 2018 -> Received 0 file descriptor(s) from systemd.
Thu Mar 22 10:30:23 2018 -> clamd daemon 0.99.3 (OS: linux-gnu, ARCH: x86_64, CPU: x86_64)
Thu Mar 22 10:30:23 2018 -> Running as user clam (UID 990, GID 988)
Thu Mar 22 10:30:23 2018 -> Log file size limited to 4294967295 bytes.
Thu Mar 22 10:30:23 2018 -> Reading databases from /var/lib/clamav
Thu Mar 22 10:30:23 2018 -> Not loading PUA signatures.
Thu Mar 22 10:30:23 2018 -> Bytecode: Security mode set to "TrustSigned".
This was around the time that I started getting emails from the notifications system saying that clamd was being shut down as the sytem was running low on memory. What it looks like to me is that Clam was trying to run and the system was shutting it dowm and then it was restarting itself again and then being shut down again in a vicious circle. My decision to reboot the sstem seems to have been the right decision in that this behaviour has now stopped and I am not getting warnings any more since I did that.
Siv -
Accepted Answer
Tony,
That bug report absolutely does look identical to what I was getting! Reading through that post it looks like there was a fix coming in the 0.99.2 version and looking at the logs we are now on
so I am concerned that maybe they have not fixed it?Thu Mar 22 10:26:43 2018 -> clamd daemon 0.99.3 (OS: linux-gnu, ARCH: x86_64, CPU: x86_64)
-
Accepted Answer
Note that report was on a different Linux distribution...
Maybe some new bug triggering the same condition, maybe a regression?
Next step maybe here if you want to pursue further?
https://www.clamav.net/documents/mailing-lists-faq
Anyway it's 3.52 am here in Sydney - past time should be tucked up in bed snoozing... -
Accepted Answer
-
Accepted Answer
Tony,
I have accepted your post identifying my issue as very similar to the one from 2016 in the Debian bug lists. Given the issue seems to have passed and may have been a one off I don't think it's worth pursuing. If it happens again then I may reach out to them and see if as you say it's a regression?
Thanks again for your and Nick's help and advice, it is appreciated.
Siv
Please login to post a reply
You will need to be logged in to be able to post a reply. Login using the form on the right or register an account if you are new here.
Register Here »