This is not going to help too much, but you can get rid of the messages in two ways. On my server I changed the LogLevel. I can't remember what you have to then restart to have the change work.

It also only masks the issue. I would go for the LogLevel solution as it stops the messages from being created in the first place. The rsyslog solution just junks the messages once they have been created.

I've also recently had a problem with very high system loads with amavis flat-lining at 100% (i.e 1 core flat out) and killing amavisd did not help as it just restarted. I can't get to the vm I had that on for teh moment, but I think the solution in the end appeared to be a memory limitation. I'd been testing too much on a small VM and not removed things like the proxy/content filter, gateway management and so on. Once I pared things back I was OK, but I suspect this won't help you.

It looks like you can change the loglevel for the current session with:

systemd-analyze set-log-level notice

Reference

The when you reboot the permanent change will take effect

I'm facing a similar situation. Starting to day, I'm getting thousands of notification on two serves. Looks like there were some automatic updates today.

User root logged in via sudo 2019-04-06 14:54:05
User root logged out via sudo 2019-04-06 14:54:05
User root logged in via sudo 2019-04-06 14:54:04
User root logged out via sudo 2019-04-06 14:54:04
User root logged out via sudo 2019-04-06 14:54:03
User root logged in via sudo 2019-04-06 14:54:03
User root logged out via sudo 2019-04-06 14:54:03
User root logged in via sudo 2019-04-06 14:54:03

I am in a similar situation. Starting today I am getting thousands of SUDO log in and SUDO log out notifications on two servers. There were some automatic updates this morning.

@Greg,
If you can get to the command line, please can you do a:

systemd-analyze set-log-level notice

Then we can get a chance to see toe wood from the trees. Does it make your command line become more responsive?

Please then have a look at "top" and see if anything is consuming a lot of resources and that your memory usage is OK

Hi Nick

I tried "systemd-analyze set-log-level notice" last night, did a reboot and it made no difference.
I am now not able to access my unit at all

Doing:

systemd-analyze set-log-level notice

is only temporary. You lose it on next reboot. To make it permanent you have to edit the LogLevel in /etc/systemd/system.conf.

Please hang fire with a clean install. I'll try to ping Dave.

Can you boot into Rescue mode. You may be able to make the edit to /etc/systemd/system.conf from there. Also, on the off-chance, try deleting /var/lib/csplugin-events/events.db.

If you can, have a look at the logs using "cat", "tail", "more" or "less" and look for anything obvious. If /var/log/messages is huge try:

sed -i '/Started Session.*of user root/d' /var/log/messages

note that if you do this on a running version of ClearOS you will need to restart rsyslog or you may find some logging stops.

In 4h30m of uptime I have 735 messages like that. That is easily bearable. I wonder what is going on?

@Leon and Greg,
Clearcenter are happy to log on and have a look, if you can get to your boxes. Please raise a General Enquiry ticket at https://secure.clearcenter.com/portal/ticket_manage.jsp and submit remote login credentials to https://secure.clearcenter.com/portal/system_password.jsp. Clearly you need to be able to get in to your box to open the incoming SSH port, so I realise it may be difficult.

Nick Howitt wrote:

@Leon and Greg,
Clearcenter are happy to log on and have a look, if you can get to your boxes. Please raise a General Enquiry ticket at https://secure.clearcenter.com/portal/ticket_manage.jsp and submit remote login credentials to https://secure.clearcenter.com/portal/system_password.jsp. Clearly you need to be able to get in to your box to open the incoming SSH port, so I realise it may be difficult.

@Nick,
Thanks. I remotely shut both servers down as they were becoming unresponsive. I will be local to the boxes Monday morning.

The proper way to turn off notices is to leave the file /etc/systemd/system.conf alone and make a file in /etc/systemd/system.conf.d/clearos-base.conf with the contents:

[Manager]

LogLevel=notice

Since the information is not typically necessary for any actionable items under ClearOS. I will be trying to place this file in the location above as it is normal to have these messages only raise if they meet the level of notice and not information.

I've created this as a bug since info logging is not important from systemd but notice logging is.

https://gitlab.com/clearos/clearfoundation/system-base/commit/dc613d20105c34390569006079b0803cf1b230b1

Nick Howitt wrote:

@Greg,
If you can get to the command line, please can you do a:

systemd-analyze set-log-level notice

Then we can get a chance to see toe wood from the trees. Does it make your command line become more responsive?

Please then have a look at "top" and see if anything is consuming a lot of resources and that your memory usage is OK

Still getting all the notifications after systemd-analyze set-log-level notice. Top shows low CPU and memory usage.

@Greg, Is your system still responsive? Log messages can be dropped easily. Also have you raised a ticket at Clearcenter?

Nick Howitt wrote:

@Greg, Is your system still responsive? Log messages can be dropped easily. Also have you raised a ticket at Clearcenter?

Yes it is responsive but VERY sluggish. I have raised a support ticket.

@Leon,
We now have a couple of patches going through and once they sync to the repos, you can update them with a:

yum update csplugin-events app-base --enablerepo=clearos-updates-testing

You are looking for csplugin-events-1.2-3.v7 and app-base-2.6.4-1.v7 or app-base-2.6.5-1.v7 (either is OK). We are also looking at patching the audit log but this will be in slow time.

If you can't wait, as an alternative solution you can add a single line to /etc/pam.d/system-auth-ac:

session [success=1 default=ignore] pam_succeed_if.so service in sudo quiet uid = 0 ruser = webconfig

It needs to go above the last two lines so the last section reads:

session     optional      pam_keyinit.so revoke

session     required      pam_limits.so

-session     optional      pam_systemd.so

session     optional      pam_mkhomedir.so umask=0077

session     [success=1 default=ignore] pam_succeed_if.so service in crond quiet use_uid

session     [success=1 default=ignore] pam_succeed_if.so service in sudo quiet uid = 0 ruser = webconfig

session     required      pam_unix.so

session     optional      pam_ldap.so

After making the change there is no need to restart anything. This change should not interfere with the patches.

You should also be able to clear down the events database with:

systemctl stop clearsync.service

rm -f /var/lib/csplugin-events/events.db

systemctl start clearsync.service

previous csplugin-events problem

I logged in to report similar issues on my servers I am getting this:

Apr 9 17:27:15 remote smbd_audit: sharon|192.168.17.188|telldir|ok|
Apr 9 17:27:15 remote smbd_audit: sharon|192.168.17.188|readdir|ok|
Apr 9 17:27:15 remote smbd_audit: sharon|192.168.17.188|translate_name|fail (Operation not supported)|
Apr 9 17:27:15 remote smbd_audit: sharon|192.168.17.188|telldir|ok|
Apr 9 17:27:15 remote smbd_audit: sharon|192.168.17.188|readdir|ok|
Apr 9 17:27:15 remote smbd_audit: sharon|192.168.17.188|translate_name|fail (Operation not supported)|
Apr 9 17:27:15 remote smbd_audit: sharon|192.168.17.188|telldir|ok|
Apr 9 17:27:15 remote smbd_audit: sharon|192.168.17.188|readdir|ok|
Apr 9 17:27:15 remote smbd_audit: sharon|192.168.17.188|translate_name|fail (Operation not supported)|
Apr 9 17:27:15 remote smbd_audit: sharon|192.168.17.188|telldir|ok|
Apr 9 17:27:15 remote smbd_audit: sharon|192.168.17.188|readdir|ok|
Apr 9 17:27:15 remote smbd_audit: sharon|192.168.17.188|translate_name|fail (Operation not supported)|
Apr 9 17:27:15 remote smbd_audit: sharon|192.168.17.188|telldir|ok|
Apr 9 17:27:15 remote smbd_audit: sharon|192.168.17.188|readdir|ok|
Apr 9 17:27:15 remote smbd_audit: sharon|192.168.17.188|translate_name|fail (Operation not supported)|
Apr 9 17:27:15 remote smbd_audit: sharon|192.168.17.188|telldir|ok|
Apr 9 17:27:15 remote smbd_audit: sharon|192.168.17.188|readdir|ok|
Apr 9 17:27:15 remote smbd_audit: sharon|192.168.17.188|translate_name|fail (Operation not supported)|
Apr 9 17:27:15 remote smbd_audit: sharon|192.168.17.188|telldir|ok|
Apr 9 17:27:15 remote smbd_audit: sharon|192.168.17.188|readdir|ok|
Apr 9 17:27:15 remote smbd_audit: sharon|192.168.17.188|translate_name|fail (Operation not supported)|

Literally thousands of logged events. It seems to relate to a Samba update that went in over last weekend early Monday morning and I am getting pages of event notifications.
Can anyone confirm if this is the same issue as reported by other users here or a different issue?

Siv

@Tony,
It is not an events problem per-se. It is just that is where the problem manifests itself, although if the events system were faster, the problem may have gone away. The real issue seems to be that the webconfig has changed from doing a single "su" to assume root piviliges to doing a "sudo" for every necessary transaction. This is generating a huge amount of transactions - look in the audit log and secure log e.g:

Apr  9 09:25:53 gateway-h sudo: pam_unix(sudo:session): session opened for user root by (uid=0)

Apr  9 09:25:53 gateway-h sudo: pam_unix(sudo:session): session closed for user root

These also go into the events system. It is possible to suppress some of these at source by modding the pam configuration but this is not a program maintained by ClearOS so they are going down the route of blocking the messages from going in the events database. At the same time they are adding an rsyslog filter to drop the messages from the secure log.

This rsyslog filter is good in that it can easily be added to to get rid of other annoying messages. That is on my to do list.

There is a residual problem which will be fixed in slow time. There is a huge amount of logging going on in the audit daemon now and the log rotates now every few hours. This will be will be looked at in slow time. It is possible to block the messages by adding a rules file to /etc/audit/rules.d and restarting the audit daemon. To block webconfig messages add:

-a never,exclude  -F uid=webconfig

.Root can similarly be blocked but this rather negates the audit daemon so this needs to be stufied. What is really needed to be blocked is where root does a "sudo" and I need to study the manual for that.

@Graham,
That is a different issue. Which log file are you seeing that in? Are you using the AD Connector? Or do you have Audit Log enabled in a flexshare?

Perhaps it would be a good idea to start a new thread and also post the output of "testparm -s"

Nick Howitt wrote:

@Graham,
That is a different issue. Which log file are you seeing that in? Are you using the AD Connector? Or do you have Audit Log enabled in a flexshare?

Perhaps it would be a good idea to start a new thread and also post the output of "testparm -s"

Nick,

This is comes to me each day as a report by email. I haven't looked at any logs as such, I just noticed in the Updates section of the web interface these notifications started being generated after some updates to Samba were applied over the weekend.

Siv

Nick Howitt wrote:

@Graham,
That is a different issue. Which log file are you seeing that in? Are you using the AD Connector? Or do you have Audit Log enabled in a flexshare?

Perhaps it would be a good idea to start a new thread and also post the output of "testparm -s"

[/quote]
I also am seeing this in the "Events and Notifications" section of the Web GUI:
User root logged out via sudo 2019-04-10 09:06:16
User root logged in via sudo 2019-04-10 09:06:15
User root logged in via sudo 2019-04-10 09:06:13
User root logged out via sudo 2019-04-10 09:06:13
User root logged in via sudo 2019-04-10 09:06:00
User root logged out via sudo 2019-04-10 09:06:00
User root logged out via sudo 2019-04-10 09:05:57
User root logged in via sudo 2019-04-10 09:05:56
User root logged in via sudo 2019-04-10 09:05:01
User root logged out via sudo 2019-04-10 09:05:01

There are currenty 10,000 odd entries and it's occurring every second. I can access the server but it's obviously hammering it whatever "it" is.

Siv

Nick Howitt wrote:

@Graham,
That is a different issue. Which log file are you seeing that in? Are you using the AD Connector? Or do you have Audit Log enabled in a flexshare?

Perhaps it would be a good idea to start a new thread and also post the output of "testparm -s"

I can't remote in via SSH as it appears to be not responding I just get "Connection timed out" so can't send you the results of testparm -s

Though I can get in via the web gui?

Siv

@Graham, from earlier in the thread:

yum update csplugin-events app-base --enablerepo=clearos-updates-testing

I'll have to get the fixes released tonight. They did not finish compiling before I shut down last night.

If you do manage to get in by ssh, you can shutdown clearsyncd with a:

service clearsync stop

Alternatively, from the webconfig, go Webconfig > System > Settings > Services and shut it down. Once you have done the update, you can start the service again.

Nick Howitt wrote:

@Graham, from earlier in the thread:

yum update csplugin-events app-base --enablerepo=clearos-updates-testing

I'll have to get the fixes released tonight. They did not finish compiling before I shut down last night.

If you do manage to get in by ssh, you can shutdown clearsyncd with a:

service clearsync stop

Alternatively, from the webconfig, go Webconfig > System > Settings > Services and shut it down. Once you have done the update, you can start the service again.

In my server I can't see System > Settings > Services. I do get Process Viewer by going to Reports > Performance And Resouces > Process Viewer and in their I get a running process list with a "Kill" button next to each one. I can find clearsyncd in there. When I hit "kill" it says "Confirm delete" and the process ID, will this stop the service or will it actually delete it, I certainly don't want to do that? If it means just stop teh service then I do.

I won't press "OK" until you get back t me.

@Graham, I don't think you have a choice. It should just kill the process from running, i.e brutally stop it. It will not delete the service. You should then regain SSH access.

I've just released the updates, but they won't come through automatically until tonight.

Nick

adding a rules file to /etc/audit/rules.d and restarting the audit daemon

You cannot do this using systemctl on a standard ClearOS system - manual start,stop and restart of auditd.. es ist verboten

Reboot, or a quick and dirty way...
1. Edit /etc/systemd/system/multi-user.target.wants/auditd.service
2. Find the line "RefuseManualStop=yes" and change to "RefuseManualStop=no"
3. Run "systemctl daemon-reload"
4. run "systemctl restart auditd"

auditd is an SSD killer, run most of my ClearOS systems on an SSD - where there is also a standard disk, edit /etc/audit/auditd.conf and log to another directory not on the SSD. For systems with SSD only - crude solution at the moment is to change the line "write_logs = yes" to "write_logs = no" in the conf file..

Still immune to csplugin-events problems as the /etc/clearsync.d/csplugin-events.conf on all systems is still being deleted by a cron job. A CPU being driven to nearly 100% by this event file is not a trivial problem...

Tony Ellis wrote:

Nick

adding a rules file to /etc/audit/rules.d and restarting the audit daemon

You cannot do this using systemctl on a standard ClearOS system - manual start,stop and restart of auditd.. es ist verboten

Are you sure? I've been doing it yesterday and today to troubleshoot adding rules to /etc/audit/rules.d.

There may be a particular auditd rule you can enable which may enforce the "RefuseManualStop" (/usr/share/doc/audit-2.8.4/rules), but with a system I installed a couple of days ago, I can stop/start/restart.

[root@danda rules.d]# systemctl stop auditd

Failed to stop auditd.service: Operation refused, unit auditd.service may be requested by dependency only (it is configured to refuse manual start/stop).

See system logs and 'systemctl status auditd.service' for details.

Same on every system - all 7.x community...

Tony Ellis wrote:

[root@danda rules.d]# systemctl stop auditd

Failed to stop auditd.service: Operation refused, unit auditd.service may be requested by dependency only (it is configured to refuse manual start/stop).

See system logs and 'systemctl status auditd.service' for details.

Same on every system - all 7.x community...

Here is a nice bug for you. Instead try:

service auditd stop

[root@gateway-h ~]# service auditd start

Redirecting to /bin/systemctl start auditd.service

[root@gateway-h ~]# systemctl stop auditd

Failed to stop auditd.service: Operation refused, unit auditd.service may be requested by dependency only (it is configured to refuse manual start/stop).

See system logs and 'systemctl status auditd.service' for details.



[root@gateway-h ~]# service auditd stop

Stopping logging:                                          [  OK  ]

I was being lazy and using the service command!

Nick Howitt wrote:

@Graham, I don't think you have a choice. It should just kill the process from running, i.e brutally stop it. It will not delete the service. You should then regain SSH access.

I've just released the updates, but they won't come through automatically until tonight.

Nick I managed to kill clearsyncd but still can't get in via SSH it just keeps timing out? The users don't appear to be having any issues so I think I will wait and see if the update gets through tonight, or do I have to run the command:

yum update csplugin-events app-base --enablerepo=clearos-updates-testing

Before I can get your fixes?

I looked again at the processes and clearsyncd has turned back on again. It did get killed through the web GUI but has started back up again.

Siv

@Siv,
I've released the programs into clearos-updates so they will update tonight. You could try Webconfig > Cloud > Updates > Software Updates and see if that pull them down. I can't test as I've updated my systems. If it shows them but does not give you the chance to update, try switching to a manual update (disable "Automatic Updates").

If you can't update, I suggest you shut down the webconfig on your PC or avoid the screens with a lot of ajax. I don't know which they are, but any with fields that update as you watch the screen, so certainly avoid the IP Settings screen.

Nick - still the same on the latest Fedora, just tried it... thanks for the workaround...

Maybe "service" is classified under "may be requested by dependency only"

Tony Ellis wrote:

Nick - still the same on the latest Fedora, just tried it... thanks for the workaround...

Maybe "service" is classified under "may be requested by dependency only"

I suspect it is just an upstream bug which someone noticed and patched and FC is way ahead of ClearOS/Centos/RHEL. We are just using the el7 version of systemd unmodified.

[edit]
auditd changed to systemd
[/edit]

Sorry - wasn't clear enough. Fedora 29 still the same result as with ClearOS/CentOS 7.x - systemctl doesn't work without the edit to the conf file, service does work regardless...

Nick Howitt wrote:

@Siv,
I've released the programs into clearos-updates so they will update tonight. You could try Webconfig > Cloud > Updates > Software Updates and see if that pull them down. I can't test as I've updated my systems. If it shows them but does not give you the chance to update, try switching to a manual update (disable "Automatic Updates").

If you can't update, I suggest you shut down the webconfig on your PC or avoid the screens with a lot of ajax. I don't know which they are, but any with fields that update as you watch the screen, so certainly avoid the IP Settings screen.

Nick,
I finally did manage to get in via SSH and was able to run the command:

yum update csplugin-events app-base --enablerepo=clearos-updates-testing

So I am hopeful that your updates will get through and quieten things down again.

I will watch to see if the other messages come up in the emailed report I get sent at the end of the day that I initially reported as I thought they were related to this. If they do I will raise them as a new thread in these forums.

As always thanks for all you do Nick.

@Nick,
I just checked my system and no outstanding updates, the following were the updates that were applied most recently and seemed to trigger all sorts of log activity:

 Package	Action	Date/Time

elfutils-default-yama-scope-0.172-2.el7 	Updated 	Apr 08, 01:02:54

linux-firmware-20180911-69.git85c5d90.el ... 	Updated 	Apr 07, 05:48:51

lm_sensors-libs-3.4.0-6.20160601gitf9185 ... 	Updated 	Apr 07, 05:48:41

samba-python-4.8.3-4.3.v7 	Updated 	Apr 07, 05:48:41

grub2-2.02.1-0.76.v7.1 	Updated 	Apr 07, 05:48:40

samba-winbind-clients-4.8.3-4.3.v7 	Updated 	Apr 07, 05:48:40

samba-client-4.8.3-4.3.v7 	Updated 	Apr 07, 05:48:39

samba-dc-4.8.3-4.3.v7 	Updated 	Apr 07, 05:48:39

libsmbclient-4.8.3-4.3.v7 	Updated 	Apr 07, 05:48:38

samba-dc-libs-4.8.3-4.3.v7 	Updated 	Apr 07, 05:48:38

samba-winbind-4.8.3-4.3.v7 	Updated 	Apr 07, 05:48:38

samba-winbind-modules-4.8.3-4.3.v7 	Updated 	Apr 07, 05:48:35

grub2-pc-2.02.1-0.76.v7.1 	Updated 	Apr 07, 05:48:34

grub2-pc-modules-2.02.1-0.76.v7.1 	Updated 	Apr 07, 05:48:34

grub2-tools-extra-2.02.1-0.76.v7.1 	Updated 	Apr 07, 05:48:33

samba-4.8.3-4.3.v7 	Updated 	Apr 07, 05:48:33

samba-common-tools-4.8.3-4.3.v7 	Updated 	Apr 07, 05:48:27

grub2-tools-2.02.1-0.76.v7.1 	Updated 	Apr 07, 05:48:26

grub2-tools-minimal-2.02.1-0.76.v7.1 	Updated 	Apr 07, 05:48:26

grub2-common-2.02.1-0.76.v7.1 	Updated 	Apr 07, 05:48:25

samba-libs-4.8.3-4.3.v7 	Updated 	Apr 07, 05:48:25

samba-client-libs-4.8.3-4.3.v7 	Updated 	Apr 07, 05:48:24

samba-common-libs-4.8.3-4.3.v7 	Updated 	Apr 07, 05:48:23

libwbclient-4.8.3-4.3.v7 	Updated 	Apr 07, 05:48:22

samba-common-4.8.3-4.3.v7 	Updated 	Apr 07, 05:48:21

mesa-libwayland-egl-17.2.3-8.20171019.el ... 	Erased 	Apr 06, 02:07:11

iwl3160-firmware-22.0.7.0-69.el7 	Updated 	Apr 06, 02:02:59

iwl3945-firmware-15.32.2.9-69.el7 	Updated 	Apr 06, 02:02:59

iwl6050-firmware-41.28.5.1-69.el7 	Updated 	Apr 06, 02:02:59

iwl100-firmware-39.31.5.1-69.el7 	Updated 	Apr 06, 02:02:58

iwl2000-firmware-18.168.6.1-69.el7 	Updated 	Apr 06, 02:02:58

iwl6000-firmware-9.221.4.1-69.el7 	Updated 	Apr 06, 02:02:58

iwl7260-firmware-22.0.7.0-69.el7 	Updated 	Apr 06, 02:02:58

iwl1000-firmware-39.31.5.1-69.el7 	Updated 	Apr 06, 02:02:57

iwl135-firmware-18.168.6.1-69.el7 	Updated 	Apr 06, 02:02:57

iwl5150-firmware-8.24.2.2-69.el7 	Updated 	Apr 06, 02:02:57

iwl6000g2b-firmware-17.168.5.2-69.el7 	Updated 	Apr 06, 02:02:57

emacs-filesystem-24.3-22.el7 	Updated 	Apr 06, 02:02:56

iwl105-firmware-18.168.6.1-69.el7 	Updated 	Apr 06, 02:02:56

iwl2030-firmware-18.168.6.1-69.el7 	Updated 	Apr 06, 02:02:56

iwl4965-firmware-228.61.2.24-69.el7 	Updated 	Apr 06, 02:02:56

iwl5000-firmware-8.83.5.1_1-69.el7 	Updated 	Apr 06, 02:02:56

iwl6000g2a-firmware-17.168.5.3-69.el7 	Updated 	Apr 06, 02:02:56

dialog-1.2-5.20130523.el7 	Updated 	Apr 06, 02:02:55

dosfstools-3.0.20-10.el7 	Updated 	Apr 06, 02:02:55

dmidecode-3.1-2.el7 	Updated 	Apr 06, 02:02:53

iw-4.3-2.el7 	Updated 	Apr 06, 02:02:53

iptraf-ng-1.1.4-7.el7 	Updated 	Apr 06, 02:02:52

libgomp-4.8.5-36.el7 	Updated 	Apr 06, 02:02:52

augeas-libs-1.4.0-6.el7_6.1 	Updated 	Apr 06, 02:02:51

Because the initial errors I posted seemed to relate to Samba and seemed to start around the time the 7th April updates were being applied i have assumed the two things are related.

One other thing that occurred was the Windows machines that had mapped links to Flexshares on the server suddenly would not connect and I had to restart the server to get them to connect up again.

Siv

@Nick,
When I ran the command it came up with "No packages marked for update":

yum update csplugin-events app-base --enablerepo=clearos-updates-testing

Loaded plugins: clearcenter-marketplace, fastestmirror

ClearCenter Marketplace: fetching repositories...

Loading mirror speeds from cached hostfile

 * clearos: www.mirrorservice.org

 * clearos-centos: download3.clearsdn.com

 * clearos-centos-sclo-rh: download3.clearsdn.com

 * clearos-centos-updates: download3.clearsdn.com

 * clearos-contribs: www.mirrorservice.org

 * clearos-epel: download3.clearsdn.com

 * clearos-fast-updates: download3.clearsdn.com

 * clearos-infra: www.mirrorservice.org

 * clearos-updates: www.mirrorservice.org

 * clearos-updates-testing: www.mirrorservice.org

 * private-clearcenter-dyndns: download1.clearsdn.com:80

 * private-clearcenter-roundcubemail: download1.clearsdn.com:80

clearos-updates-testing                                  | 3.5 kB     00:00     

(1/2): clearos-updates-testing/7/updateinfo                |   96 B   00:00     

(2/2): clearos-updates-testing/7/primary_db                | 736 kB   00:00     



No packages marked for update



[root@remote ~]# service clearsync start

Redirecting to /bin/systemctl start clearsync.service

Siv

The update that you should be running is csplugin-events-1.2-3.v7

http://koji.clearos.com/koji/buildinfo?buildID=2618

You can look at what version you are running by using:

rpm -qi csplugin-events

If you are not running this version, run the following to clean up your yum cache:

yum --enablerepo=* clean all

As you can see from koji, this package is in clearos-updates. If you are running ClearOS Community you will see this by just doing a 'yum update' after doing the 'yum --enablerepo=* clean all'

If you are running ClearOS Business and have installed these updates already, then you will need to enable the clearos-udpates repo in order to grab this update:

yum --enablerepo=clearos-updates update csplugin-events

That is weird. Just a plain old "yum update" should work as I moved them into the clearos-update this morning. The --enablerepo bit is irrelevant now and the yum command should work with or without it. Can I suggest you leave the clearsync service disabled for the moment (or until you get the updates) and then do a:

yum clean all && yum update

Dave Loper wrote:

The update that you should be running is csplugin-events-1.2-3.v7

http://koji.clearos.com/koji/buildinfo?buildID=2618

You can look at what version you are running by using:

rpm -qi csplugin-events

If you are not running this version, run the following to clean up your yum cache:

yum --enablerepo=* clean all

As you can see from koji, this package is in clearos-updates. If you are running ClearOS Community you will see this by just doing a 'yum update' after doing the 'yum --enablerepo=* clean all'

If you are running ClearOS Business and have installed these updates already, then you will need to enable the clearos-udpates repo in order to grab this update:

yum --enablerepo=clearos-updates update csplugin-events

Dave,

Thanks for the advice, I ran the commands and I __was__ running the previous version.
I then did:

yum --enablerepo=* clean all

Then:
yum update

Below is the response:

[root@remote ~]# yum update

Loaded plugins: clearcenter-marketplace, fastestmirror

ClearCenter Marketplace: fetching repositories...

Determining fastest mirrors

 * clearos: www.mirrorservice.org

 * clearos-centos: download3.clearsdn.com

 * clearos-centos-sclo-rh: download3.clearsdn.com

 * clearos-centos-updates: download3.clearsdn.com

 * clearos-contribs: www.mirrorservice.org

 * clearos-epel: download3.clearsdn.com

 * clearos-fast-updates: download3.clearsdn.com

 * clearos-infra: www.mirrorservice.org

 * clearos-updates: www.mirrorservice.org

 * private-clearcenter-dyndns: download3.clearsdn.com:80

 * private-clearcenter-roundcubemail: download1.clearsdn.com:80

clearos                                                  | 3.7 kB     00:00     

clearos-centos                                           | 3.0 kB     00:00     

clearos-centos-sclo-rh                                   | 3.0 kB     00:00     

clearos-centos-updates                                   | 3.0 kB     00:00     

clearos-contribs                                         | 3.5 kB     00:00     

clearos-epel                                             | 2.9 kB     00:00     

clearos-fast-updates                                     | 3.0 kB     00:00     

clearos-infra                                            | 3.5 kB     00:00     

clearos-updates                                          | 3.5 kB     00:00     

(1/13): clearos/7/group_gz                                 | 1.6 kB   00:00     

(2/13): clearos-contribs/7/updateinfo                      |   96 B   00:00     

(3/13): clearos-contribs/7/primary_db                      |  71 kB   00:00     

(4/13): clearos/7/primary_db                               | 739 kB   00:00     

(5/13): clearos-fast-updates/x86_64/primary_db             |  10 kB   00:00     

(6/13): clearos-infra/7/updateinfo                         |   96 B   00:00     

(7/13): clearos-infra/7/primary_db                         |  11 kB   00:00     

(8/13): clearos-updates/7/updateinfo                       |   96 B   00:00     

(9/13): clearos-updates/7/primary_db                       | 861 kB   00:00     

(10/13): clearos-centos-sclo-rh/x86_64/primary_db          | 3.5 MB   00:07     

(11/13): clearos-centos-updates/x86_64/primary_db          | 2.3 MB   00:09     

(12/13): clearos-epel/7/x86_64/primary_db                  | 7.4 MB   00:09     

(13/13): clearos-centos/x86_64/primary_db                  | 5.0 MB   00:14     

private-clearcenter-dyndns                               | 3.0 kB     00:00     

private-clearcenter-dyndns/primary_db                    | 2.6 kB     00:00     

private-clearcenter-roundcubemail                        | 3.0 kB     00:00     

private-clearcenter-roundcubemail/primary_db             | 2.5 kB     00:00     

Resolving Dependencies

--> Running transaction check

---> Package app-base.noarch 1:2.6.3-1.v7 will be updated

---> Package app-base.noarch 1:2.6.5-1.v7 will be an update

---> Package app-base-core.noarch 1:2.6.3-1.v7 will be updated

---> Package app-base-core.noarch 1:2.6.5-1.v7 will be an update

---> Package csplugin-events.x86_64 0:1.2-2.v7 will be updated

---> Package csplugin-events.x86_64 0:1.2-3.v7 will be an update

--> Finished Dependency Resolution



Dependencies Resolved



================================================================================

 Package              Arch        Version            Repository            Size

================================================================================

Updating:

 app-base             noarch      1:2.6.5-1.v7       clearos-updates       32 k

 app-base-core        noarch      1:2.6.5-1.v7       clearos-updates      454 k

 csplugin-events      x86_64      1.2-3.v7           clearos-updates       84 k



Transaction Summary

================================================================================

Upgrade  3 Packages



Total download size: 570 k

Is this ok [y/d/N]: y

Downloading packages:

Delta RPMs disabled because /usr/bin/applydeltarpm not installed.

(1/3): app-base-2.6.5-1.v7.noarch.rpm                      |  32 kB   00:00     

(2/3): app-base-core-2.6.5-1.v7.noarch.rpm                 | 454 kB   00:00     

(3/3): csplugin-events-1.2-3.v7.x86_64.rpm                 |  84 kB   00:00     

--------------------------------------------------------------------------------

Total                                              1.0 MB/s | 570 kB  00:00     

Running transaction check

Running transaction test

Transaction test succeeded

Running transaction

  Updating   : 1:app-base-core-2.6.5-1.v7.noarch                            1/6 

  Updating   : 1:app-base-2.6.5-1.v7.noarch                                 2/6 

  Updating   : csplugin-events-1.2-3.v7.x86_64                              3/6 

  Cleanup    : 1:app-base-2.6.3-1.v7.noarch                                 4/6 

  Cleanup    : 1:app-base-core-2.6.3-1.v7.noarch                            5/6 

  Cleanup    : csplugin-events-1.2-2.v7.x86_64                              6/6 

  Verifying  : 1:app-base-core-2.6.5-1.v7.noarch                            1/6 

  Verifying  : csplugin-events-1.2-3.v7.x86_64                              2/6 

  Verifying  : 1:app-base-2.6.5-1.v7.noarch                                 3/6 

  Verifying  : csplugin-events-1.2-2.v7.x86_64                              4/6 

  Verifying  : 1:app-base-2.6.3-1.v7.noarch                                 5/6 

  Verifying  : 1:app-base-core-2.6.3-1.v7.noarch                            6/6 



Updated:

  app-base.noarch 1:2.6.5-1.v7           app-base-core.noarch 1:2.6.5-1.v7     

  csplugin-events.x86_64 0:1.2-3.v7     



Complete!

[root@remote ~]# 

So I now have "csplugin-events.x86_64 0:1.2-3.v7", let's hope this qquitens down the logging of events.
I will see if after applying this update the log files stop writing events every second or so.

Should I restart the server or a specific service?

Siv

Nick Howitt wrote:

That is weird. Just a plain old "yum update" should work as I moved them into the clearos-update this morning. The --enablerepo bit is irrelevant now and the yum command should work with or without it. Can I suggest you leave the clearsync service disabled for the moment (or until you get the updates) and then do a:

yum clean all && yum update

Sorry Nick your post came in whilst I was working through Dave's recommendation.
It does appear that I have now updated the csplugin.

From the command response I posted do you think that this should do the trick or am I missing something else?

Siv

Both clearsync and rsyslog should have restarted as part of the update if they were already running. This should fix all the events messages about root opening and closing sessions. You can clean the events log completely with:

systemctl stop clearsync.service

rm -f /var/lib/csplugin-events/events.db

systemctl start clearsync.service

It should sort out some of the responsiveness, but I don't think it will do anything about the samba issue and that could also be hammering the events database. Only time will tell.

Nick Howitt wrote:

Both clearsync and rsyslog should have restarted as part of the update if they were already running. This should fix all the events messages about root opening and closing sessions. You can clean the events log completely with:

systemctl stop clearsync.service

rm -f /var/lib/csplugin-events/events.db

systemctl start clearsync.service

It should sort out some of the responsiveness, but I don't think it will do anything about the samba issue and that could also be hammering the events database. Only time will tell.

Nick,

Thanks for the update. I ran these three commands none gave any response so I assume they all completed successfully.
I will watch out for my emailed report tonight and will post a new thread here tomorrow if I am still getting the samba errors.

Siv

Hi All

Thank you for all the effort that was put into resolving this issue.
I was able to leave my server off until today, not everyone can do this, and waited for a solution - i just ran the update from the web interface the moment i booted.
After the update ran, i also did:

systemctl stop clearsync.service
rm -f /var/lib/csplugin-events/events.db
systemctl start clearsync.service

Everything is now responding and i do not see any more log files going "NUTS"

Thank you again for everyone involved

Leon

Nick, Dave,

All seems well today, I have had no pages of log files in my reports, in fact I had no report from the server last night so whatever you guys did it has really quietened things down.
I checked through some of the other logs and nothing appears to be going mad today.
Thanks again for all your help resolving this.


Siv

Thank you Nick for working on my server. Everything is good. I updated my second server today via the webclient and everything looks good this server as well.

There are a few other minor improvements to be made here. Some of this activity is because we changed the way that the ClearOS system interacts with the core and we are using a 'sudo' method which fixes some best practices issues but also made the logging issues. We still have a bunch of useless logging events and we are going to be making targeted filters so that the logs do not rotate like made with useless data that we already know about.

@Siv,
Did your "smbd_audit" messages also disappear? If so, I am not sure that it is anything we fixed and may have been a separate issue. I've not been able to replicate it.

Nick Howitt wrote:

@Siv,
Did your "smbd_audit" messages also disappear? If so, I am not sure that it is anything we fixed and may have been a separate issue. I've not been able to replicate it.

Nick,

No, since the updates the machine has quietened down and I am not getting the same events logged.
Whatever you did fixed that issue as well.

Siv

I have followed this thread and made sure the updates were installed and am still having this problem, thousands instances per hour of
User root logged in via sudo 2019-04-15 18:16:19
User root logged out via sudo 2019-04-15 18:16:19
User root logged in via sudo 2019-04-15 18:16:19
User root logged out via sudo 2019-04-15 18:16:19
User root logged in via sudo 2019-04-15 18:16:19
User root logged out via sudo 2019-04-15 18:16:19
User root logged in via sudo 2019-04-15 18:16:17
User root logged out via sudo 2019-04-15 18:16:17
User root logged in via sudo 2019-04-15 18:16:17
User root logged out via sudo 2019-04-15 18:16:17
It started about 10 days ago when my system updated to 7.6, it gets to the point that it causes the dashboard to give a http 500 error. This morning I had 1.5 million notices before clearing them when I couldn't access the dashboard.

@Brendon,
What is the output from:

rpm -q app-base csplugin-events system-base

The problem largely goes away of you close down your webconfig screen, and some pages are less susceptible than others (IP Settings is terrible). So, to get your system responsive again, come out of the webconfig and wait a while. You may hear your disk activity quieten down.

You can remove all the messages in bulk with:

systemctl stop clearsync.service

rm -f /var/lib/csplugin-events/events.db

systemctl start clearsync.service

Nick Howitt wrote:

@Brendon,
What is the output from:

rpm -q app-base csplugin-events system-base

rpm -q app-base csplugin-events system-base

app-base-2.6.7-2.v7.noarch

csplugin-events-1.2-3.v7.x86_64

system-base-7.6.3-1.v7.x86_64

I don't understand this. Those events are info events so should not be being pushed into the events now. What is the contents of /etc/systemd/system.conf.d/clearos-base.conf and what is the result of:

grep LogLevel /etc/systemd/system.conf

Can you also try the manual command which only works until the next reboot:

systemd-analyze set-log-level notice

This should have been fixed by system-base.

In your secure log are you seeing horrible logging or not? This should have been fixed by app-base, but we will be moving the fix to system-base. Your packages are out of sync here.

Please be careful as you've been updating packages out of clearos-updates-testing. You've pulled the latest app-base from there, so the filtering of secure messages has been dropped and it is now in system-base which you have not pulled. Please do a:

yum update system-base --enablerepo=clearos-updates testing

This should quieten down the secure log but does not affect the events.

Nick Howitt wrote:

I don't understand this. Those events are info events so should not be being pushed into the events now. What is the contents of /etc/systemd/system.conf.d/clearos-base.conf and what is the result of:

grep LogLevel /etc/systemd/system.conf

Can you also try the manual command which only works until the next reboot:

systemd-analyze set-log-level notice

This should have been fixed by system-base.

In your secure log are you seeing horrible logging or not? This should have been fixed by app-base, but we will be moving the fix to system-base. Your packages are out of sync here.

Please be careful as you've been updating packages out of clearos-updates-testing. You've pulled the latest app-base from there, so the filtering of secure messages has been dropped and it is now in system-base which you have not pulled. Please do a:

yum update system-base --enablerepo=clearos-updates testing

This should quieten down the secure log but does not affect the events.

# grep LogLevel /etc/systemd/system.conf
#LogLevel=info

I am getting a ton of these sudo info logging in my secure log
Apr 16 11:45:02 GW sudo: pam_unix(sudo:session): session opened for user root by (uid=0)
Apr 16 11:45:02 GW sudo: pam_unix(sudo:session): session closed for user root
Apr 16 11:45:03 GW sudo: pam_unix(sudo:session): session opened for user root by (uid=0)
Apr 16 11:45:03 GW sudo: pam_unix(sudo:session): session closed for user root

I will try updating the system base from testing and see if that helps, thanks!

system-base will reduce the logging into /var/log/secure but not the events into the events manager (which is the killer). You will need to get the LogLevel=notice to work.

Looks like I'm having the same issues as Brendon (User root logged in and out via sudo continuously)

Results of what you asked above:
app-base-2.6.9-2.v7.noarch
csplugin-events-1.2-3.v7.x86_64
system-base-7.6.5-1.v7.x86_64

Are you sure your messages are current? Pleas can you clear your messages database with a:

systemctl stop clearsync.service

rm -f /var/lib/csplugin-events/events.db

systemctl start clearsync.service

Then have another look.

Jeff Rynders wrote:

Looks like I'm having the same issues as Brendon (User root logged in and out via sudo continuously)

Results of what you asked above:
app-base-2.6.9-2.v7.noarch
csplugin-events-1.2-3.v7.x86_64
system-base-7.6.5-1.v7.x86_64

Hi Jeff, I have not found a solution thus far, it is still logging sudo events like crazy. It is not as bad if you don't have webconfig open in a browser but it does not stop it. I'm wondering how concerned I should be with it thrashing logs on an SSD drive, but luckily it is running in a VM that is backed up frequently.

@Brendon and Jeff,
Please can you raise a General Enquiry ticket at https://secure.clearcenter.com/portal/ticket_manage.jsp and submit remote log in credentials to https://secure.clearcenter.com/portal/system_password.jsp and I'll see if someone can have a look tomorrow. This will be done free of charge.

I have investigated Jeff's system and the root cause could be faced by a few people.
Csplugin-events updated during the big 7.6 update. The only thing it did was to replace /etc/clearos/events.d/20-user-auth.conf which added a filter for the two events you were seeing. In Jeff's case he had edited the file in early 2017. Unfortunately, in the rpm system this file is flagged as a config file, so if yum notices the conf file has been edited, it does not replace it, but creates the new one with an rpmnew extension when it updates.

In this sort of case the user should really regularly go through all rpmnew files ("locate rpmnew") and make a judgement call about them in case they need to make adjustments to their edited file. In this case I moved the original file to /root and renamed the rpmnew file to /etc/clearos/events.d/20-user-auth.conf and restarted clearsync. For the sake of cleanliness I also removed the old events database.

What should now be done is to compare the old /root/20-user-auth.conf to the new /etc/clearos/events.d/20-user-auth.conf and see if there are any changes in the old file which you want to bring into the new file.

In terms of Clearcenter, we could consider making the changes differently and adding an upgrade script to add the new lines to the old file to cover the case that it has been edited. This is possible and will be discussed in a technical meeting, but the file structure is no so neat for inserting lines programatically.

I'm still getting the same issue - syslog is swamped by large numbers of "sudo" events, mostly coming from webconfig so the rate of entries depends on what is happening there. Not enough to affect system performance but makes it very hard to spot "real" events. Seems to have been going on for some time though is causing annoyance now as it is making syslog difficult to use for other purposes.

System is fully updated as far as I am aware and no custom edits made to any of the config files mentioned above. Specifically, the three main packages involved are at or above the level required:

Installed Packages

app-base.noarch                                                            1:2.7.4-1.v7

csplugin-events.x86_64                                                     1.2-3.v7

system-base.x86_64                                                         7.6.5-1.v7

• File /etc/systemd/system.conf has "LogLevel=info" but that line (like all in section [Manager]) is commented out.
  
• File /etc/systemd/system.conf.d/clearos-base.conf contains a single entry "LogLevel=notice"
  
• File /etc/clearos/events.d/20-user-auth.conf hasn't been edited in any way, and is dated 09-Apr-2019.
  

I tried the command

systemd-analyze set-log-level notice

in the current session but that makes no difference.

Have I missed out on something here?

Thanks,
Andrew

PS: I inadvertently replied to this post when I was trying to edit it - can someone please delete my reply below?

@ Andrew,
While this is going on, please do not sit in the Webconfig IP Settings screen as it will massively ramp up the number of events and could bring your system to its knees.
Please can you check if you have a /etc/clearos/events.d/20-user-auth.conf.rpmnew? If so, you've probably made an edit to the /etc/clearos/events.d/20-user-auth.conf and you now need to rationalise the two files (easiest is to move the /etc/clearos/events.d/20-user-auth.conf out of the way and rename the rpmnew file), then, in slow time, work out any edits you made to the original file and decide if you want them in the new file. Then reset the events with:

systemctl stop clearsync.service

rm -f /var/lib/csplugin-events/events.db

systemctl start clearsync.service

Andrew Wasielewski wrote:

I'm still getting the same issue - syslog is swamped by large numbers of "sudo" events, mostly coming from webconfig so the rate of entries depends on what is happening there. Not enough to affect system performance but makes it very hard to spot "real" events. Seems to have been going on for some time though is causing annoyance now as it is making syslog difficult to use for other purposes.

System is fully updated as far as I am aware and no custom edits made to any of the config files mentioned above. Specifically, the three main packages involved are at or above the level required:
[code type="markup"]Installed Packages
app-base.noarch 1:2.7.4-1.v7
csplugin-events.x86_64 1.2-3.v7
system-base.x86_64 7.6.5-1.v7

• File /etc/systemd/system.conf has "LogLevel=info" but that line (like all in section [Manager]) is commented out.
  


• File /etc/systemd/system.conf.d/clearos-base.conf contains a single entry "LogLevel=notice"
  


• File /etc/clearos/events.d/20-user-auth.conf hasn't been edited in any way, and is dated 09-Apr-2019.
  

I tried the command systemd-analyze set-log-level notice in the current session but that makes no difference.

Have I missed out on something here?

Thanks,
Andrew

Hi Nick,

The IP Settings screen certainly does generate a lot of events... There are no .rpmnew files, and none of the three config files have been edited. I was under the impression from the message trail that the underlying cause had been addressed by package updates 2-3 months back and the only issue was whether these had been applied successfully or not. My system is updated automatically and all packages are currently up to date.

Let me know if there is anything else in config I can check or any other testign I can do.

Andrew

I did not read you post properly. I thought you were talking about the events log. Please can you post a log snippet?

[edit]
.... and confirm which log it is coming from?
[/edit]

This is the sort of log entry I get in /var/log/messages (from journalctl -f):

Jul 27 21:41:02 gateway.wasielewski sudo[3535]: pam_unix(sudo:session): session opened for user root by (uid=0)

Jul 27 21:41:02 gateway.wasielewski sudo[3535]: pam_unix(sudo:session): session closed for user root

Jul 27 21:41:02 gateway.wasielewski sudo[3538]: pam_unix(sudo:session): session opened for user root by (uid=0)

Jul 27 21:41:02 gateway.wasielewski sudo[3538]: pam_unix(sudo:session): session closed for user root

Jul 27 21:41:02 gateway.wasielewski sudo[3541]: pam_unix(sudo:session): session opened for user root by (uid=0)

Jul 27 21:41:02 gateway.wasielewski sudo[3541]: pam_unix(sudo:session): session closed for user root

Not identical to all the other log entries that have been reported, but definitely generated by any kind of root access, especially via the webconsole.

Those should be filtered out by an rsyslog filter, /etc/rsyslog.d/system-base.conf which is provided by system-base-7.6.5-1.v7, which you have. Can you check you have the file? If you do and have not rebooted recently, please reboot to get the latest kernel into operation (the kernel has nothing to do with the issue but it is a security release). Obviously this will also restart the rsyslog daemon.

Hi Nick,
I have the correct version of the system-base package and the /etc/rsyslog.d/system-base.conf file exists - datestamp Apr-12-2019 15:26, bytecount 290, and content appears to do exactly what is required. I have rebooted several times in last few days (though had not rebooted prior to that for about 4 months) and also restarted rsyslog service just now. Still makes no difference so not clear what is going wrong...

Thanks,
Andrew

I don't understand, then, what is going wrong. If you restart rsyslog are there any error messages (Don't worry about ~ being deprecated, but you can change each instance to "stop" without the quotes if you want).

The rsyslog filter will not stop the messages from showing in "journalctl -xe" but should stop anything from appearing in the logs.

There is one more thing you could try. In /etc/pam.d/system-auth-ac change:

session     [success=1 default=ignore] pam_succeed_if.so service in sudo quiet uid = 0 ruser = webconfig

to

session     [success=1 default=ignore] pam_succeed_if.so service in crond quiet use_uid

session     [success=1 default=ignore] pam_succeed_if.so service in sudo quiet uid = 0 ruser = webconfig

I am not sure what else this affects.

This is not a file maintained by ClearOS so could get stomped on with an updated. I't prefer to track down why the rsyslog filter is not working.

Ah ha - I think the problem is how I am looking at the logs. The "sudo" messages are indeed not going into /var/log/messages. However they do show up in "journalctl -xe", and "journalctl -f" which what I use to keep an eye on logs. I used to use "tail -f /var/log/messages" though I think that got deprecated in favour of journalctl, which is indeed more flexible in most cases. I'll have a dig as there is probably a way to apply the same kind of filter there.

Apologies for my lack of understanding of how logging and viewing of logs works.

You can apply the edit to /etc/pam.d/system-auth-ac, but I don't know how to do it with multiple filters. It works with the sudo messages.

[edit]
I believe all rsyslog does is take the raw messages from journalctl, filter them and distribute them to the various log files..
[/edit]