Gordon came into support and what we found was that a restore attempt earlier had made it so that our fix doesn't work because the restore killed all his LDAP database. Basically, if you have a compromised system and attempt to recover from a backup configuration it will try to restore with the failed slapd.conf file and will error out. Unfortunately, it will have already dropped your users from your LDAP database in the process so if you fix the slapd.conf file and start 'slapd' it will run but have no users in the database.

The process is a bit advanced to recover from this so if anyone has this particular nuance of attempting to recover from the backup config and failing, please hop into support for ClearCARE and let us help you fix it. Basically what you have to do is to create a merged recovery file in Remote Configuration Backup of a previous backup and the fixed slapd.conf.

You create a directory.
Copy the configuration backup file you want to restore from.
Extract it (tar xvcf filename) and move the archive tgz file away
Copy the correct slapd.conf file (with the Kopano fixes) into the right place in the archive
Validate the permissions (ldap:ldap)
Archive up the file (cd /place/you/extracted/the/file && tar /tmp/ldaprecovery.tgz -cvzf *)
Then move the ldap recovery file back to /var/clearos/configuration_backup
Then go back into Webconfig and restore it.

If you need us to do this for you, please open a ticket and we can get it resolved for you.

Gordon Howell wrote:
What else can I do?

Raise a ticket (a general enquiry ticket)

Remember to make sure you have a strong password before you open the incoming firewall to SSH.

Dave Loper wrote:

My bad...

systemctl restart slapd

OK, so SLAPD has restarted, can see it in the web interface running, however, still no user accounts and SMB will not start, exits with a message that control process exited with an error code.

[root@server openldap]# systemctl restart smb
Job for smb.service failed because the control process exited with error code. See "systemctl status smb.service" and "journalctl -xe" for details.

Running the SLAPD status command returns the following:

slapd.service - OpenLDAP Server Daemon
Loaded: loaded (/usr/lib/systemd/system/slapd.service; enabled; vendor preset: disabled)
Active: active (running) since Fri 2017-11-03 15:26:59 EDT; 1min 12s ago
Docs: man:slapd
man:slapd-config
man:slapd-hdb
man:slapd-mdb
file:///usr/share/doc/openldap-servers/guide.html
Process: 6998 ExecStart=/usr/sbin/slapd -u ldap -h ${SLAPD_URLS} $SLAPD_OPTIONS (code=exited, status=0/SUCCESS)
Process: 6980 ExecStartPre=/usr/libexec/openldap/prestart.sh (code=exited, status=0/SUCCESS)
Main PID: 7000 (slapd)
CGroup: /system.slice/slapd.service
└─7000 /usr/sbin/slapd -u ldap -h ldap://127.0.0.1/

Nov 03 15:26:59 server.wa4rts.net systemd[1]: Starting OpenLDAP Server Daemon...
Nov 03 15:26:59 server.wa4rts.net prestart.sh[6980]: Configuration directory '/etc/openldap/slapd.d' does not exist.
Nov 03 15:26:59 server.wa4rts.net prestart.sh[6980]: Warning: Usage of a configuration file is obsolete!
Nov 03 15:26:59 server.wa4rts.net runuser[6984]: pam_unix(runuser:session): session opened for user ldap by (uid=0)
Nov 03 15:26:59 server.wa4rts.net runuser[6984]: pam_unix(runuser:session): session closed for user ldap
Nov 03 15:26:59 server.wa4rts.net slapd[6998]: @(#) $OpenLDAP: slapd 2.4.44 (Aug 12 2017 06:10:11) $
[email protected]:/builddir/build/BUILD/openldap-2.4.44/openldap-2.4.44/servers/slapd
Nov 03 15:26:59 server.wa4rts.net systemd[1]: Started OpenLDAP Server Daemon.

So it appears that despite warnings, the OpenLDAP daemon is running. I've restarted the system to ensure that the latest build is running.

Ownership of the SLAPD.CONF file is correct. What else can I do?

Thank you.

Robert wrote:

Dear Dave,

I removed the folder as you said and put in the config file (slapd.conf) as you wrote in the other thread. I replaces the domain names and the ldap starts now, but samba, zarafa, ... do still now work ... .

You wrote in the other thread:

Password: Must have the valid password (for slapd.conf)

What does that mean? Where do I get that?

Thank you.

Best wishes,

Robert

The valid password can be found in /var/clearos/openldap/config.php - use the bind_pw_hash.

You can validate that 'slapd' (the database behind openldap) is running with:

systemctl status slapd

Also, you can fill out the support ticket as 'General Inquiry' if needed.

Gordon,

Go to https://secure.clearcenter.com/portal/ticket_manage.jsp to submit a support ticket.

My bad...

systemctl restart slapd

In attempting to do the manual reset, the command "systemctl restart openldap" returns "unit not found" as though the openldap server is gone.

I tried submitting a ticket through the server interface unsuccessfully as I don't have incident support. I'd prefer to do it that way if you can tell me how to submit a ticket. Thanks.

Gordon,

Please open a ticket and include your phone number. I will call you with more instructions.

https://secure.clearcenter.com/portal/system_password.jsp

Or try as Robert suggests:

cp /etc/openldap/slapd.conf/bak /etc/openldap/slapd.conf
chown ldap:ldap /etc/openldap/slapd.conf
systemctl restart slapd

Dear Nick, Dave and Patrick,

Using the /etc/openldap/slapd.conf.bak file did the trick. I just needed to add:

# Kopano extension
include /etc/openldap/schema/kopano.schema

+ rename it to slapd.conf and make it owned by ldap:ldap. I think this will also solve Gordon's problem.

Thanks again.

Best wishes,

Robert

Nick Howitt wrote:

Have you tried Dave's fix in the other thread you posted to?

Yes. Folder removed, file modified. SLAPD will start only if I modify the original .BAK file to add the new schema, keeping original passwords, and when it does, the user accounts are not found, and none of the email users can authenticate. I apologize for cross posting in search of an answer. I came across this post after I put in another query.

Gordon,

A restore from backup will not necessarily fix the issue. From what we are seeing, the configuration files have typically been corrupted months prior to the event. Meaning, it looks like the problem existed from a long time ago and the update merely restarts openldap and exposes the issue.

The steps to fix are, ether:

Open up a ClearCare support ticket and give remote access to the technician who can log in and fix the issue for you over SSH. (One of my team members or I will fix this issue for ANYONE to opens a ClearCARE ticket with this issue as a remote SSH support session.)

-or-

Manually fix the issue

-You must remove/move the /etc/openldap/slapd.d directory introduced by the CentOS patch
---ie. mv /etc/openldap/slapd.d /tmp/

-You must remove/move the slapd.conf introduced by the CentOS patch
--You can determine that this file is the borked one because it will NOT have the ClearOS schema at the start of the file
---ie. mv /etc/openldap/slapd.conf /tmp/

-You must restore a valid slapd.conf file. This is typically found in /etc/openldap/slapd.conf.bak
--ie. cp /etc/openldap/slapd.conf.bak /etc/openldap/slapd.conf

-If your recovered slapd.conf file does NOT have the kopano schema, add it back in after Zarafa and before the 'Global configuration directives'
--# Kopano extension
--include /etc/openldap/schema/kopano.schema

-You must ensure that the 'ldap' account is the owner of the slapd.conf file
--ie. chown ldap:ldap /etc/openldap/slapd.conf

-You must restart openldap
--systemctl restart openldap
--systemctl status openldap

-If you rebooted your server in an attempt to fix the issue and are running Samba, you must restart the samba services
--systemctl restart smb
--systemctl restart nmb
--systemctl restart winbind

If you have a file called /etc/openldap/slapd.conf.bak, use that. It is very close to being accurate. On some systems it does not have the Kopano extensions but this is easily added back in:

############LOOKS LIKE THIS IN THE FILE################
# Zarafa extension
include /etc/openldap/schema/zarafa.schema

# Kopano extension
include /etc/openldap/schema/kopano.schema


# Global configuration directives
#####################################################

Failing that and you need to rebuild from scratch, you can use the template I provided and the information in this file to rebuild the domain name and the password:

/var/clearos/openldap/config.php

The password must already been set in the "old" slapd.conf.
Only replace the Schemes in to the slapd.conf with the Schemes Dave has posted.

Robert wrote:

Dear Dave,

I removed the folder as you said and put in the config file (slapd.conf) as you wrote in the other thread. I replaces the domain names and the ldap starts now, but samba, zarafa, ... do still now work ... .

You wrote in the other thread:

Password: Must have the valid password (for slapd.conf)

What does that mean? Where do I get that?

Thank you.

Best wishes,

Robert

This worked out for me.

Thanks Dave

Dave Loper wrote:

Some systems were running a patch that originated from CentOS that borks their LDAP. The 3 steps to validate are:

1) Move the /etc/openldap/slapd.d folder out of the way. (The CentOS RPM creates this folder)
mv /etc/openldap/slapd.d /tmp/

2) Make sure ........................

Dear Dave,

I removed the folder as you said and put in the config file (slapd.conf) as you wrote in the other thread. I replaces the domain names and the ldap starts now, but samba, zarafa, ... do still now work ... .

You wrote in the other thread:

Password: Must have the valid password (for slapd.conf)

What does that mean? Where do I get that?

Thank you.

Best wishes,

Robert

One more report back and I am done until somebody fixes this problem. SLAPD did restart after I changed the conf file. When I restarted the server, it allowed me access to the account manager, however, it says "No Object" when I try to look at users or groups. I tried once again to load a backup from several days ago, and it quit at line 63 of the SLAPD.CONF file, which, when I looked to see what the problem was, turned out to be the "new" file without the schema information included. It was referencing a directory in /VAR which isn't there. I rebooted the server and went back a version to no avail.

I hope someone at CLEAROS is looking at this problem and will put out a fix shortly, as my email server is not functioning at all, since it can't verify passwords. All was fine here until about 10:30 this morning when a couple of more autoupdate files hit. (US Eastern Daylight time...)

Geep

Have you tried Dave's fix in the other thread you posted to?

This is becoming quite frustrating. I found that modifying the .BAK file for SLAPD.CONF with the additional information needed to update the file allowed SLAPD to start. However, the account manager is now saying my credentials are invalid to access it. I can log in and out of the dashboard interface with root credentials, however, SMB will not start and the mail server doesn't recognize any of the passwords that it should and is offline. I have not restarted the CLEAROS server, guess that is the next thing to try, but am then concerned that I will lose connectivity to see answers about solving this issue. Is there a way to just rollback the latest changes until this issue gets solved? Unhappy.

It looks as though at least some of the problem is that the latest slapd.conf file contains ONLY an include reference to core.schema. None of the others are found in the file, at least on my system. I am going to try copying them in and see if that fixes the problem. There was a slapd.d folder, which I have removed to no avail.

Some systems were running a patch that originated from CentOS that borks their LDAP. The 3 steps to validate are:

1) Move the /etc/openldap/slapd.d folder out of the way. (The CentOS RPM creates this folder)
mv /etc/openldap/slapd.d /tmp/

2) Make sure that you have a valid slapd.conf file in the /etc/openldap/ directory. This file should have (at least) all of the schema:

# Schemas
#----------------------------------------------------------

# Core schemas
include /etc/openldap/schema/core.schema
include /etc/openldap/schema/cosine.schema
include /etc/openldap/schema/inetorgperson.schema

# ClearFoundation base
include /etc/openldap/schema/rfc2307bis.schema
include /etc/openldap/schema/clearfoundation.schema

# ClearCenter extension
include /etc/openldap/schema/clearcenter.schema

# Password policy extension
include /etc/openldap/schema/ppolicy.schema

# RADIUS extension
include /etc/openldap/schema/RADIUS-LDAPv3.schema

# Kolab extension
include /etc/openldap/schema/rfc2739.schema
include /etc/openldap/schema/kolab2.schema

# Horde extension
include /etc/openldap/schema/horde.schema

# Samba extension
include /etc/openldap/schema/samba3.schema

# OwnCloud
include /etc/openldap/schema/owncloud.schema

# Zarafa extension
include /etc/openldap/schema/zarafa.schema

# Kopano extension
include /etc/openldap/schema/kopano.schema

The Kopano is new so if the file you have doesn't have all of these and the Kopano as well, please find a valid copy in backup or as one of the files in this directory. Restore this file to its proper place.

3) Validate that the slapd.conf file is owned by ldap. If not, run:
chown ldap:ldap /etc/openldap/slapd.conf

NOTE: You can see a template copy of slapd.conf in this post: https://sfj48-fkj200.heiksthsd.cf/clearfoundation/social/community/account-manager-won-t-start,-slapd,-smb-fail-to-start-on-reboot

In the cases where this was a reported issue on Community, users had enabled (at some point in time) the updates for CentOS and had installed LDAP packages from CentOS that seeded the problem. In this case, it was evidenced by a 'slapd.d' directory in /etc/openldap.

Guys,

I'm having the same problem.
LDAP is offline after the update.

Hi Nick,

Sorry, forgot that. It is:

# Network mode

MODE="standalone"



# Network interface roles

EXTIF="enp0s20f0"

LANIF=""

DMZIF=""

HOTIF=""



# Domain and Internet Hostname

DEFAULT_DOMAIN="email.com"

INTERNET_HOSTNAME="server.email.com"



# Extra LANS

EXTRALANS=""



# ISP Maximum Speeds

Thanks.

Best wishes,

Robert

I don't think we are going to get anywhere with this as it may more fundamental as you note. Can you still give the contents of /etc/clearos/network.conf?

Hi Nick,

Maybe another hint. When I run journalctl -xe:

-- Unit slapd.service has begun starting up.

runuser[10204]: pam_unix(runuser:session): session opened for user ldap by (uid=0)

runuser[10204]: pam_unix(runuser:session): session closed for user ldap

prestart.sh[10200]: Checking configuration file failed:

prestart.sh[10200]: 59fc404a User Schema load failed for attribute "pwdMaxRecordedFailure". Error code 17: attribute type undefined

prestart.sh[10200]: 59fc404a config error processing olcOverlay={0}ppolicy,olcDatabase={3}bdb,cn=config: User Schema load failed for att

prestart.sh[10200]: slaptest: bad configuration file!

nslcd[12291]: [69e373] <passwd="ldap.ldap"> failed to bind to LDAP server ldap://127.0.0.1/: Can't contact LDAP server: Transport endpoi

nslcd[12291]: [69e373] <passwd="ldap.ldap"> no available LDAP server found: Can't contact LDAP server: Transport endpoint is not connect

slapd[10219]: @(#) $OpenLDAP: slapd 2.4.44 (Aug 12 2017 06:10:11) $

                                                      [email protected]:/builddir/build/BUILD/openldap-2.4.44/openldap-2.4.44/servers/slapd

slapd.service: control process exited, code=exited status=1

Failed to start OpenLDAP Server Daemon.

-- Subject: Unit slapd.service has failed

I looked into prestart.sh, but I do not really know what to look for. The User Schema seems to be off:

#!/bin/sh



# Run the upstream pre-start script

/usr/libexec/openldap/check-config.sh



# Load sysconfig parameters

[ ! -e /etc/sysconfig/slapd ] && exit 0

source /etc/sysconfig/slapd



[ -z "$SLAPD_URLS" ] && exit 0

[ -z "$BIND_POLICY" ] && exit 0



# Ugh - this permission issue catches end users all the time

chown -R ldap.ldap /var/lib/ldap



# Set SLAPD_URLS based on BIND_POLICY

if [ "$BIND_POLICY" == "localhost" ]; then

    # Do not bind to localhost:389 if Samba Directory is configured to run

    SAMBA_DIRECTORY_CONFIGURED=`grep "^driver[[:space:]]*=[[:space:]]*samba_directory" /var/clearos/accounts/config 2>/dev/null`

    if [ -n "$SAMBA_DIRECTORY_CONFIGURED" ]; then

        urls="ldaps://127.0.0.1/"

    else

        urls="ldap://127.0.0.1/"

    fi

elif [ "$BIND_POLICY" == "lan" ]; then

    urls="ldap://127.0.0.1/"



    if [ -x '/usr/sbin/network' ]; then

        lan_ips=`/usr/sbin/network --get-lan-ips | grep -v debug`

    else

        lan_ips=""

    fi



    for ip in $lan_ips; do

        urls="$urls ldaps://$ip/"

    done

elif [ "$BIND_POLICY" == "all" ]; then

    urls="ldap://127.0.0.1/ ldaps:///"

else

    exit 0

fi

if [ "$SLAPD_URLS" != "$urls" ]; then

    urls_sed=`echo $urls | sed 's/\//\\\\\//g'`

    sed -i -e "s/^SLAPD_URLS=.*/SLAPD_URLS=\"$urls_sed\"/" /etc/sysconfig/slapd

fi

Thank you.

Best wishes,

Robert

I'm not going to be able to look for a while as I am at work. It probably means trying to figure what the prestart.sh for slapd is looking at and what the slaptest file is, but I have a feeling the slaptest is a binary file or points to one.

[edit]
Please can you have a look at your network.conf file as I mentioned earlier
[/edit]

Hi Nick,

Thanks for your reply. In yum log everything looks OK.

In system I found:

Nov 3 05:30:28 server software-updates: log: Warning: slapd.service changed on disk. Run 'systemctl daemon-reload' to reload units.
==> I tried "systemctl daemon-reload", does not change anything.

In messages I found:
Nov 3 05:30:30 server prestart.sh: slaptest: bad configuration file!
Nov 3 05:30:30 server systemd: slapd.service: control process exited, code=exited status=1
Nov 3 05:30:30 server systemd: Unit slapd.service entered failed state.
Nov 3 05:30:30 server systemd: slapd.service failed.

So the config file got changed during the update. Question is to what and what does not fit anymore. Can I reset the config file or something like that?

Thank you.

Best wishes,

Robert

Can you check you yum.log. You may have received a large update last night as the 7.4 update was pushed into the repos yesterday.

Are there any relevant messages in /var/log/messages or /var/log/system?

Are there any interfaces showing in /etc/clearos/network.conf which don't exist? (e.g in the output of "ifconfig")