I had this issue today after adding another external NIC to the server. Turns out it was caused by me forgetting to untick the use automatic DNS and Clearos defaulting to the DNS from the new ADSL router and note my internal DNS of AD. Simple little thing, much frustration.

Hopefully that may save someone else some time.

By way of follow up in case it helps someone else

Based on Ben's suggestions

In

/usr/lib/systemd/system/webconfig.service

I changed the line starting with

After=network.target

to

After=network.service remote-fs.target nss-lookup.target

And the 'Account System is offline' issue went away after a reboot.

Steve.

There is some interesting stuff on network.target on the web, including this link where they say it should only really be used for shutdown. Waiting for it won't guarantee slapd an IP address if it is trying to publish its services. I don't think network-online.target looks like much help either as what happens if no interfaces come up e.g. during an offline installation. Would that then hang the start up?

Does the network service rely on external factors (such as obtaining DHCP leases)? In that case its start up time could be unpredictable.

I'm just throwing some thoughts into the pot and could be well wide of the mark.

Not to go offtrack, but we know about the arpwatch issue...it's been fixed in version 2.1a15.30.v7.1 and available with:

yum --enablerepo=clearos-updates upgrade arpwatch

Being promoted to veified repos (Home and Business Editions) next week.

B

Cheers Ben

As it may be something connected with network.service, I just thought I would mention that my install sits on virtual machine and gets its IP via dhcp from a COS6.5 virtual machine on the same server. The two setups are using similar hostnames e.g. COS6.5 = server.lan and COS7.2 = second.server.lan.

The only other thing is that I have just had a look at the boot.log and found this in it:

[ OK ] Started ACPI Event Daemon.
[FAILED] Failed to start Arpwatch ethernet/IP address tracking on ens160.
See 'systemctl status [email protected]' for details.

I'm not sure if this helps or hinders or is nothing at all.

Steve.

Thanks Stephen,

See how your nework.service takes a while to start (5.119s) and the completion ends past the end time of the webconfig.service. You are the second instance where we have confirmed this and in both cases, you guys are the ones seeing the 'Account is Offline' message on boot.

I resolved this earlier today with another customer by editing:

/usr/lib/systemd/system/webconfig.service

And modifying the 'After' line so it waits until after network.service instead of network.target and rebooting.

Your confirmation would be another step towards a permanent fix, however, I'd like to better understand the repercussions and need to consult with Peter B (who's currently away until next week). Darryl's also going to spend some cycles to see why this PHP library behaives like it does, because ultimately, anything we do in systemd is a work-around...a solution is trickier and in likely to be as a PHP patch upstream.

B.

doh, just realised how to attach the file

edit: although it wont allow svg files

Ben

I've just emailed over the svg as I couldn't work out how to paste the file properly.

Anyway, I've done your fix as stated and it is now working for me.

Many thanks
Steve

We made some huge progress today in that we were able to find a way to duplicate it.

It all depends on the order of the systemd startup services.

If webconfig comes up too soon before network.service (note, the systemd configlet for webconfig already has an 'after=network.target'), we can make it fail in the same way you guys have reported here (the post boot group).

However, the only way we've been able to make it fail is to hack the systemd configlet for the webconfig service to purposefully start up before. We cannot make it fail with the default systemd files.

If you want to prove you have an issue, run:

systemd-analyze plot > /tmp/startup.svg

Paste the SVG output here or send to [email protected].

We're interested to know how close together the webconfig.service and network.service services are.

As for a fix....a bit of a hack, but we believe editing:

/usr/libexec/webconfig/prestart.sh

Add close to the top of that file, add a "sleep 5":

#!/bin/sh



sleep 5



# Environment

KEY="/usr/clearos/sandbox/etc/httpd/conf/server.key"

...

..

.

This should delay Webconfig service from starting from after network.service has done all its 'bits' and be OK.

The true fix is to see what's really going on the the PHP ldap library...now that we can reproduce, we'll try and investigate and see if we can suggest an upstream patch.

Cheers,

Ben

Made the change and the problem is still there, this was the latest error log output:

[Tue Aug 02 15:11:18.762603 2016] [mpm_prefork:notice] [pid 808] AH00170: caught SIGWINCH, shutting down gracefully
[Tue Aug 02 15:11:40.616333 2016] [suexec:notice] [pid 729] AH01232: suEXEC mechanism enabled (wrapper: /usr/clearos/sandbox/usr/sbin/suexec)
[Tue Aug 02 15:11:40.667662 2016] [ssl:warn] [pid 729] AH02292: Init: Name-based SSL virtual hosts only work for clients with TLS server name indication support (RFC 4366)
[Tue Aug 02 15:11:40.726412 2016] [auth_digest:notice] [pid 729] AH01757: generating secret for digest authentication ...
[Tue Aug 02 15:11:40.727738 2016] [lbmethod_heartbeat:notice] [pid 729] AH02282: No slotmem from mod_heartmonitor
[Tue Aug 02 15:11:40.731520 2016] [ssl:warn] [pid 729] AH02292: Init: Name-based SSL virtual hosts only work for clients with TLS server name indication support (RFC 4366)
[Tue Aug 02 15:11:42.066963 2016] [mpm_prefork:notice] [pid 729] AH00163: Apache/2.4.6 (ClearOS) OpenSSL/1.0.1e-fips configured -- resuming normal operations
[Tue Aug 02 15:11:42.067006 2016] [core:notice] [pid 729] AH00094: Command line: '/usr/sbin/webconfig -D FOREGROUND'
ldap_url_parse_ext(ldap://localhost/)
ldap_init: trying /etc/openldap/ldap.conf
ldap_init: using /etc/openldap/ldap.conf
ldap_init: HOME env is NULL
ldap_init: LDAPCONF env is NULL
ldap_init: LDAPRC env is NULL
ldap_create
ldap_url_parse_ext(ldap://127.0.0.1)
ldap_bind_s
ldap_simple_bind_s
ldap_sasl_bind_s
ldap_sasl_bind
ldap_send_initial_request
ldap_new_connection 1 1 0
ldap_int_open_connection
ldap_connect_to_host: TCP 127.0.0.1:389
ldap_connect_to_host: getaddrinfo failed: Address family for hostname not supported
ldap_err2string

Then repeating the same as before

Steve

Thanks Stephen,

If you edit /usr/lib/systemd/system/webconfig.service

and change this line:

After=network.target remote-fs.target nss-lookup.target

To:

After=network.target remote-fs.target nss-lookup.target dnsmasq.service

And you reboot, does it resolve the 'Offline' issue on reboot? Not saying this is an eloquent or 'goto' solution...just trying to understand which race condition we're up against in the boot up sequence to get around this.

B.

Well I removed the fail2ban reload just in case but no joy. I now have an error log with over 8,000 lines of the same as above. Just before the repeating lines the log shows:

[Tue Aug 02 14:34:37.831914 2016] [suexec:notice] [pid 808] AH01232: suEXEC mechanism enabled (wrapper: /usr/clearos/sandbox/usr/sbin/suexec)
[Tue Aug 02 14:34:37.849228 2016] [ssl:warn] [pid 808] AH02292: Init: Name-based SSL virtual hosts only work for clients with TLS server name indication support (RFC 4366)
[Tue Aug 02 14:34:37.883966 2016] [auth_digest:notice] [pid 808] AH01757: generating secret for digest authentication ...
[Tue Aug 02 14:34:37.884785 2016] [lbmethod_heartbeat:notice] [pid 808] AH02282: No slotmem from mod_heartmonitor
[Tue Aug 02 14:34:37.887015 2016] [ssl:warn] [pid 808] AH02292: Init: Name-based SSL virtual hosts only work for clients with TLS server name indication support (RFC 4366)
[Tue Aug 02 14:34:39.600969 2016] [mpm_prefork:notice] [pid 808] AH00163: Apache/2.4.6 (ClearOS) OpenSSL/1.0.1e-fips configured -- resuming normal operations
[Tue Aug 02 14:34:39.601009 2016] [core:notice] [pid 808] AH00094: Command line: '/usr/sbin/webconfig -D FOREGROUND'
ldap_url_parse_ext(ldap://localhost/)
ldap_init: trying /etc/openldap/ldap.conf
ldap_init: using /etc/openldap/ldap.conf
ldap_init: HOME env is NULL
ldap_init: LDAPCONF env is NULL
ldap_init: LDAPRC env is NULL
ldap_create
ldap_url_parse_ext(ldap://127.0.0.1)
ldap_bind_s
ldap_simple_bind_s
ldap_sasl_bind_s
ldap_sasl_bind
ldap_send_initial_request
ldap_new_connection 1 1 0
ldap_int_open_connection
ldap_connect_to_host: TCP 127.0.0.1:389
ldap_connect_to_host: getaddrinfo failed: Address family for hostname not supported
ldap_err2string

Hopefully this might help track down the issue. Anyway I am going to reinstall and see what happens, I'll let you know if the problem reoccurs.

Steve

Hi Ben

I don't know if this helps or not, but I now have 2,000 lines in the error log repeating this:-

ldap_err2string
ldap_bind_s
ldap_simple_bind_s
ldap_sasl_bind_s
ldap_sasl_bind
ldap_send_initial_request
ldap_new_connection 1 1 0
ldap_int_open_connection
ldap_connect_to_host: TCP 127.0.0.1:389
ldap_connect_to_host: getaddrinfo failed: Address family for hostname not supported
ldap_err2string
ldap_bind_s
ldap_simple_bind_s
ldap_sasl_bind_s
ldap_sasl_bind
ldap_send_initial_request
ldap_new_connection 1 1 0
ldap_int_open_connection
ldap_connect_to_host: TCP 127.0.0.1:389
ldap_connect_to_host: getaddrinfo failed: Address family for hostname not supported
ldap_err2string


This is a brand new COS 7.2 (business - evaluation) install. Installed three days ago behind a COS 6.5 box (which provides the 7.2 setup with its IP address) the only changes I have made in the 7.2 setup are:-
Installed Webmin and applied bug file 9541 workaround (reload fail2ban).

Steve

I'm still not sure if this issue is behind us or not...one one hand, looks like the update (which closed open LDAP connections when a PHP class is done with them) looks to have permanently solved one customer I worked with.

However, yesterday we got a report of "Account is offline" issue whenever the server was rebooted...I can't think that open connections could be an issue when a server is rebooted, so I am at a loss to explain yesterday's report...though I could not and did not confirm it.

If anyone coming across this thread has this issue and can reproduce it repeatedly, can you I ask you to do one of two things (or both);

1. In /usr/clearos/apps/ldap/libraries/LDAP_Client.php

Look for the function/line:

protected function _bind($mode)

{

And add to the top of that function so it looks like this:

protected function _bind($mode)

    {

        clearos_profile(__METHOD__, __LINE__);



        ldap_set_option(NULL, LDAP_OPT_DEBUG_LEVEL, 7);

Don't restart webconfig...that will temporarily fix your issue...just reload the page that uses the LDAP PHP library (eg. the users page @ /app/users).

Once done, grab your /var/log/webconfig/error_log file and post that last 20+ lines of debug output while you're seeing the "Account is offline" message.

2. Create a support ticket here and I will work with you free of charge, regardless of what edition you're running if you are OK with spending some cycles with me to reproduce the issue and get this debug off the system in hopes of moving this bug off permanently.

Thanks,

Ben

Just a quick follow up to the 'Account is offline' issue...my hope is that this issue is now permanently resolved with an update to app-ldap (version 2.1.7-1) with the fix suggested in the bug tracker here.

I say 'hope' because it was an intermittent bug..however, the customer I've been working with used to see it pop up 3-4 times a week, and has now had over three weeks without seeing it once.

B.

For anyone coming across this thread and if you're running v7, first thing to try if you ever see this message is:

service webconfig restart

It may just be the ldap PHP extension inside Webconfig acting up...we've not been able to track it down.

B.

pete910 wrote:

 systemctl smb stop

 systemctl nmb stop

 systemctl winbind stop

 systemctl nscd stop

 systemctl stop smb

 systemctl stop nmb

 systemctl stop winbind

 systemctl stop nscd

 systemctl stop nslcd

 systemctl stop slapd

Then do

rm /var/clearos/accounts/initialized  /var/clearos/accounts/config  /var/clearos/ldap/initialized  /var/clearos/openldap/config.php /var/clearos/mode/mode.conf /var/clearos/samba/init* -fv



cd /var/lib/ldap



rm -df *

then I did

yum reinstall webconfig-php-ldap-5.4.16-36.v7.1.x86_64 app-ldap-core-2.1.6-1.v7.noarch app-openldap-core-2.1.7-1.v7.noarch app-openldap-directory-core-2.1.6-1.v7.noarch openldap-clients-2.4.40-8.v7.x86_64 openldap-2.4.40-8.v7.x86_64 php-ldap-5.4.16-36.el7_1.x86_64 nss-pam-ldapd-0.8.13-8.el7.x86_64 openldap-servers-2.4.40-8.v7.x86_64 freeradius-ldap-3.0.4-6.el7.x86_64

Then reboot,
DO NOT install/initialise the Directory server it will fail again after you've initialised. That's what was causing it on my end, There has been updates since so dont know if its a unknown/weird issues that's now fixed.

This question takes me two days. Then I found if you modify the .ini file in /usr/clearos/sandbox/etc/php.d let php service reload, everything will be fine untill next reboot.
So ,I think the reason is the service startup order.

Rodrigo Lozada wrote:

Thanks for the info.

One question, what do you mean by "Dot not install the directory server" ?

That is precisely what I'm trying to do. I want to manage the users on the system but it fails.

Sorry if I misunderstood you.

I should have been more clear, sorry.

I mean the "Directory server app" via the market place/store.

The accounting system uses ldap fine, as soon as I installed the "Directory server app" from the market in caused problems for me and ended up with "accounting system offline"

If you don't need remote access to the directory it's not needed

HTH

Thanks for the info.

One question, what do you mean by "Dot not install the directory server" ?

That is precisely what I'm trying to do. I want to manage the users on the system but it fails.

Sorry if I misunderstood you.

 systemctl smb stop

 systemctl nmb stop

 systemctl winbind stop

 systemctl nscd stop

 systemctl stop smb

 systemctl stop nmb

 systemctl stop winbind

 systemctl stop nscd

 systemctl stop nslcd

 systemctl stop slapd

Then do

rm /var/clearos/accounts/initialized  /var/clearos/accounts/config  /var/clearos/ldap/initialized  /var/clearos/openldap/config.php /var/clearos/mode/mode.conf /var/clearos/samba/init* -fv



cd /var/lib/ldap



rm -df *

then I did

yum reinstall webconfig-php-ldap-5.4.16-36.v7.1.x86_64 app-ldap-core-2.1.6-1.v7.noarch app-openldap-core-2.1.7-1.v7.noarch app-openldap-directory-core-2.1.6-1.v7.noarch openldap-clients-2.4.40-8.v7.x86_64 openldap-2.4.40-8.v7.x86_64 php-ldap-5.4.16-36.el7_1.x86_64 nss-pam-ldapd-0.8.13-8.el7.x86_64 openldap-servers-2.4.40-8.v7.x86_64 freeradius-ldap-3.0.4-6.el7.x86_64

Then reboot,
DO NOT install/initialise the Directory server it will fail again after you've initialised. That's what was causing it on my end, There has been updates since so dont know if its a unknown/weird issues that's now fixed.

pete910, can you share what you did to get it working? that would be the change from service to systemctl and ldap dir contents and path to delete?

Cheers Nick, That did it!

Just had to change it from service to systemctl ! had to delete the contents of the ldap dir too then do a reboot.

Edit:

Only snag is windows networking (samba) throws "Software has not been initialized."

Link. Note the health warning. I guess the v6 instructions work on v7.