Nick Howitt wrote:

Blame that one on me.The cyrus-imap jail was not working correctly (at all) so I pushed a couple of updates. It looks like I may have missed the check to see if the file existed in the first place or something similar has gone wrong. I guess I'll have to have another look at it. It will be harder to fix now!

Thanks for the heads up.

Thank you Nick for your incredible work !

Blame that one on me.The cyrus-imap jail was not working correctly (at all) so I pushed a couple of updates. It looks like I may have missed the check to see if the file existed in the first place or something similar has gone wrong. I guess I'll have to have another look at it. It will be harder to fix now!

Thanks for the heads up.

I post this here because it is the first (among very few) that I found while googling my issue on ClearOs7.

After rebooting a test server that was sleepy for a few weeks and after all the updates were completed I check the Attack Detector app to find out that fail2ban wasn’t running.
Each attempt to start it from the webconfig or terminal failed right away.

I used

/usr/bin/fail2ban-client -v -v start

to have more information and saw:

INFO     Loading files: ['/etc/fail2ban/jail.d/clearos-cyrus-imap.conf']

ERROR  Failed during configuration: File contains no section headers.

file: /etc/fail2ban/jail.d/clearos-cyrus-imap.conf, line: 1

'port = imap,imap3,imaps,pop3,pop3s\n'

To check I deleted "/etc/fail2ban/jail.d/clearos-cyrus-imap.conf" and fail2ban restarted right successfully.

I don’t use a mail server or have any need for incoming mail on this unit.
Should I tried to understand what was wrong with "clearos-cyrus-imap.conf" or it shouldn’t be here anyway ?

Bernard

It could be. local fires off after 90 and custom. You certainly don't need it anymore now ClearOS provide the 90 file and you are using ipset jails.

Nick,

I've continued to check this out and haven't been able to figure anything more until today.

I noticed in the /etc/clearos/firewall.d/local there is a line

service fail2ban restart

and nothing else.

Also there is a /etc/clearos/firewall.d/90-attack-detector file with a bunch of code.

Is it possible that the 90-attack-detector has started fail2ban and local tries to start fail2ban causing it to hang?

Thanks again for your help and patience as I try to figure this out.

I've just tested and yes, there is a firewall restart when adding a block. That's a bit brutal.

The equivalent of the 5.x rc.firewall.local is /etc/clearos/firewall.d/local. Each time you save changes to it the firewall restarts as well. Again use $IPTABLES or "iptables -w". Stick them in the ipv4 section.

You appear to have a mixture of ipset and iptables blocks. ClearOS seems to work better with ipset jails so I'd be tempted to change the banaction in jail.conf or jail.local to "iptables-ipset-proto6". I've done that for mine.

Otherwise I don't know how to tidy things up. I'd be tempted to try a "firewall-start -d" if it is the firewall which is causing the problem. This will start the firewall in debug mode and give you some more output. as it starts. You can correlate the ouput to /var/log/system and/var/log/fail2ban.log

Nick Howitt wrote:
It looks like a firewall restart has happened and this has wiped a lot of f2b firewall bits. Why it restarted, I don't know. When that happens, when f2b tries to shut down, lots of its rules which it wants to delete no longer exist generating those errors. Ignore them.

It restarted when I added an IP to the Firewall:Incoming list in the Firewall App. This happens each and every time I add an IP to the list. It isn't in the custom firewall menu.

Nick Howitt wrote:Correct me if I'm wrong, but this is ClearOS 7. Do you have app-attack-detector installed any more?

Yes, this is COS 7.x. I reinstalled app-attack-detector from the MarketPlace after I realized that the problems that I was having was with using systemd log. I left my changes in the jail.local but everything else is standard.

Nick Howitt wrote:What firewall files do you have related to fail2ban or attack-detector in /etc/clearos/firewall.d/?

I haven't added anything to the firewall.d manually here is the list:

ll /etc/clearos/firewall.d/*

-rwxr-xr-x 1 root root   95 Feb  4 11:36 /etc/clearos/firewall.d/10-ntp

-rw-r--r-- 1 root root 1156 Aug 20 11:10 /etc/clearos/firewall.d/10-snortsam

-rwxr-xr-x 1 root root 1580 Feb 14 13:37 /etc/clearos/firewall.d/90-attack-detector

-rwxr-xr-x 1 root root  326 Jan 24 03:09 /etc/clearos/firewall.d/custom

-rwxr-xr-x 1 root root  212 Jan 11 17:24 /etc/clearos/firewall.d/local

-rwxr-xr-x 1 root root 1467 Dec 12 14:52 /etc/clearos/firewall.d/types

Nick Howitt wrote:Also what is the contents of /etc/clearos/firewall.d/custom and /etc/clearos/firewall.d/local, if any.

cat /etc/clearos/firewall.d/custom 

#######################################

# Created by API - Please Do NOT Edit #

#######################################



# IPv4 Custom Firewall Rules

#===========================



if [ "$FW_PROTO" == "ipv4" ]; then true

fi



# IPv6 Custom Firewall Rules

#===========================



if [ "$FW_PROTO" == "ipv6" ]; then true

fi

cat /etc/clearos/firewall.d/local 

# This script is run after every firewall restart.  Add custom rules here.

# Ensure you use $IPTABLES instead of calling iptables directly if you wish

# to avoid xtable locking problems.

service fail2ban restart

Nick Howitt wrote:AFAIK, the firewall only restarts when deleting a rule, not when adding it.

In my case it restarts after adding a rule. I haven't tried deleting any rules.
It should be able to delete a rule without restarting too.

Nick Howitt wrote:Although it restarts when you change and save /etc/clearos/firewall.d/local, so it may do the same with /etc/clearos/firewall.d/custom, which I think is what you are effectively editing.

I haven't a clue where the block rules that are added in the Firewall:Incoming are added. But each time I add one, the firewall restarts and kills fail2ban.

On COS5.2, I had a list of IPs that I blocked because they were a pest. It was some sort of local file rc.firewall.local that had a list of commands with IPTABLES -A INPUT -s ipaddress -j -DROP. I used to manually add the IPs there so they would reblock upon a system restart. The IPTABLES command would add the block manually without restarting the firewall. I don't know where to add these now with this new firewall.d.

It looks like a firewall restart has happened and this has wiped a lot of f2b firewall bits. Why it restarted, I don't know. When that happens, when f2b tries to shut down, lots of its rules which it wants to delete no longer exist generating those errors. Ignore them.

Correct me if I'm wrong, but this is ClearOS 7. Do you have app-attack-detector installed any more? What firewall files do you have related to fail2ban or attack-detector in /etc/clearos/firewall.d/?

Also what is the contents of /etc/clearos/firewall.d/custom and /etc/clearos/firewall.d/local, if any.

AFAIK, the firewall only restarts when deleting a rule, not when adding it.

[edit]
Although it restarts when you change and save /etc/clearos/firewall.d/local, so it may do the same with /etc/clearos/firewall.d/custom, which I think is what you are effectively editing.
[/edit]

So here is what happened when I added an ip address to the Network:Firewall:Incoming Firewall:Blocked Incoming Connections (Add).

2018-02-15 15:20:30,159 fail2ban.transmitter [12468]: DEBUG Command: ['stop']

2018-02-15 15:20:30,159 fail2ban.asyncserver [12468]: DEBUG Removed socket file /var/run/fail2ban/fail2ban.sock

2018-02-15 15:20:30,159 fail2ban.asyncserver [12468]: DEBUG Socket shutdown

2018-02-15 15:20:30,159 fail2ban.server [12468]: INFO Stopping all jails

2018-02-15 15:20:30,159 fail2ban.server [12468]: DEBUG Stopping jail cyrus-imap

2018-02-15 15:20:30,229 fail2ban.filterpoll [12468]: DEBUG cyrus-imap filter terminated

2018-02-15 15:20:30,689 fail2ban.actions [12468]: DEBUG Flush ban list

2018-02-15 15:20:30,689 fail2ban.action [12468]: DEBUG iptables -w -D INPUT -p tcp -m multiport --dports imap3,imaps -j f2b-cyrus-imap

iptables -w -F f2b-cyrus-imap

iptables -w -X f2b-cyrus-imap

2018-02-15 15:20:30,793 fail2ban.action [12468]: ERROR iptables -w -D INPUT -p tcp -m multiport --dports imap3,imaps -j f2b-cyrus-imap

iptables -w -F f2b-cyrus-imap

iptables -w -X f2b-cyrus-imap -- stdout: ''

2018-02-15 15:20:30,794 fail2ban.action [12468]: ERROR iptables -w -D INPUT -p tcp -m multiport --dports imap3,imaps -j f2b-cyrus-imap

iptables -w -F f2b-cyrus-imap

iptables -w -X f2b-cyrus-imap -- stderr: "iptables v1.4.21: Couldn't load target `f2b-cyrus-imap':No such file or directory\n\nTry `iptables -h' or 'iptables --help' for more information.\niptables: No chain/target/match by that name.\niptables: No chain/target/match by that name.\n"

2018-02-15 15:20:30,794 fail2ban.action [12468]: ERROR iptables -w -D INPUT -p tcp -m multiport --dports imap3,imaps -j f2b-cyrus-imap

iptables -w -F f2b-cyrus-imap

iptables -w -X f2b-cyrus-imap -- returned 1

2018-02-15 15:20:30,794 fail2ban.actions [12468]: ERROR Failed to stop jail 'cyrus-imap' action 'iptables-multiport': Error stopping action

Traceback (most recent call last):

File "/usr/lib/python2.7/site-packages/fail2ban/server/actions.py", line 241, in run

action.stop()

File "/usr/lib/python2.7/site-packages/fail2ban/server/action.py", line 372, in stop

raise RuntimeError("Error stopping action")

RuntimeError: Error stopping action

2018-02-15 15:20:30,810 fail2ban.actions [12468]: DEBUG cyrus-imap: action terminated

2018-02-15 15:20:30,810 fail2ban.jail [12468]: INFO Jail 'cyrus-imap' stopped

2018-02-15 15:20:30,811 fail2ban.server [12468]: DEBUG Stopping jail postfix-sasl

2018-02-15 15:20:31,246 fail2ban.filterpoll [12468]: DEBUG postfix-sasl filter terminated

2018-02-15 15:20:31,594 fail2ban.actions [12468]: DEBUG Flush ban list

2018-02-15 15:20:31,594 fail2ban.action [12468]: DEBUG iptables -w -D INPUT -p tcp -m multiport --dports smtp,465,submission,imap3,imaps,pop3,pop3s -j f2b-postfix-sasl

iptables -w -F f2b-postfix-sasl

iptables -w -X f2b-postfix-sasl

2018-02-15 15:20:31,699 fail2ban.action [12468]: ERROR iptables -w -D INPUT -p tcp -m multiport --dports smtp,465,submission,imap3,imaps,pop3,pop3s -j f2b-postfix-sasl

iptables -w -F f2b-postfix-sasl

iptables -w -X f2b-postfix-sasl -- stdout: ''

2018-02-15 15:20:31,699 fail2ban.action [12468]: ERROR iptables -w -D INPUT -p tcp -m multiport --dports smtp,465,submission,imap3,imaps,pop3,pop3s -j f2b-postfix-sasl

iptables -w -F f2b-postfix-sasl

iptables -w -X f2b-postfix-sasl -- stderr: "iptables v1.4.21: Couldn't load target `f2b-postfix-sasl':No such file or directory\n\nTry `iptables -h' or 'iptables --help' for more information.\niptables: No chain/target/match by that name.\niptables: No chain/target/match by that name.\n"

2018-02-15 15:20:31,699 fail2ban.action [12468]: ERROR iptables -w -D INPUT -p tcp -m multiport --dports smtp,465,submission,imap3,imaps,pop3,pop3s -j f2b-postfix-sasl

iptables -w -F f2b-postfix-sasl

iptables -w -X f2b-postfix-sasl -- returned 1

2018-02-15 15:20:31,700 fail2ban.actions [12468]: ERROR Failed to stop jail 'postfix-sasl' action 'iptables-multiport': Error stopping action

Traceback (most recent call last):

File "/usr/lib/python2.7/site-packages/fail2ban/server/actions.py", line 241, in run

action.stop()

File "/usr/lib/python2.7/site-packages/fail2ban/server/action.py", line 372, in stop

raise RuntimeError("Error stopping action")

RuntimeError: Error stopping action

2018-02-15 15:20:31,700 fail2ban.actions [12468]: DEBUG postfix-sasl: action terminated

2018-02-15 15:20:31,700 fail2ban.jail [12468]: INFO Jail 'postfix-sasl' stopped

2018-02-15 15:20:31,701 fail2ban.server [12468]: DEBUG Stopping jail sshd

2018-02-15 15:20:32,150 fail2ban.filtersystemd [12468]: DEBUG sshd filter terminated

2018-02-15 15:20:32,380 fail2ban.actions [12468]: DEBUG Flush ban list

2018-02-15 15:20:32,381 fail2ban.action [12468]: DEBUG iptables -w -D INPUT -m set --match-set f2b-sshd src -j DROP

ipset flush f2b-sshd

ipset destroy f2b-sshd

2018-02-15 15:20:32,485 fail2ban.action [12468]: DEBUG iptables -w -D INPUT -m set --match-set f2b-sshd src -j DROP

ipset flush f2b-sshd

ipset destroy f2b-sshd -- stdout: ''

2018-02-15 15:20:32,485 fail2ban.action [12468]: DEBUG iptables -w -D INPUT -m set --match-set f2b-sshd src -j DROP

ipset flush f2b-sshd

ipset destroy f2b-sshd -- stderr: 'iptables: Bad rule (does a matching rule exist in that chain?).\n'

2018-02-15 15:20:32,485 fail2ban.action [12468]: DEBUG iptables -w -D INPUT -m set --match-set f2b-sshd src -j DROP

ipset flush f2b-sshd

ipset destroy f2b-sshd -- returned successfully

2018-02-15 15:20:32,485 fail2ban.actions [12468]: DEBUG sshd: action terminated

2018-02-15 15:20:32,486 fail2ban.jail [12468]: INFO Jail 'sshd' stopped

2018-02-15 15:20:32,486 fail2ban.server [12468]: DEBUG Stopping jail sshd-ddos

2018-02-15 15:20:32,491 fail2ban.actions [12468]: DEBUG Flush ban list

2018-02-15 15:20:32,492 fail2ban.action [12468]: DEBUG iptables -w -D INPUT -m set --match-set f2b-sshd-ddos src -j DROP

ipset flush f2b-sshd-ddos

ipset destroy f2b-sshd-ddos

2018-02-15 15:20:32,596 fail2ban.action [12468]: DEBUG iptables -w -D INPUT -m set --match-set f2b-sshd-ddos src -j DROP

ipset flush f2b-sshd-ddos

ipset destroy f2b-sshd-ddos -- stdout: ''

2018-02-15 15:20:32,596 fail2ban.action [12468]: DEBUG iptables -w -D INPUT -m set --match-set f2b-sshd-ddos src -j DROP

ipset flush f2b-sshd-ddos

ipset destroy f2b-sshd-ddos -- stderr: 'iptables: Bad rule (does a matching rule exist in that chain?).\n'

2018-02-15 15:20:32,596 fail2ban.action [12468]: DEBUG iptables -w -D INPUT -m set --match-set f2b-sshd-ddos src -j DROP

ipset flush f2b-sshd-ddos

ipset destroy f2b-sshd-ddos -- returned successfully

2018-02-15 15:20:32,596 fail2ban.actions [12468]: DEBUG sshd-ddos: action terminated

2018-02-15 15:20:33,151 fail2ban.filtersystemd [12468]: DEBUG sshd-ddos filter terminated

2018-02-15 15:20:33,152 fail2ban.jail [12468]: INFO Jail 'sshd-ddos' stopped

2018-02-15 15:20:33,153 fail2ban.server [12468]: DEBUG Remove PID file /var/run/fail2ban/fail2ban.pid

2018-02-15 15:20:33,153 fail2ban.server [12468]: INFO Exiting Fail2ban

The firewall has restarted as far as I can tell. Attack Detector shows "Restarting" but isn't. [So here I am full circle back to the start]

fail2ban-client status

ERROR  Failed to access socket path: /var/run/fail2ban/fail2ban.sock. Is fail2ban running?

systemctl status -l fail2ban.service

● fail2ban.service - Fail2Ban Service

   Loaded: loaded (/usr/lib/systemd/system/fail2ban.service; enabled; vendor preset: disabled)

   Active: inactive (dead) since Thu 2018-02-15 15:20:33 EST; 18min ago

     Docs: man:fail2ban(1)

  Process: 14983 ExecStop=/usr/bin/fail2ban-client stop (code=exited, status=0/SUCCESS)

  Process: 12465 ExecStart=/usr/bin/fail2ban-client -x start (code=exited, status=0/SUCCESS)

 Main PID: 12468 (code=killed, signal=TERM)



Feb 15 15:01:47 domain.com systemd[1]: Starting Fail2Ban Service...

Feb 15 15:01:47 domain.com fail2ban-client[12465]: 2018-02-15 15:01:47,746 fail2ban.server         [12466]: INFO    Starting Fail2ban v0.9.5

Feb 15 15:01:47 domain.com fail2ban-client[12465]: 2018-02-15 15:01:47,746 fail2ban.server         [12466]: INFO    Starting in daemon mode

Feb 15 15:01:48 domain.com systemd[1]: Started Fail2Ban Service.

Feb 15 15:20:30 domain.com systemd[1]: Stopping Fail2Ban Service...

Feb 15 15:20:33 domain.com fail2ban-client[14983]: Shutdown successful

Please note: at 15:01 I added DEBUG to the log level and did a condrestart. It is now 15:41 and attack detector hasn't started.

If I do

systemctl start fail2ban

it hangs. If I do

systemctl stop fail2ban

systemctl start fail2ban

it starts again.

Nick Howitt wrote:

There is no way adding a firewall rule should stop f2b.

I am wondering how much work it would be to reload ClearOS? Brutal but it may be the best. Otherwise we can start again with the troubleshooting now netify has gone.

Hi Nick.

I really don't want to reinstall COS. My installation, now 5 months old, is only COS + a few Marketplace apps. Getting websites up, mail running etc. etc. will take days that I don't have time for at the moment. I haven't spent any time doing any customizations like I did with COS5.2. It's a vanilla install. Interestingly enough this all started this year around the time of the firmware updates due to the Intel security issue. Before that, everything ran OK.

I left the logging for f2b as default but will change that right after I post this message.

While I agree that adding a firewall rule shouldn't stop f2b, it appears that the behaviour is something like this: add a firewall rule, restart the firewall, restart netify, restart attack detector etc.

I agree it should be "just add this ip address to the block list". Is there a way to trace the firewall add rule behaviour in detail?

Since netify was hanging (I have a strong suspicion that it hung at the reference to the non-existant dat file ...) and systemd never got a message that netify hadn't started, I think the restart of the attack detector timed out. ???

As a debugging thought, can you change loglevel to DEBUG in /etc/fail2ban/fail2ban.conf?

There is no way adding a firewall rule should stop f2b.

I am wondering how much work it would be to reload ClearOS? Brutal but it may be the best. Otherwise we can start again with the troubleshooting now netify has gone.

I've finally got all the protocol filter and netify stuff removed.

I'm still seeing a problem with the Attack Detector/fail2ban.

If I add an IP to the block list on the incoming firewall rules, then the Attack Detector shuts down and hangs on the restart. I think the shut down and restart makes sense but I'm still at a loss why the Attack Detector won't restart.

It feels like I've spent days trying to figure this out and end up back in the same place. Aaaaaaahhhhhh.

To remove the protocol filter, you can do a "yum remove app-protocol*, but it won't remove the netify bits. If you do a "yum remove netify" it will remove the protocol filter and possibly the application filter. It wil also remove app-netify-core and app-netify-fwa core. I don't know what these are. You can then do a "yum reinstall app-application*" to reinstall the application filter and it will bring back in the dependencies it needs.

You're way ahead of me as I don't use gateway management or the application filter so I have not seen the interactions, if any.

If you enable jails through jails.local, I agree, the get enabled even if the clearos- setting is disabled and vice versa! Go figure.

My hack is:

#!/bin/sh



# IPv4 only for now

#------------------



if [ "$FW_PROTO" != "ipv4" ]; then

    return 0

fi



# Bail if fail2ban is not running

#--------------------------------



if [ ! -e /var/run/fail2ban/fail2ban.sock ]; then

    return 0

fi



# Config

#-------



ACTION_ADD="/var/clearos/attack_detector/run/f2b-add"

ACTION_DELETE="/var/clearos/attack_detector/run/f2b-delete"



rm -f $ACTION_ADD

rm -f $ACTION_DELETE



# Regenerate iptables commands

#-----------------------------



JAILS=`LANG=en_US fail2ban-client status | grep "Jail list" | sed -e 's/.*Jail list://' -e 's/,//g'`



for JAIL in $JAILS; do

    ACTION=`fail2ban-client get $JAIL actions | tail -n 1`

    if [[ $ACTION =~ ipset ]]; then

        fail2ban-client get $JAIL action $ACTION actionstart | grep "<iptables>" >> $ACTION_ADD

        #PROPERTIES=`fail2ban-client get $JAIL actionproperties $ACTION | sed -e 's/.*://' -e 's/,//g'`

        PROPERTIES="iptables chain name blocktype lockingopt protocol port returntype"



        for PROPERTY in $PROPERTIES; do

            CHECK=`echo $PROPERTY | grep -v ^known | grep -v ^action`

            if [ -n "$CHECK" ]; then

                VALUE=`fail2ban-client get $JAIL action $ACTION $PROPERTY`

                VALUE_SED=`echo $VALUE |  sed 's/\//\\\\\//g'`

                sed -i -e "s/<$PROPERTY>/$VALUE_SED/" $ACTION_ADD

            fi

        done

    else

        fail2ban-client reload $JAIL > /dev/null 2>&1

    fi

done

# the "&" in the above fail2ban-client command is optional and alows the script to finish a bit quicker



# Stops script throwing an error if no ipset type jails are enabled as ACTION_ADD will not exist

if [ -f $ACTION_ADD ]; then

    grep " \-I " $ACTION_ADD | sed "s/ \-I / -D /" > $ACTION_DELETE



    sh $ACTION_DELETE > /dev/null 2>&1

    sh $ACTION_ADD >/dev/null 2>&1

fi



# Exit 0

:

Compare it to the original /etc/clearos/firewall.d/90-attack-detector. I've only added a few lines and it is irrelevant if every jail uses ipset sets.

And one more update.
I wanted to remove the protocol filter but could figure out what it was called in the marketplace. Ended up on the Application filter which is also stopped. Tried to start the Application filter (which I was using) and it killed the Attack Detector too. This Gateway Management Community is running though.

I'm stumped.

An update.

I reinstalled Attack Detector. It runs OK when netify-fwa is stopped. I think it is using my jails.local rather than the clearos-*.conf but I'm not sure since I tried to restart the protocol filter and it brought everything crashing down again. Now I have to figure out the command sequence to stop/halt netify-fwa starting and get the fail2ban going again.

I think the protocol filter has to go. Maybe that will fix things. I haven't been using it anyway so maybe less complex is better. It's still frustrating since all this stuff ran great on COS5.2.

When it has been removed, I'll check if the iptables are properly blocking stuff.

Nick Howitt wrote:
app-attack-detector should play well with your existing f2b installation. Just check where your default banaction is set and what to. Mine is set in jail.conf and is set to "iptables-ipset-proto6". IMHO this is better than iptables as a ban action. Also it plays well with the configlet ClearOS drop into /etc/clearos/firewall.d to restart the jails. This configlet only works with ipset type jails.

I copied the iptables-ipset-proto6-allports when defined in the Clearos actions and put them into the jail.local. This is where I enabled the jails that I am using. I also changed the ban action to DROP in the iptables-common.local following on from your recommendation.

I've been searching for what all these actions.d are and when & why you use them. So far I haven't found anything helpful in the fail2ban docs. They are still at v0.8 and we have 0.9.5.

Nick Howitt wrote:
This answers your other point. There is no need to update the ClearOS set up for the firewall rules. If the jails are ipset type they already take care of that. You can hack their script to make it reload non-ipset jails as well. It needs an if...then...do_the_ClearOS_stuff...else reload_the_jail. I have done that then converted everything to ipset anyway. The ClearOS view is that if the user installs app-attack-detector then f2b runs for its bits. If the user wants other jails then it is up to the user to get them going.

I'm not quite sure what your hack does but that is part of re-learning how fail2ban works.

Thanks again for all your support.

app-attack-detector should play well with your existing f2b installation. Just check where your default banaction is set and what to. Mine is set in jail.conf and is set to "iptables-ipset-proto6". IMHO this is better than iptables as a ban action. Also it plays well with the configlet ClearOS drop into /etc/clearos/firewall.d to restart the jails. This configlet only works with ipset type jails.

This answers your other point. There is no need to update the ClearOS set up for the firwall rules. If the jails are ipset type they already take care of that. You can hack their script to make it reload non-ipset jails as well. It needs an if...then...do_the_ClearOS_stuff...else reload_the_jail. I have done that then converted everything to ipset anyway. The ClearOS view is that if the user installs app-attack-detector then f2b runs for its bits. If the user wants other jails then it is up to the user to get them going.

Nick Howitt wrote:

I suspect the final solution will be to reinstall app-attack-detector.

Thanks Nick. That makes sense.
If I reinstall app-attack-detector from the Marketplace will it play nice with the existing fail2ban installation.
I assume that it won't over write the .local files for jail and iptables-common.local ?

Given what you needed to do to get fail2ban ie. create the 30-fail2ban file, shouldn't this solution be updated in COS 7.4? The Attack Detector is a Marketplace app in the end.

I suspect the final solution will be to reinstall app-attack-detector. netify-fwa is playing round with the firewall. If it restarts the firewall your f2b chains get wiped. What I used to do was have a file, /etc/clearos/firewall.d/30-fail2ban, and in it I put:

[ -e /var/run/fail2ban/fail2ban.pid ] && systemctl reload fail2ban.service &

This forces a fail2ban reload every time f2b reloads. app-attack-detector does something similar but cleverer for ipset jails.

What your output is showing is that a lot of the iptables firewall set up is not there as f2b shuts down. To be fair it always generates errors as you can't easily test the existence of a chain, but you can test the existence of firewall rules.

An update.

This appears to be somehow related to a problem I'm having with netify-fwa. Link to other discussion.
After watching fail2ban for a few days I know that it is running properly now. I think I could probably install the Attack Detector again.

What I have noticed from the logs is that fail2ban is identifying issues but it isn't blocking or dropping anything. I adjusted the settings in the jail so that some of the repeat offenders would get blocked but nothing happens. Interestingly the repeat offenders come back just over 30 min. The default jail time is 30 min. So I've increased the time and I get the ban statement but nothing is showing up in the iptables.

Can you check the following log entries and confirm if what I think is happening is actually happening? What I think is that fail2ban thinks there should be some banned IPs so when it goes through a shut down it is trying identify the ban/rejected IPs and finds none. So it throws an error. This shows only the two jails that are running.

2018-01-31 09:22:54,731 fail2ban.server [12324]: INFO Stopping all jails

2018-01-31 09:22:55,491 fail2ban.action [12324]: ERROR iptables -w -D INPUT -p tcp -m multiport --dports imap3,imaps -j f2b-cyrus-imap

iptables -w -F f2b-cyrus-imap

iptables -w -X f2b-cyrus-imap -- stdout: ''

2018-01-31 09:22:55,491 fail2ban.action [12324]: ERROR iptables -w -D INPUT -p tcp -m multiport --dports imap3,imaps -j f2b-cyrus-imap

iptables -w -F f2b-cyrus-imap

iptables -w -X f2b-cyrus-imap -- stderr: "iptables v1.4.21: Couldn't load target `f2b-cyrus-imap':No such file or directory\n\nTry `iptables -h' or 'iptables --help' for more information.\niptables: No chain/target/match by that name.\niptables: No chain/target/match by that name.\n"

2018-01-31 09:22:55,491 fail2ban.action [12324]: ERROR iptables -w -D INPUT -p tcp -m multiport --dports imap3,imaps -j f2b-cyrus-imap

iptables -w -F f2b-cyrus-imap

iptables -w -X f2b-cyrus-imap -- returned 1

2018-01-31 09:22:55,492 fail2ban.actions [12324]: ERROR Failed to stop jail 'cyrus-imap' action 'iptables-multiport': Error stopping action

2018-01-31 09:22:55,492 fail2ban.jail [12324]: INFO Jail 'cyrus-imap' stopped

2018-01-31 09:22:56,506 fail2ban.action [12324]: ERROR iptables -w -D INPUT -p tcp -m multiport --dports smtp,465,submission,imap3,imaps,pop3,pop3s -j f2b-postfix-sasl

iptables -w -F f2b-postfix-sasl

iptables -w -X f2b-postfix-sasl -- stdout: ''

2018-01-31 09:22:56,506 fail2ban.action [12324]: ERROR iptables -w -D INPUT -p tcp -m multiport --dports smtp,465,submission,imap3,imaps,pop3,pop3s -j f2b-postfix-sasl

iptables -w -F f2b-postfix-sasl

iptables -w -X f2b-postfix-sasl -- stderr: "iptables v1.4.21: Couldn't load target `f2b-postfix-sasl':No such file or directory\n\nTry `iptables -h' or 'iptables --help' for more information.\niptables: No chain/target/match by that name.\niptables: No chain/target/match by that name.\n"

2018-01-31 09:22:56,507 fail2ban.action [12324]: ERROR iptables -w -D INPUT -p tcp -m multiport --dports smtp,465,submission,imap3,imaps,pop3,pop3s -j f2b-postfix-sasl

iptables -w -F f2b-postfix-sasl

iptables -w -X f2b-postfix-sasl -- returned 1

2018-01-31 09:22:56,507 fail2ban.actions [12324]: ERROR Failed to stop jail 'postfix-sasl' action 'iptables-multiport': Error stopping action

2018-01-31 09:22:56,507 fail2ban.jail [12324]: INFO Jail 'postfix-sasl' stopped

2018-01-31 09:22:56,509 fail2ban.server [12324]: INFO Exiting Fail2ban

Tracing the rules back, you need to find where blocktype is defined. It looks like mine is in /etc/fail2ban/action.d/iptables-blocktype.local but that may have been my addition. /etc/fail2ban/action.d/iptables-common.conf indicates iptables-blocktype.local is obsolete but in it I have a section which says:

[INCLUDES]



after = iptables-blocktype.local

        iptables-common.local

# iptables-blocktype.local is obsolete

In iptables-blocktype.local I have:

# Fail2Ban configuration file

#

# Author: Daniel Black

#

# This is a included configuration file and includes the defination for the blocktype

# used in all iptables based actions by default.

#

# The user can override the default in iptables-blocktype.local

[Init]

blocktype = DROP

This suggest to me it may be my override.

I can't reliably comment on systemd and journalmatch as it takes us back to where we were at the start. All you can do is try something and see if it works. Mine does with the default configs.

Nick Howitt wrote:
Personally I'd change the default rule from REJECT to DROP and remove the reject-with icmp-port-unreachable bit. If I am blocking them, I don't see the point in telling them. I am almost tempted to blackhole them!

Thanks again, Nick.
I didn't set the iptable command. This was created by default fail2ban or the banaction, I think.

Your comment makes total sense. I'm not sure how to go about changing the default rule to DROP and remove the icmp-port-unreachable.

I've been reading about this jailmatch in jail.conf, journalctl man page and systemd.journal-fields man page. It would appear there is a problem with using fail2ban and systemd journals if the filter doesn't have a journalmatch definition. I can't figure out this journal match definition so I've change the following in my jail.local

[postfix-sasl]

enabled = true

port     = smtp,465,submission,imap3,imaps,pop3,pop3s

logpath  = /var/log/maillog

backend  = polling

journalmatch =

bantime  = 43200

findtime = 1800

maxretry = 3



[cyrus-imap]

enabled = true

port   = imap3,imaps

logpath = /var/log/maillog

backend = polling

journalmatch =

bantime  = 43200

findtime = 1800

maxretry = 3

I've directed both to the maillog and the errors are gone.

In the iptables I have this

# iptables -L -n

Chain INPUT (policy DROP)

target     prot opt source               destination         

f2b-cyrus-imap  tcp  --  0.0.0.0/0            0.0.0.0/0            multiport dports 220,993

f2b-postfix-sasl  tcp  --  0.0.0.0/0            0.0.0.0/0            multiport dports 25,465,587,220,993,110,995



...



Chain f2b-cyrus-imap (1 references)

target     prot opt source               destination         

RETURN     all  --  0.0.0.0/0            0.0.0.0/0           



Chain f2b-postfix-sasl (1 references)

target     prot opt source               destination         

RETURN     all  --  0.0.0.0/0            0.0.0.0/0         

I'll watch this over the next few days and see what happens.

No it is not blocking everything. The key bit is the "match-set f2b-cyrus-imap src". This tells me that ipset rules are being used. It will block everything from the source address (src) listed in the ipset set f2b-cyrus-imap. I have a little whizz-bang to check these sets. Put the following in a file, make it executable and run it:

for SET in `ipset list -name | grep f2b` ; do

	ipset list $SET -output save | grep add | awk '{print $2 " "  $3}'

done

Personally I'd change the default rule from REJECT to DROP and remove the reject-with icmp-port-unreachable bit. If I am blocking them, I don't see the point in telling them. I am almost tempted to blackhole them!

Nick Howitt wrote:

There is a fail2ban-systemd in epel-unverified. I have not researched it but I'm not sure why you want it.

I thought it was necessary to get fail2ban to be able to be enabled by systemctl. After reading a bit more it looks like fail2ban > 0.9 is systemctl enabled.

Anyway, I was able to get it going. I'm still getting the following notice for cyrus-imap and proftpd

fail2ban.filtersystemd [16999]: NOTICE Jail started without 'journalmatch' set. Jail regexs will be checked against all journal entries, which is not advised for performance reasons.

For postfix-sasl it is OK.

fail2ban.jail           [8727]: INFO    Creating new jail 'postfix-sasl'

fail2ban.jail           [8727]: INFO    Jail 'postfix-sasl' uses systemd

fail2ban.jail           [8727]: INFO    Initiated 'systemd' backend

fail2ban.filter         [8727]: INFO    Set maxRetry = 5

fail2ban.actions        [8727]: INFO    Set banTime = 43200

fail2ban.filter         [8727]: INFO    Set findtime = 1800

fail2ban.filtersystemd  [8727]: INFO    Added journal match for: '_SYSTEMD_UNIT=postfix.service'

I see in iptables

REJECT     tcp  --  anywhere             anywhere             multiport dports imap3,imaps match-set f2b-cyrus-imap src reject-with icmp-port-unreachable

REJECT     tcp  --  anywhere             anywhere             multiport dports smtp,urd,submission,imap3,imaps,pop3,pop3s match-set f2b-postfix-sasl src reject-with icmp-port-unreachable

REJECT     tcp  --  anywhere             anywhere             multiport dports ftp,ftp-data,ftps,ftps-data match-set f2b-proftpd src reject-with icmp-port-unreachable

Isn't this blocking everything?

There is a fail2ban-systemd in epel-unverified. I have not researched it but I'm not sure why you want it.

Nick Howitt wrote:

Posts crossed. Delete everything in /etc/fail2ban except /etc/fail2ban/jail.d then do a:

yum install fail2ban-server

Done.
Is there nothing like

fail2ban-systemd

available?

Posts crossed. Delete everything in /etc/fail2ban except /etc/fail2ban/jail.d then do a:

yum install fail2ban-server

Be careful if you go it alone. Files in /etc/fail2ban/jail.d are added by their apps and not by app-attack-detector. The other thing which you'll need to get round is that a firewall restart will wipe the f2b firewall rules so you may want to drop a configlet into /etc/clearos/firewall.d to restart f2b if it is running. That will add the rules again. You can make it run asynchronously by adding a "&" at the end of the restart command.

nuke wrote:
I'm going to uninstall again and see if I can get fail2ban to run by itself without the Attack Detector envelope.

Which doesn't work.

yum install fail2ban fail2ban-systemd

Loaded plugins: clearcenter-marketplace, fastestmirror

ClearCenter Marketplace: fetching repositories...

Loading mirror speeds from cached hostfile

 * clearos: mirror2-newyork.clearos.com

 * clearos-centos: download4.clearsdn.com

 * clearos-centos-sclo-rh: download4.clearsdn.com

 * clearos-centos-updates: download4.clearsdn.com

 * clearos-contribs: mirror2-newyork.clearos.com

 * clearos-epel: download4.clearsdn.com

 * clearos-fast-updates: download4.clearsdn.com

 * clearos-infra: mirror2-newyork.clearos.com

 * clearos-updates: mirror2-newyork.clearos.com

 * private-clearcenter-dnsthingy: download4.clearsdn.com:80

 * private-clearcenter-dyndns: download4.clearsdn.com:80

 * private-clearcenter-plex: download4.clearsdn.com:80

No package fail2ban available.

No package fail2ban-systemd available.

Error: Nothing to do

Grrrrr.

It installed as part of the Attack Detector, so why can't yum find it now? Do I have to manually download the RPMs or add another repository?

Nick Howitt wrote:
The jail takes the definition from jail.conf initially then overrides with the settings in jail.d/clearos-postfix-sasl.conf. It looks like you still have the default settings. Does it keep running like this or is it still failing?

I haven't had any time to look at this for a few days. This morning I got another email warning

/etc/cron.daily/logrotate:



ERROR  Failed to access socket path: /var/run/fail2ban/fail2ban.sock. Is fail2ban running?

and when I checked, Fail2ban has stopped again and it looks like it's stuck trying to restart.

This is frustrating.

I'm going to uninstall again and see if I can get fail2ban to run by itself without the Attack Detector envelope.

The jail takes the definition from jail.conf initially then overrides with the settings in jail.d/clearos-postfix-sasl.conf. It looks like you still have the default settings. Does it keep running like this or is it still failing?

If you wanted to test, you could try overriding the findtime for the jail e.g by adding a file /etc/fail2ban/jail.local and putting in it:

[postfix-sasl]

findtime = 86400

maxretry = 2

It is a bit aggressive, but if you have set up your mobile devices, they should never trip it so, only bad attempts will be logged.

As a lateral jump, do you have postfix authentication on? I would turn it off and open incoming port tcp:465. ClearOS will still allow authentication on SMTPS/port 465. Then you switch all mobile devices to SMTPS. Fixed LAN devices can also be switched or, if you're a little lazy like me, you could add your LAN to the trusted networks so no authentication is needed on SMTP/port 25.

Nick Howitt wrote:
What does your [postfix-sasl] jail look like?

Thanks again Nick.

I'm not sure I understand correctly. Is the [postfix-sasl] jail what is written in the jail.conf file & clear postfix conf file?

The info from the jail.conf is default. I wasn't going to change anything until I got it working right.

findtime = 600
bantime = 600
maxretry = 5

[postfix-sasl]



port     = smtp,465,submission,imap3,imaps,pop3,pop3s

# You might consider monitoring /var/log/mail.warn instead if you are

# running postfix since it would provide the same log lines at the

# "warn" level but overall at the smaller filesize.

logpath  = %(postfix_log)s

backend  = %(postfix_backend)s

and the jail.d/clearos-postfix-sasl.conf

# This file is controlled by the ClearOS API, please do not edit!

# If you would like to customize parameters, add a new configlet file.

[postfix-sasl]

enabled = true

bantime  = 86400

Eventually, I want to get some of the jails for Apache going.

"Found" means one of your jails has found a single match. You then need to look in your jails to see how many messages need to be found (maxretry) in what period (findtime). Note that the default findtime is 600 seconds and the default maxretry is 5 so you need 5 "Found" messages in 5 minutes in a jail before there is a ban. All your "Found" entries are well spaced. The Attack Detector app logs only show the bans. If you go to the log viewer Webconfig > Reports > Performance and Resources > Log Viewer you can view the full fail2ban log. You can then filter it for "Ban" (capital B or you get everything) to see your bans. Or do as you did and view the file in /var/log/fail2ban.

What does your [postfix-sasl] jail look like?

The webconfig log shows nothing even when there are items in the fail2ban.log file.

Nick Howitt wrote:

I don't particularly want to try it for the moment, but I don't believe the webconfig will uninstall the fail2ban-server component which is the one I was trying to refresh. I could be wrong, but have a look in your yum log

Nick, the webconfig does uninstall all three files and reinstalls all three again. From my yum log:

--------------------- yum Begin ------------------------ 

 Packages Installed:

    1:app-attack-detector-core-2.3.1-1.v7.noarch

    1:app-attack-detector-2.3.1-1.v7.noarch

    fail2ban-server-0.9.5-3.el7.noarch

 

 Packages Updated:



 ....



 Packages Erased:

    1:app-attack-detector-core-2.3.1-1.v7.noarch

    1:app-attack-detector-2.3.1-1.v7.noarch

    fail2ban-server-0.9.5-3.el7.noarch

 ---------------------- yum End ------------------------- 

It appears to be running, ie. service status is green, but I'm still seeing the journal match warning.

I don't think it is actually blocking anything. If I remember the last time I dug into fail2ban, the log would show that it found an issue, that it had been blocked and after some period of time, that it was released again. The log only show that it found an issue. The logging is the same level of detail as the COS5.2 install.

Here is the log from the start up to now. At the bottom you can see postfix-sasl and cyrus-imap have "Found" issues but no action occurs. Am I reading this correctly?

2018-01-22 15:01:11,099 fail2ban.server [17582]: INFO Changed logging target to /var/log/fail2ban.log for Fail2ban v0.9.5

2018-01-22 15:01:11,099 fail2ban.database [17582]: INFO Connected to fail2ban persistent database '/var/lib/fail2ban/fail2ban.sqlite3'

2018-01-22 15:01:11,101 fail2ban.jail [17582]: INFO Creating new jail 'sshd'

2018-01-22 15:01:11,120 fail2ban.jail [17582]: INFO Jail 'sshd' uses systemd

2018-01-22 15:01:11,142 fail2ban.jail [17582]: INFO Initiated 'systemd' backend

2018-01-22 15:01:11,143 fail2ban.filter [17582]: INFO Set maxRetry = 5

2018-01-22 15:01:11,144 fail2ban.actions [17582]: INFO Set banTime = 86400

2018-01-22 15:01:11,145 fail2ban.filter [17582]: INFO Set findtime = 600

2018-01-22 15:01:11,145 fail2ban.filter [17582]: INFO Set maxlines = 10

2018-01-22 15:01:11,223 fail2ban.filtersystemd [17582]: INFO Added journal match for: '_SYSTEMD_UNIT=sshd.service + _COMM=sshd'

2018-01-22 15:01:11,230 fail2ban.jail [17582]: INFO Creating new jail 'sshd-ddos'

2018-01-22 15:01:11,230 fail2ban.jail [17582]: INFO Jail 'sshd-ddos' uses systemd

2018-01-22 15:01:11,231 fail2ban.jail [17582]: INFO Initiated 'systemd' backend

2018-01-22 15:01:11,232 fail2ban.filter [17582]: INFO Set maxRetry = 5

2018-01-22 15:01:11,233 fail2ban.actions [17582]: INFO Set banTime = 86400

2018-01-22 15:01:11,233 fail2ban.filter [17582]: INFO Set findtime = 600

2018-01-22 15:01:11,235 fail2ban.filtersystemd [17582]: INFO Added journal match for: '_SYSTEMD_UNIT=sshd.service + _COMM=sshd'

2018-01-22 15:01:11,242 fail2ban.jail [17582]: INFO Creating new jail 'proftpd'

2018-01-22 15:01:11,242 fail2ban.jail [17582]: INFO Jail 'proftpd' uses systemd

2018-01-22 15:01:11,243 fail2ban.jail [17582]: INFO Initiated 'systemd' backend

2018-01-22 15:01:11,243 fail2ban.filter [17582]: INFO Set maxRetry = 5

2018-01-22 15:01:11,244 fail2ban.actions [17582]: INFO Set banTime = 86400

2018-01-22 15:01:11,244 fail2ban.filter [17582]: INFO Set findtime = 600

2018-01-22 15:01:11,262 fail2ban.jail [17582]: INFO Creating new jail 'postfix-sasl'

2018-01-22 15:01:11,262 fail2ban.jail [17582]: INFO Jail 'postfix-sasl' uses systemd

2018-01-22 15:01:11,263 fail2ban.jail [17582]: INFO Initiated 'systemd' backend

2018-01-22 15:01:11,263 fail2ban.filter [17582]: INFO Set maxRetry = 5

2018-01-22 15:01:11,264 fail2ban.actions [17582]: INFO Set banTime = 86400

2018-01-22 15:01:11,264 fail2ban.filter [17582]: INFO Set findtime = 600

2018-01-22 15:01:11,268 fail2ban.filtersystemd [17582]: INFO Added journal match for: '_SYSTEMD_UNIT=postfix.service'

2018-01-22 15:01:11,274 fail2ban.jail [17582]: INFO Creating new jail 'cyrus-imap'

2018-01-22 15:01:11,274 fail2ban.jail [17582]: INFO Jail 'cyrus-imap' uses systemd

2018-01-22 15:01:11,275 fail2ban.jail [17582]: INFO Initiated 'systemd' backend

2018-01-22 15:01:11,275 fail2ban.filter [17582]: INFO Set maxRetry = 5

2018-01-22 15:01:11,276 fail2ban.actions [17582]: INFO Set banTime = 86400

2018-01-22 15:01:11,276 fail2ban.filter [17582]: INFO Set findtime = 600

2018-01-22 15:01:11,287 fail2ban.jail [17582]: INFO Jail 'sshd' started

2018-01-22 15:01:11,290 fail2ban.jail [17582]: INFO Jail 'sshd-ddos' started

2018-01-22 15:01:11,291 fail2ban.filtersystemd [17582]: NOTICE Jail started without 'journalmatch' set. Jail regexs will be checked against all journal entries, which is not advised for performance reasons.

2018-01-22 15:01:11,292 fail2ban.jail [17582]: INFO Jail 'proftpd' started

2018-01-22 15:01:11,295 fail2ban.jail [17582]: INFO Jail 'postfix-sasl' started

2018-01-22 15:01:11,301 fail2ban.filtersystemd [17582]: NOTICE Jail started without 'journalmatch' set. Jail regexs will be checked against all journal entries, which is not advised for performance reasons.

2018-01-22 15:01:11,335 fail2ban.jail [17582]: INFO Jail 'cyrus-imap' started

2018-01-22 15:35:57,214 fail2ban.filter [17582]: INFO [postfix-sasl] Found 185.140.12.121

2018-01-22 16:21:24,571 fail2ban.filter [17582]: INFO [postfix-sasl] Found 185.222.209.14

2018-01-22 17:10:05,104 fail2ban.filter [17582]: INFO [cyrus-imap] Found 60.166.12.117

2018-01-22 19:29:14,303 fail2ban.filter [17582]: INFO [postfix-sasl] Found 185.222.209.14

2018-01-22 20:19:37,799 fail2ban.filter [17582]: INFO [postfix-sasl] Found 185.140.12.121

2018-01-22 20:22:41,692 fail2ban.filter [17582]: INFO [cyrus-imap] Found 116.248.41.55

2018-01-22 21:12:16,228 fail2ban.filter [17582]: INFO [cyrus-imap] Found 61.166.110.62

2018-01-22 22:19:46,027 fail2ban.filter [17582]: INFO [postfix-sasl] Found 185.222.209.14

2018-01-23 01:06:43,758 fail2ban.filter [17582]: INFO [postfix-sasl] Found 185.140.12.121

2018-01-23 01:12:02,954 fail2ban.filter [17582]: INFO [postfix-sasl] Found 185.222.209.14

2018-01-23 03:03:01,138 fail2ban.filter [17582]: INFO [cyrus-imap] Found 60.223.252.6

2018-01-23 05:51:19,053 fail2ban.filter [17582]: INFO [postfix-sasl] Found 185.140.12.121

I don't particularly want to try it for the moment, but I don't believe the webconfig will uninstall the fail2ban-server component which is the one I was trying to refresh. I could be wrong, but have a look in your yum log

I guess I'm an impatient guy. I just left it race the FF interface and came back a while later and found it was uninstalled.
So have installed it again (forgot to delete the /etc/fail2ban folder ) ) and it looks like it works with all the jails.
In the Services list it shows the green light.
So I'm left scratching my head as to what caused all this time suck.
Thanks again for your help, Nick. I appreciate all your patience with helping me through this.

Removing from the Webconfig will only remove the app- packages and not fail2ban, I believe. Instead do a:

yum remove fail2ban-server

Abort it if it tries to remove more than fail2ban-server and app-attack*.

Be very careful if you ever use yum to remove packages. It can wreck your system if it removes too many dependencies (don't ever try "yum remove gcc*").

Nick Howitt wrote:

You could possibly uninstall fail2ban-server and then delete everything in /etc/fail2ban except for anything in /etc/fail2ban/jail.d then reinstall it (make a backup of /etc/fail2ban first).

I just tried to uninstall and it hangs the GUI using FF and Chrome. No idea if it has removed app-attack-detector; app-attack-detector-core; and fail2ban-server. Maybe a manual uninstall is required?

Can't check this manually as the webGUI uninstall is locking yum.

Another app is currently holding the yum lock; waiting for it to exit...

[edit]added new info[/edit]

You could possibly uninstall fail2ban-server and then delete everything in /etc/fail2ban except for anything in /etc/fail2ban/jail.d then reinstall it (make a backup of /etc/fail2ban first).

I was working on this but not getting very far. This morning I have this in my email from anacron:

etc/cron.daily/logrotate:



ERROR  Failed to access socket path: /var/run/fail2ban/fail2ban.sock. Is fail2ban running?

I didn't realize that fail2ban had crapped out again. This is frustrating.

Would uninstall/reinstall do any good?

Your knowledge is getting beyond mine now. I have not looked at journalmatch or backend at all.

Yesterday I was talking to Peter and he also checked:

fail2ban-client get postfix-sasl logpath

and got the same results but it appears not to matter as he is getting blocks in his jail (I don't as I only have secure ports open and they rarely get attacked). It may be due to the backend sucking out messages before they hit the logs but I don't know for sure. It is conjecture, but it does mean the default jail is not buggy.

If you really want to check, using "sed -e 's/#.*$//' -e '/^$/d' /etc/fail2ban/jail.conf" and cutting down the output to the defaults, postfix-sasl and sshd-ddos I get:

[INCLUDES]

before = paths-fedora.conf

[DEFAULT]

ignoreip = 127.0.0.1/8

ignorecommand =

bantime  = 600

findtime  = 600

maxretry = 5

backend = auto

usedns = warn

logencoding = auto

enabled = false

filter = %(__name__)s

destemail = root@localhost

sender = root@localhost

mta = sendmail

protocol = tcp

chain = INPUT

port = 0:65535

fail2ban_agent = Fail2Ban/%(fail2ban_version)s

banaction = iptables-ipset-proto6

banaction_allports = iptables-ipset-proto6-allports

action_ = %(banaction)s[name=%(__name__)s, bantime="%(bantime)s", port="%(port)s", protocol="%(protocol)s", chain="%(chain)s"]

action_mw = %(banaction)s[name=%(__name__)s, bantime="%(bantime)s", port="%(port)s", protocol="%(protocol)s", chain="%(chain)s"]

            %(mta)s-whois[name=%(__name__)s, sender="%(sender)s", dest="%(destemail)s", protocol="%(protocol)s", chain="%(chain)s"]

action_mwl = %(banaction)s[name=%(__name__)s, bantime="%(bantime)s", port="%(port)s", protocol="%(protocol)s", chain="%(chain)s"]

             %(mta)s-whois-lines[name=%(__name__)s, sender="%(sender)s", dest="%(destemail)s", logpath=%(logpath)s, chain="%(chain)s"]

action_xarf = %(banaction)s[name=%(__name__)s, bantime="%(bantime)s", port="%(port)s", protocol="%(protocol)s", chain="%(chain)s"]

             xarf-login-attack[service=%(__name__)s, sender="%(sender)s", logpath=%(logpath)s, port="%(port)s"]

action_cf_mwl = cloudflare[cfuser="%(cfemail)s", cftoken="%(cfapikey)s"]

                %(mta)s-whois-lines[name=%(__name__)s, sender="%(sender)s", dest="%(destemail)s", logpath=%(logpath)s, chain="%(chain)s"]

action_blocklist_de  = blocklist_de[email="%(sender)s", service=%(filter)s, apikey="%(blocklist_de_apikey)s", agent="%(fail2ban_agent)s"]

action_badips = badips.py[category="%(__name__)s", banaction="%(banaction)s", agent="%(fail2ban_agent)s"]

action_badips_report = badips[category="%(__name__)s", agent="%(fail2ban_agent)s"]

action = %(action_)s

[sshd-ddos]

port    = ssh

logpath = %(sshd_log)s

backend = %(sshd_backend)s

[postfix-sasl]

port     = smtp,465,submission,imap3,imaps,pop3,pop3s

logpath  = %(postfix_log)s

I enable mine by jails.local rather than via the Webconfig and my jails.local (cut down) is:

[DEFAULT]

ignoreip = 127.0.0.1/8 172.17.2.0/23 192.168.10.0/24 192.168.30.0/24 10.8.0.0/24 67.18.3.134 173.255.233.57 159.203.59.228 194.62.204.0/22 194.62.208.0/22 194.62.212.0/23 217.243.151.200/29

[postfix-sasl]

enabled  = true

maxretry = 1

bantime  = 432000

findtime = 14400

and I don't use sshd-ddos. My filter is a bit different from default but that does not matter. Mu paths-common.conf and paths-fedora.conf are default

Nick Howitt wrote:
When f2b went from 0.8.x to 0.9.x a lot of things were parametrised. In jail.conf you have "before = paths-fedora.conf", so have a look in paths-fedora.conf where you'll find "before = paths-common.conf". In paths-common.conf you'll find "postfix_log = %(syslog_mail_warn)s" and "syslog_mail_warn =" which takes us back to the paths-fedora.conf for its override "syslog_mail_warn = /var/log/maillog" and bingo! It does make tracking down entries convoluted but I can see why they've done it to make it easier to work with multiple distro's.

Yes, it is convoluted. I can follow it thanks to your explanation. I have following in /etc/fail2ban/paths-fedora.conf

[DEFAULT]

syslog_mail = /var/log/maillog

syslog_mail_warn = /var/log/maillog

Nick Howitt wrote:
Changing "logpath = %(syslog_mail)s" which works with other postfix jails does not work here! It looks like you have to comment out the "backend" line in jail.conf or (better) set "backend = auto" in jail.local. I smell a bug here.

I have

backend = auto

in jail.conf. This doesn't look right. If we're using systemd for the logs then it should be

backend = systemd

. Setting "auto" means that it will use pyinotify, gamin or polling.
From the jail.conf:

# "backend" specifies the backend used to get files modification.

# Available options are "pyinotify", "gamin", "polling", "systemd" and "auto".

# This option can be overridden in each jail as well.

#

# pyinotify: requires pyinotify (a file alteration monitor) to be installed.

#              If pyinotify is not installed, Fail2ban will use auto.

# gamin:     requires Gamin (a file alteration monitor) to be installed.

#              If Gamin is not installed, Fail2ban will use auto.

# polling:   uses a polling algorithm which does not require external libraries.

# systemd:   uses systemd python library to access the systemd journal.

#              Specifying "logpath" is not valid for this backend.

#              See "journalmatch" in the jails associated filter config

# auto:      will try to use the following backends, in order:

#              pyinotify, gamin, polling.

#

# Note: if systemd backend is chosen as the default but you enable a jail

#       for which logs are present only in its own log files, specify some other

#       backend for that jail (e.g. polling) and provide empty value for

#       journalmatch. See https://github.com/fail2ban/fail2ban/issues/959#issuecomment-74901200

backend = auto

Do I need to search for and add "journalmatch = " into the jail.local for the two services that crashed to make this work?
But then systemd shouldn't be used since the backend is auto. Oh my head hurts.
[edit]Doh! Idiot at the keyboard. Fix final comment[/edit]

The big list gives nothing I was looking for a file I don't have so did not know its exact location, but you don't have it either which is good.

When f2b went from 0.8.x to 0.9.x a lot of things were parametrised. In jail.conf you have "before = paths-fedora.conf", so have a look in paths-fedora.conf where you'll find "before = paths-common.conf". In paths-common.conf you'll find "postfix_log = %(syslog_mail_warn)s" and "syslog_mail_warn =" which takes us back to the paths-fedora.conf for its override "syslog_mail_warn = /var/log/maillog" and bingo! It does make tracking down entries convoluted but I can see why they've done it to make it easier to work with multiple distro's.

The fail2ban-server command should point to a jail so:

fail2ban-client get postfix-sasl logpath

What I don't understand is why I get:

fail2ban-client get postfix-sasl logpath

No file is currently monitored

It works with other jails. Changing "logpath = %(syslog_mail)s" which works with other postfix jails does not work here! It looks like you have to comment out the "backend" line in jail.conf or (better) set "backend = auto" in jail.local. I smell a bug here.

I've been checking the COS5.2 fail2ban jail.local and jail.conf and log locations where in the jail.* files. I was using backend = gamin.

In my old jail.local I had:

# This jail forces the backend to "polling".



[sasl-iptables]



enabled  = true

filter   = sasl

backend  = polling

action   = iptables-multiport[name=sasl, port="25,465,993,995", protocol=tcp]

           sendmail-whois[name=sasl, dest=root]

logpath  = /var/log/maillog

bantime  = 43200

findtime = 1800

maxretry = 3



# added bantime and findtime to sasl because of problem with 1 IP 

# coming every 10 min for days on end



[postfix-tcpwrapper]



enabled  = true

filter   = postfix

action   = iptables[name=postfix, port=25, protocol=tcp]

#           sendmail-whois[name=postfix, dest=root]

logpath  = /var/log/maillog

bantime  = 1209600

findtime = 1800

maxretry = 3

Since COS7 is using systemd with logs, is where the problem is???
In the new jail.conf it shows

logpath  = %(postfix_log)s

backend  = %(postfix_backend)s

I can't figure out what it is referencing.
When I query what log postfix is using for postfix it says

fail2ban-client get postfix logpath

ERROR  NOK: ('postfix',)

Sorry but the jail 'postfix' does not exist

I guess this is correct since it isn't starting so not reading any logs. This is going beyond my novice knowledge.
Could this be an issue if both postfix-sasl and cyrus jails are reading the same log at the same time?

Actually this below is more disheartening.

cat /etc/fail2ban/jail.d/clearos-cyrus-imap.conf 

# This file is controlled by the ClearOS API, please do not edit!

# If you would like to customize parameters, add a new configlet file.

[cyrus-imap]

enabled = true

bantime  = 86400

from jail.conf

[cyrus-imap]



port   = imap3,imaps

logpath = %(syslog_mail)s

backend = %(syslog_backend)s

fail2ban-client get cyrus-imap logpath

No file is currently monitored

Huh? So the jail is "enabled" but not checking any log? Then how is it determining when to block anything?
{Maybe this is why nothing shows up in the ipblock list?}

[edit] added some more info to post & fixed typo[/edit]

Nick Howitt wrote:
Can you give the output to:

ls -l /etc/fail2ban/* -r

Happy to. Big list.

Thanks in advance Nick!

ls -l /etc/fail2ban/* -r

-rw-r--r--  1 root root   290 Jul 14  2016 /etc/fail2ban/paths-osx.conf

-rw-r--r--  1 root root   659 Jul 14  2016 /etc/fail2ban/paths-opensuse.conf

-rw-r--r--  1 root root  1174 Jul 14  2016 /etc/fail2ban/paths-freebsd.conf

-rw-r--r--  1 root root  1070 Jul 14  2016 /etc/fail2ban/paths-fedora.conf

-rw-r--r--  1 root root   642 Jul 14  2016 /etc/fail2ban/paths-debian.conf

-rw-r--r--  1 root root  2375 Jul 14  2016 /etc/fail2ban/paths-common.conf

-rw-r--r--  1 root root   683 Sep 18 17:55 /etc/fail2ban/jail.local.tmp

-rw-r--r--  1 root root 21024 Sep  8 12:24 /etc/fail2ban/jail.conf

-rw-r--r--  1 root root  2328 Jul 14  2016 /etc/fail2ban/fail2ban.conf



/etc/fail2ban/jail.d:

total 20

-rw-r--r-- 1 root root 238 Jan 14 20:58 clearos-sshd-ddos.conf

-rw-r--r-- 1 root root 227 Jan 14 21:19 clearos-sshd.conf

-rw-r--r-- 1 root root 179 Jan 14 21:18 clearos-proftpd.conf

-rw-r--r-- 1 root root 185 Jan 14 21:00 clearos-postfix-sasl.conf

-rw-r--r-- 1 root root 182 Dec  8  2016 clearos-cyrus-imap.conf



/etc/fail2ban/filter.d:

total 328

-rw-r--r-- 1 root root  503 Jul 14  2016 xinetd-fail.conf

-rw-r--r-- 1 root root  520 Jul 14  2016 wuftpd.conf

-rw-r--r-- 1 root root  444 Jul 14  2016 webmin-auth.conf

-rw-r--r-- 1 root root  627 Jul 14  2016 vsftpd.conf

-rw-r--r-- 1 root root  374 Jul 14  2016 uwimap-auth.conf

-rw-r--r-- 1 root root  821 Jul 14  2016 tine20.conf

-rw-r--r-- 1 root root  645 Jul 14  2016 suhosin.conf

-rw-r--r-- 1 root root  363 Jul 14  2016 stunnel.conf

-rw-r--r-- 1 root root  761 Jul 14  2016 sshd-ddos.conf

-rw-r--r-- 1 root root 3125 Jul 14  2016 sshd.conf

-rw-r--r-- 1 root root  199 Jul 14  2016 squirrelmail.conf

-rw-r--r-- 1 root root  206 Jul 14  2016 squid.conf

-rw-r--r-- 1 root root 1094 Jul 14  2016 solid-pop3d.conf

-rw-r--r-- 1 root root  472 Jul 14  2016 sogo-auth.conf

-rw-r--r-- 1 root root  706 Jul 14  2016 slapd.conf

-rw-r--r-- 1 root root  371 Jul 14  2016 sieve.conf

-rw-r--r-- 1 root root 2471 Oct  3  2016 sendmail-reject.conf

-rw-r--r-- 1 root root  385 Oct  3  2016 sendmail-auth.conf

-rw-r--r-- 1 root root  570 Jul 14  2016 selinux-ssh.conf

-rw-r--r-- 1 root root  517 Jul 14  2016 selinux-common.conf

-rw-r--r-- 1 root root  821 Jul 14  2016 screensharingd.conf

-rw-r--r-- 1 root root 1367 Jul 14  2016 roundcube-auth.conf

-rw-r--r-- 1 root root 1286 Jul 14  2016 recidive.conf

-rw-r--r-- 1 root root  795 Jul 14  2016 qmail.conf

-rw-r--r-- 1 root root 2409 Jul 14  2016 pure-ftpd.conf

-rw-r--r-- 1 root root 1216 Jul 14  2016 proftpd.conf

-rw-r--r-- 1 root root  483 Jul 14  2016 postfix-sasl.conf

-rw-r--r-- 1 root root  454 Jul 14  2016 postfix-rbl.conf

-rw-r--r-- 1 root root 1289 Jul 14  2016 postfix.conf

-rw-r--r-- 1 root root  188 Jul 14  2016 portsentry.conf

-rw-r--r-- 1 root root  834 Jul 14  2016 php-url-fopen.conf

-rw-r--r-- 1 root root  568 Jul 14  2016 perdition.conf

-rw-r--r-- 1 root root  814 Jul 14  2016 pam-generic.conf

-rw-r--r-- 1 root root 1905 Jul 14  2016 oracleims.conf

-rw-r--r-- 1 root root  495 Jul 14  2016 openwebmail.conf

-rw-r--r-- 1 root root  459 Jul 14  2016 openhab.conf

-rw-r--r-- 1 root root  707 Jul 14  2016 nsd.conf

-rw-r--r-- 1 root root 1427 Jul 14  2016 nginx-limit-req.conf

-rw-r--r-- 1 root root  442 Jul 14  2016 nginx-http-auth.conf

-rw-r--r-- 1 root root  528 Jul 14  2016 nginx-botsearch.conf

-rw-r--r-- 1 root root 1594 Jul 14  2016 named-refused.conf

-rw-r--r-- 1 root root  400 Jul 14  2016 nagios.conf

-rw-r--r-- 1 root root  891 Jul 14  2016 mysqld-auth.conf

-rw-r--r-- 1 root root  652 Jul 14  2016 murmur.conf

-rw-r--r-- 1 root root  773 Jul 14  2016 monit.conf

-rw-r--r-- 1 root root  323 Jul 14  2016 lighttpd-auth.conf

-rw-r--r-- 1 root root  482 Jul 14  2016 kerio.conf

drwxr-xr-x 2 root root   34 Sep  8 12:24 ignorecommands

-rw-r--r-- 1 root root  404 Jul 14  2016 horde.conf

-rw-r--r-- 1 root root 1158 Jul 14  2016 haproxy-http-auth.conf

-rw-r--r-- 1 root root  512 Jul 14  2016 guacamole.conf

-rw-r--r-- 1 root root  322 Jul 14  2016 gssftpd.conf

-rw-r--r-- 1 root root  236 Jul 14  2016 groupoffice.conf

-rw-r--r-- 1 root root 1209 Jul 14  2016 froxlor-auth.conf

-rw-r--r-- 1 root root  963 Jul 14  2016 freeswitch.conf

-rw-r--r-- 1 root root 2158 Jul 14  2016 exim-spam.conf

-rw-r--r-- 1 root root 1810 Jul 14  2016 exim.conf

-rw-r--r-- 1 root root  423 Jul 14  2016 exim-common.conf

-rw-r--r-- 1 root root 1282 Jul 14  2016 ejabberd-auth.conf

-rw-r--r-- 1 root root  557 Jul 14  2016 drupal-auth.conf

-rw-r--r-- 1 root root 1696 Jul 14  2016 dropbear.conf

-rw-r--r-- 1 root root 1778 Jul 14  2016 dovecot.conf

-rw-r--r-- 1 root root  345 Jul 14  2016 directadmin.conf

-rw-r--r-- 1 root root  443 Jul 14  2016 cyrus-imap.conf

-rw-r--r-- 1 root root  490 Jul 14  2016 courier-smtp.conf

-rw-r--r-- 1 root root  393 Jul 14  2016 courier-auth.conf

-rw-r--r-- 1 root root  252 Jul 14  2016 counter-strike.conf

-rw-r--r-- 1 root root 1863 Jul 14  2016 common.conf

-rw-r--r-- 1 root root  520 Jul 14  2016 botsearch-common.conf

-rw-r--r-- 1 root root 2436 Jul 14  2016 asterisk.conf

-rw-r--r-- 1 root root 1156 Jul 14  2016 assp.conf

-rw-r--r-- 1 root root 1014 Jul 14  2016 apache-shellshock.conf

-rw-r--r-- 1 root root  346 Jul 14  2016 apache-pass.conf

-rw-r--r-- 1 root root 2000 Jul 14  2016 apache-overflows.conf

-rw-r--r-- 1 root root 1187 Jul 14  2016 apache-noscript.conf

-rw-r--r-- 1 root root  596 Jul 14  2016 apache-nohome.conf

-rw-r--r-- 1 root root  402 Jul 14  2016 apache-modsecurity.conf

-rw-r--r-- 1 root root  268 Jul 14  2016 apache-fakegooglebot.conf

-rw-r--r-- 1 root root  813 Jul 14  2016 apache-common.conf

-rw-r--r-- 1 root root 1273 Jul 14  2016 apache-botsearch.conf

-rw-r--r-- 1 root root 2745 Jul 14  2016 apache-badbots.conf

-rw-r--r-- 1 root root 3241 Jul 14  2016 apache-auth.conf

-rw-r--r-- 1 root root  442 Jul 14  2016 3proxy.conf



/etc/fail2ban/fail2ban.d:

total 0



/etc/fail2ban/action.d:

total 204

-rw-r--r-- 1 root root  6018 Jul 14  2016 xarf-login-attack.conf

-rw-r--r-- 1 root root  1330 Jul 14  2016 symbiosis-blacklist-allports.conf

-rw-r--r-- 2 root root  5921 Oct  3  2016 smtp.pyo

-rw-r--r-- 2 root root  5921 Oct  3  2016 smtp.pyc

-rw-r--r-- 1 root root  6021 Jul 14  2016 smtp.py

-rw-r--r-- 1 root root  2981 Jul 14  2016 shorewall-ipset-proto6.conf

-rw-r--r-- 1 root root   798 Jul 14  2016 sendmail.conf

-rw-r--r-- 1 root root  1023 Jul 14  2016 route.conf

-rw-r--r-- 1 root root  3146 Jul 14  2016 nsupdate.conf

-rw-r--r-- 1 root root   496 Jul 14  2016 nftables-multiport.conf

-rw-r--r-- 1 root root  3680 Jul 14  2016 nftables-common.conf

-rw-r--r-- 1 root root   489 Jul 14  2016 nftables-allports.conf

-rw-r--r-- 1 root root  5233 Jul 14  2016 mynetwatchman.conf

-rw-r--r-- 1 root root  1556 Jul 14  2016 mail.conf

-rw-r--r-- 1 root root  2282 Jul 14  2016 iptables-xt_recent-echo.conf

-rw-r--r-- 1 root root  1508 Jul 14  2016 iptables-new.conf

-rw-r--r-- 1 root root  1910 Jul 14  2016 iptables-multiport-log.conf

-rw-r--r-- 1 root root  1431 Jul 14  2016 iptables-multiport.conf

-rw-r--r-- 1 root root  1798 Jul 14  2016 iptables-ipset-proto6.conf

-rw-r--r-- 1 root root  1755 Jul 14  2016 iptables-ipset-proto6-allports.conf

-rw-r--r-- 1 root root  1828 Jul 14  2016 iptables-ipset-proto4.conf

-rw-r--r-- 1 root root  1350 Jul 14  2016 iptables.conf

-rw-r--r-- 1 root root  1868 Jul 14  2016 iptables-common.conf

-rw-r--r-- 1 root root  1437 Jul 14  2016 iptables-allports.conf

-rw-r--r-- 1 root root  2689 Jul 14  2016 firewallcmd-rich-rules.conf

-rw-r--r-- 1 root root  3223 Jul 14  2016 firewallcmd-rich-logging.conf

-rw-r--r-- 1 root root  2005 Jul 14  2016 firewallcmd-new.conf

-rw-r--r-- 1 root root  2088 Jul 14  2016 firewallcmd-multiport.conf

-rw-r--r-- 1 root root  1530 Jul 14  2016 firewallcmd-ipset.conf

-rw-r--r-- 1 root root  1538 Jul 14  2016 firewallcmd-allports.conf

-rw-r--r-- 1 root root  1133 Jul 14  2016 dummy.conf

-rw-r--r-- 1 root root  7524 Jul 14  2016 dshield.conf

-rw-r--r-- 1 root root  1931 Jul 14  2016 cloudflare.conf

-rw-r--r-- 1 root root  2631 Jul 14  2016 blocklist_de.conf

-rw-r--r-- 2 root root 11791 Oct  3  2016 badips.pyo

-rw-r--r-- 2 root root 11791 Oct  3  2016 badips.pyc

-rw-r--r-- 1 root root 10620 Jul 14  2016 badips.py

-rw-r--r-- 1 root root   631 Jul 14  2016 badips.conf

-rw-r--r-- 1 root root   587 Jul 14  2016 apf.conf

If ssh is not open then there is no point in running f2b against the ssh ports.

Can you give the output to:

ls -l /etc/fail2ban/* -r

Nick Howitt wrote:

HI Nuke,
Can you disable all jails in the webconfig then start them one by one? By your logs it looks like sshd-ddos and postfix-sasl are failing. I use postfix-sasl but no sshd jails as I don't open ssh to the internet. My postfix-sasl is slightly altered from base so I wonder if that is why mine works.

Hi Nick.
Thanks again for your help.
I disabled all jails except for cyrus-imapd and fail2ban started OK.
I don't have port 22 open to the outside either. I presume I don't have to worry about this sshd-ddos running if the port is closed?
I'd like to get postfix-sasl running as I'm running a home mail server.
There looks to be some sort of clear files in addition to the jail.conf to control fail2ban. If I make changes in a jail.local am going to break something with COS7?
It's been years since I adjusted/tuned fail2ban on COS5.2.

HI Nuke,
Can you disable all jails in the webconfig then start them one by one? By your logs it looks like sshd-ddos and postfix-sasl are failing. I use postfix-sasl but no sshd jails as I don't open ssh to the internet. My postfix-sasl is slightly altered from base so I wonder if that is why mine works.