Write only changes to file - avoid duplicates

I want to create a file, to save a list of fail2ban blocked ip addresses. So I thought I'd create a loop that will check with fail2ban every minute, and write the ip addresses to a file.

while true; do echo $(fail2ban-client status asterisk-iptables | grep 'IP list' | sed 's/.*://g' | sed -e 's/^[ \t]*//') | tee -a ~/ips; sleep 60; done

I used tee -a to append, not re-write the file every minute. Of course the problem with this approach is that ~/ips is filled with duplicates every minute.

Is there a way to tell tee not to write an ip address in the file, if it already exists? Of course there's a good chance my whole approach is completely wrong, so I'm open to suggestions.

Thanks

No. The reason is that tee doesn't work on "lines" or "words" or anything like that: it works on data streams (redirections) and what is in these streams is of no concern to tee .

I don't know with how much data we are dealing here, so i just hope my suggestions will not be outright unfeasible. Instead of writing the list directly to the file you could write a program which checks one line after the other if it is already there and write it only if it is not. This program would get the output of your command as input. You could write to a named pipe instead of a file. To be honest i found at least two questionable points in your program:

while true; do echo $(fail2ban-client status asterisk-iptables | grep 'IP list' | sed 's/.*://g' | sed -e 's/^[ \t]*//') | tee -a ~/ips; sleep 60; done

First: the calls to sed and grep could be combined:

while true; do echo $(fail2ban-client status asterisk-iptables | sed -n '/IP list/ {;s/.*://g'; s/^[ \t]*//;p;}') | tee -a ~/ips; sleep 60; done

Second, what exactly do you need the tee at all for? It makes two datastreams out of one, so that you can process it two times insteead of one (actually this is where the name comes from - the "t-piece" which plumbers use to make two hoses out of one). But you only have one data stream (which goes into the file), so you do not need another, no?

while true; do echo $(fail2ban-client status asterisk-iptables | sed -n '/IP list/ {;s/.*://g'; s/^[ \t]*//;p;}') >> ~/ips; sleep 60; done

On a further thought i do not understand what the "echo" and the subshell is for either:

while :; do fail2ban-client status asterisk-iptables | sed -n '/IP list/ {;s/.*://g'; s/^[ \t]*//;p;}' >> ~/ips; sleep 60; done

So, make ~/.ips a named pipe, and let a separate script read it and put the result into a final file, something like this (its input will come from ~/.ips ):

#! /bin/ksh

typeset finalfile="/some/where/final"

exec 3>>"$finalfile"

while read line ; do
     if grep -qv "$line" "$finalfile" ; then
          print -u3 - "$line"
     fi
done

exec 3>&-

exit 0

Again, if this is feasible depends on how big the file is and how many entries per minute arrive. If the numbers are small enough this will work.

I hope this helps.

bakunin

Wouldn't a periodical sort -u of the final result file solve those problems in one go at minimal cost? I'd guess searching the entire file for each new IP could become resource hungry.

That depends. In principle: yes, that would suffice - IF the period between runs of sort -u is sufficiently small to fit thread-O/Ps needs. What exactly "what he needs" is and what therefore "sufficiently small" constitutes is everybodies guess. IF (and again, who knows if this is a moot point or not) there must not be any duplicates in the final file THEN my approach (or something along similar lines) will be necessary. Also, if appending the output file happens without strict regularity the run of the sort -u might collide with the new entries overwriting them - concurrent access can have that effect. It would need some sort of inter-process communication in this case.

But all that is conjecture. Thread-O/P will have to state his problem more clearly to select the right approach.

I hope this helps.

bakunin

Actually instead of exporting the banned IP addresses every 60 seconds and writing them to a file, I ended up reading fail2ban log files to find which IP's were banned.

cat /var/log/fail2ban.log* | grep -o 'Ban.*' | sed 's/\<Ban\> //g' | sort -u

The actual issue that I am trying to solve, is that the current version of FreePBX can only run fail2ban v0.8.14, which does not maintain it's list of banned IP addresses after a reboot. So I wanted to somehow save the list of banned IP's, and after reboot, manually ban them again. Something like this:

for ip in $(cat /var/log/fail2ban.log* | grep -o 'Ban.*' | sed 's/\<Ban\> //g' | sort -u); do fail2ban-client set asterisk-iptables banip $ip; done

Which again is not really working 100%, so now I'm thinking of using iptables commands to ban the IP's, not fail2ban commands.

But I appreciate your suggestions and the discussion. It helps with the learning.