String processing to compare dates

Hi all,

I have been scripting a shell script to allow me to easily create htaccess users and add them to existing htaccess groups etc.

However, the main part of the script that I am trying to accomplish is to store additional metadata in comments in the htpasswd file. I am trying to store an expiry date in the comments, such that the htpasswd file can be scanned and any users with expiry dates that are in the past can be deleted.

The syntax of the htpasswd file with expiry date metadata is:

username:ABCDEFG   (tab)   # comment [exp:YYYY-MM-DD]

I am now trying to write the script to scan the htpasswds file and delete expired users. This is what the script needs to do:

  1. Check each line in htpasswds
  2. If the user in htpasswds has no comment, or if the expiry date of the current user is a date that has not yet passed, then skip this user.
  3. Else, delete this user from the htpasswds now.
  4. Scan the htgroups file for all instances of this user and delete them (being weary of comments in the file)
  5. If, after deleting the user from a group, the group is left empty, delete the group (as well as a few lines of comments appearing above the group) from htgroups.

This is my current code to achieve the above steps. I stopped however because I am sure there is a more efficient way to achieve this (Perl, Awk, Sed, ..., I'm not really sure).

# loop through each line of htpasswd
(
    while read line; do            
        # extract the username from the line
        line_user=$(echo `echo "$line" | cut -d: -f1`)
        [ -n "$VERBOSE" ] && echo "Checking expiry of user '$line'" >&2
        
        # extract the expiry date from the line
        line_date=$(echo `echo "$line" | grep -o "\[exp:[0-9]\{4\}-[0-9]\{2\}-[0-9]\{2\}\]" | grep -o "[0-9]\{4\}-[0-9]\{2\}-[0-9]\{2\}"`)

        if [ -n "$line_date" ]; then
            EXPIRED=
            
            # break the expiry date into a year, month and date
            line_year=$(echo `echo "$line_date" | cut -d- -f1`)
            line_month=$(echo `echo "$line_date" | cut -d- -f2`)
            line_date=$(echo `echo "$line_date" | cut -d- -f3`)
            
            if [ $line_year -lt $today_year ]; then
                EXPIRED=1
            elif [ $line_year -eq $today_year ]; then
                if [ $line_month -lt $today_month ]; then
                    EXPIRED=1
                elif [ $line_month -eq $today_month ]; then
                    if [ $line_date -lt $today_date ]; then
                        EXPIRED=1
                   fi
                fi
            fi
            
            if [ -z "$EXPIRED" ]; then
                # not expired, output the user back to htpasswds
                echo "$line"
            else
                [ -n "$VERBOSE" ] && echo "User has expired. Deleting..." >&2
                
                echo "Haven't yet implemented deleting user from htgroups. This needs to be done manually!" >&2
                # also delete the user from htgroups
                #(
                #    while read line2; do
                #        #new_group=$(echo `echo "$line2" | grep -o "^[^#].*:*\[ $line_user \]" | grep -o "^[^#].*:*\[$line_user \]" | grep -o "^[^#].*:*\[ $line_user\]"sed 's/^[^#].*:$/'`)
                #    done < $HTGROUPS
                
                #) > $HTGROUPS.tmp    
            fi
        else
            # user doesn't have an expiry date, output the user back to htpasswds
            echo "$line"
        fi
    done < $HTPASSWD
) > $HTPASSWD.tmp

# move temp file
rm -f $HTPASSWD
mv $HTPASSWD.tmp $HTPASSWD

Try something like this:

EXPDATE='2011-10-11'
perl -lne '/\[exp:([\d-]+)\]/ ? ($1 gt "'"$EXPDATE"'" && print) : print' htpasswd

PS
It's only safe if you hardcode EXPDATE, do not get from an untrusted source.

Thank! That works great.

The other (harder) part, however, is to also delete these users from htgroups (I'm guessing this would need to be done before deleting the users from htpasswd)

I'm sure it may be easy and this can be done at the same time like this (pseudo-code):

perl -lne '/(user).*(date)/ ? (right_date($2) && ++$users{$1} && print) : print;
END {
  remove all users from keys %users from htgroups
}

But please, if you want to get real help give real examples of your input.

Thanks for your assistance. I am not familiar with Perl but I will give this a go.

Also, here is some examples of the input:

htpasswd:

osFtsoPO:ZjCqTssMg26GU
zNXOutfZ:K6wkNAmazZpgQ
iNTdEl4A:fbvbYtt2CG7GE
LM6G4M5r:JvaytqY5iQT6s
7b5ZVtfC:1YOTiDP7QMu46
ssUBkol0:Xpcxzcpkea00M
o1JvNOqA:hS7KXYCVPVZzk
Wjg28Q7Z:txvKZULw6IGQY
9BYLeZqm:o.brY4eMwo83Y
iUyLbtL4:2BPsVeT6INCjU
gW8Ts5QB:ES7wPmhv5zWkc   # [exp:2011-08-07]
xeI2BlBF:5UkRPOAmmvYQ6   # [exp:2011-08-07]
DqgUnQq7:wODD.q7G6tBG.   # [exp:2011-08-07]
APr62acP:bwSs.xqxzVhf2   # [exp:2011-08-07]
m8eJuHgJ:UGnggpDogE0mE   # [exp:2011-08-07]
qbIv1jvM:TEtfj3paSfXI6   # [exp:2011-08-07]
StcEfJjw:Yadibnucypobw   # [exp:2011-08-07]
ZZ7XQnKZ:k50CN9MTktDZM   # [exp:2011-08-07]
J4WJgqzb:648PPzACrHXU2   # [exp:2011-08-07]
i19tECpj:PBRPcIwa/Xrh.   # [exp:2011-08-07]
w9LVCtgF:sE0f4YvY2ZbFo   # [exp:2011-07-27]
np6EplLx:o/RoBYsDNWOa.   # [exp:2011-07-27]
lH9BiLaD:qFIPpYcg/uwxc   # [exp:2011-07-27]
Tq7bxObg:NNWjVwLvgEM2.   # [exp:2011-07-27]
Sprx0OHu:XizJkBT4U9CiE   # [exp:2011-07-27]
2d8NI6Sl:05CBhrUFVC7oI   # [exp:2011-07-27]
2eUHSw2J:gFLMdtYDQ41Q2   # [exp:2011-07-27]
GrVq3bXQ:BOcpjHrkFp6Lg   # [exp:2011-07-27]
lhDpIeid:wb0yrVMlQ9k7.   # [exp:2011-07-27]
XqxThbtH:elmYRVI8nI45E   # [exp:2011-07-27]
EVu0Y1E6:5eyhQUfeV6ayQ   # Comment [exp:2011-07-31]
IMHJb2s5:O13UEZuE/33Zo   # Comment [exp:2011-07-31]
Px1ck3Me:M1A.BkDbP.55.   # Comment [exp:2011-07-31]
hgAXHHk5:hU0pejGjTMG9M   # Comment [exp:2011-07-31]
abcd:VIEzA5I5IxEMc   # Comment [exp:2011-07-31]

htgroups:

################################################
# Group: test
# Members: (unknown)
test: f5JJTB6g WJXn6PHy d9NME1OB 6kKq8P8G g2EMf77J fWegqo2e XmexEc5S vgxH2txy hFFBvK35 6aLtuE8h z2jSdCMC rVmIJGoI JbZVO119 vUp33XWQ Wo7rP4qW A4Fb6E4w S3UpACU4 m6zXlxTq cuq6Pgbp 0FI0jXp4

################################################
# Group: test2
# Members: (unknown)
test2: atcF6fv6 rWTTjVAn NDjgFwZs eepg7TvA HaCx3x2a Sda2JzbW q8I6KL9V 7l6MSr7b tjg3AjoW 1NuSi63A abc

################################################
# Group: test3
# Members: (unknown)
test3: abcd

################################################
# Group: group1
# Members: (unknown)
group1: i2KWJtCH 1pHpcx7B YGDu8AcS WaTE3DMx KBBG0dPq fpiaRmLl oAQKdEDq oqbkUmUI nB0LQE4Y c4HhXeFz

If user "abcd" was removed from htpasswd, then the text:

################################################
# Group: test3
# Members: (unknown)
test3: abcd

should be removed from htgroups.

Sorry, I forget to add about "real examples" - remove all confidential information if there is.
Ok, for trying to write the script this will be enough:

qwerty:2BPsVeT6INCjU
abcd:ES7wPmhv5zWkc   # [exp:2011-08-07]
zxcv:5eyhQUfeV6ayQ   # Comment [exp:2011-09-31]

One more question. What to do if we have a such record in htgroups (and only abcd is expired):

################################################
# Group: group1
# Members: (unknown)
group1: abcd qwerty zxcv aaaa

Yeah all records are randomly generated. This file isn't in use.

If only abcd is removed: the output should be:

################################################
# Group: group1
# Members: (unknown)
group1: qwerty zxcv aaaa

This script should work but I haven't tested it thoroughly.

#!/usr/bin/env perl

use warnings;
use strict;

use constant PASSWD => 'htpasswd';
use constant GROUPS => 'htgroups';
use constant NEW    => '.new';

sub usage {
    die "Usage: $0 EXPDATE, where EXPDATE is in YYYY-MM-DD format"
}

usage if @ARGV != 1 or $ARGV[0] !~ /\d{4}-\d{2}-\d{2}/;

my $expdate = shift;
my %users;

# don't output expired users from PASSWD to PASSWD.NEW
# and remember them in %users

open my $fpasswd, '<', PASSWD or die $!;
open my $fpasswd_new, '>', PASSWD . NEW or die $!;

while (<$fpasswd>) {
    if (/^(\w+):.*\[exp:([\d-]+)\]/) {
        if ($2 lt $expdate) {
            $users{$1}++;
        } else {
            print $fpasswd_new $_;
        }
    } else {
        print $fpasswd_new $_;
    }
}

close $fpasswd;
close $fpasswd_new;

# remove users from %users from records of GROUPS file
# and don't output the records to GROUPS.NEW where no more users

# records in GROUPS should be separated by blank lines

open my $fgroups, '<', GROUPS or die $!;
open my $fgroups_new, '>', GROUPS . NEW or die $!;

local $/;
$/="";

# not very robust but this may be enough
while (<$fgroups>) {
    for my $user (keys %users) {
        s/$user//;
    }
    s/ +/ /g;   # here may appear extra spaces so squash them
    s/ +$//gm;

    print $fgroups_new $_ if !/^[^#]\w+:\s*$/m;
}

close $fgroups;
close $fgroups_new;

Works amazingly. Thank you so much