use loop to search

Shell Programming and Scripting
1

Hi,
I need to search through a file to find the last one occurence of my data.

File1 is like this:
sososo
abc123
ssssss
abc456
ssssss
abc345

I need to loop through to identify the last occurence of "abc".
So "abc345" is the record that I need.

Can someone give me some ideas?

2

It matters how your file is formated and where else abc might come up, but for the example you showed, this will work.

grep "abc" file1|tail -1

3

Sorry. I forgot to mention this will be a loop.
It means that I will not stop after I found 'abc'. I will need to save it to a file and continue to search another pattern like the last occurence of "ss" and append it to the same file.

Basically I need to compare the first 5 charactors of each record and only pick the last occurence of the matched(as long as it matches on the first 5 chars) record.I need to search through the whole file to identify any record that first 5 chars are duplicated and only pick the last record.
I hope this helps...

??

4

As suggested by RTM you can use grep.
Place your "search patterns" in a file say patternfile
and search for them in your datafile

/usr/xpg4/bin/grep -f patternfile datafile

As always you can redirect the output to a file of choice

Chill
enc.

5

The problem is that I don't know my exact 'search pattern'.

It can be any duplicated record in a file.

It's like I need to do some clean up in the house.

6

You will get a decent answer if you specify the whole problem, not just the pieces that you think you need.

Can you explain how you are identifying duplicates? For example, it sounds like you need sort.

sort allows you to sort on a keyfield and eliminate duplicates in the keyfield. If we knew what was needed we could probably find an answer.

7

Sorry about my poor English.
I won't have totally identical records. Maybe two records will be like this:

12345apple
12345pear
12345orange

I need to only pick '12345orange' because it's the last one that showed up with duplicated '12345'. I only care first 5 charactors in each row.

The whole file will be like the above format. I only care the first 5 charactors in each row. If duplicates happens, I only need to pick the last occurence. If no duplicates, I need to pick that only one.

Sort command is a good hint. I am thinking how to utilize it...

8

Okay - try something like this:

#!/bin/ksh 
# file: dedup.sh  (be sure to chmod +x dedup.sh )
# $1 = file name - first parameter
# usage: dedup.sh myfile > newfile
old_rec="-1"
            # sort the file
sort $1 -o $1
            # process the sorted file
while read rec
do

# check first five chars of the record
        # first time thru the loop
    if [ "$old_rec" = "-1" ]; then
        old_rec="$rec"
        old_key=`expr substr "$rec" 1 5`
        continue
    fi
        # check if record keys match
    rec_key=`expr substr "$rec" 1 5`
    if [ "$old_key" = "$rec_key" ]; then
        # they match so keep reading file, use latest record
        old_rec="$rec"
        continue
    fi
# we have a new key, print last record
    echo "$old_rec"
    # set up for new record
    old_rec="$rec"
    old_key=`expr substr "$rec" 1 5`
# read from $1 the filename 
done < $1

exit

This works on a dummy file with 20 records. Print the last of the duplicated keys. key is the first five characters of the record

9

Thanks Jim.
Hi, it's working!
I was trying to use 'sort -un 'command to pick the last duplicate.
But this command can only pick from the 3 occurence, it will not recoginize from the 4th occurence.

I ran this script and found out that it does not print the last record in the file. So I put a echo "$old_rec" command right outside the while statement. Now it's working.

But I do have a question about changing shell within script.
Can I do that?

My script was started with using '#/bin/sh'. I don't know what shell this is. I was trying to copy your script to mine. but it won't run well with this shell. so I add one line like this:
/usr/bin/ksh
to start your script. after processing, I return back to my shell by typing
/usr/bin/sh
It seems weird because I will be prompted with $ and I have to type exit and get out twice, but the result is good.

10

>>/usr/bin/ksh
>>to start your script. after processing, I return back to my >>shell by typing
>>/usr/bin/sh

By doing above 2 things u are starting 2 more new shells over your current shell ; U had to exit 2 times to retain the current shell.

To know the shell u are working on ...

grep "userid" /etc/passwd

it gives the shell u are working on unless it is changed by .profile.

If shell gives prblem some times run the script like this .

./script

Hope this helps.