Splitting text file to several other files using sed.

Shell Programming and Scripting
1

I'm trying to figure out how to do this efficiently with as little execution time as possible and I'm pretty sure using sed is the best way. However I'm new to sed and all the reading and examples I've found don't seem to show a similar exercise:

I have a long text file (i'll call it all_files.txt) listing all the files on the system, each line showing the checksum, permissions, date, and file name with path. For example:

683706D9 104775 Sep 27 12:00:04 1999 /bin/Audio
4C799E06 100775 Nov 14 17:33:11 1997 /bin/Blkfsys
C851669A 104775 Oct 04 14:08:38 1996 /bin/Dev16
CA4B42E7 100775 Nov 21 11:58:06 1996 /bin/Dev16.ansi
FF4396D0 100775 Oct 04 14:06:03 1996 /bin/Dev16.par

Some of these files are categorized according to some other text files listing the files belonging to that category. For example, the file catA.dat may be the following:

/bin/Dev16.par
/bin/some_other_file
/home/stuff/another_file

Similar lists would exist for catB.dat and catC.dat.

What should happen is all the lines in the original file which belong to a certain category will be deleted from the original file and copied to a new file, say catA_list and catB_list, etc. So in the end only the files not assigned to any category are left in all_files.txt.

Is there an easy way to do this? I've figured out how to use sed to delete lines, but to output them to 3 different files based on matches from reference text files is confusing me. Any ideas would be greatly appreciated!!

2

The following example demonstrates how to write results
out to 3 different files

#!/usr/bin/ksh

tmp=file.$$

cat <<EOT >$tmp
683706D9 104775 Sep 27 12:00:04 1999 /bin/Audio
4C799E06 100775 Nov 14 17:33:11 1997 /bin/Blkfsys
C851669A 104775 Oct 04 14:08:38 1996 /bin/Dev16
CA4B42E7 100775 Nov 21 11:58:06 1996 /bin/Dev16.ansi
FF4396D0 100775 Oct 04 14:06:03 1996 /bin/Dev16.par
EOT

sed -n -e '/^68/w ./out1' -e '/^C8/w ./out2' -e '/^CA/w ./out3' $tmp

rm $tmp
exit 0
3

Thanks! That helps, although I'm seeing other complications here. For one, I won't know what I'm searching for since this will come as input in from other files (catA_list, catB_list, catC_list). I was initially thinking I could use a loop to read each file from the category lists, working on each category at a time:

while read AFILE
do
sed -n -e '\|"$AFILE"$| {
w /catA_list
d
}' <all_files.txt>tmpfile
done<catA.dat

However 2 more loops would have to be used for Category B and Category C files. Essentially this would be looping through the file many times. I'm not even sure this would work correctly. I'm thinking there has to be a more efficient way to do this.

4

Ok, I've figured out a solution that mostly works, however I can't get it to work when passing an argument into the sed regular expression. I basically have this for extracting any line for a file in category A:

while read AFILE
do
sed -e '\|'"$AFILE"'$|{
w /tmp/catA_list
d
}' </tmp/all_files>/tmp/non_AFiles

done<catA.dat

The regular expression works fine if I substitute a specific file name. But passing the argument this way it will only find one of the files in the catA list and not the others. Is there a better way to pass the argument here?

Also the file name includes the full path, hence why '/' is not being used as a delimiter for the expression.