How to parse through a file and based on condition form another output file

UNIX for Advanced & Expert Users
1

I have one file say CM.txt which contains values like below.Its just a flat file

1000,A,X
1001,B,Y
1002,B,Z
...
..

total around 4 million lines of entries will be in that file.
Now i need to write another file CM1.txt which should have

1000,1
1001,2
1002,3
....
...
..

Here i am putting 1,2,3 based on condition in first file
if A + X then 1
if B+Y then 2
if B+Z then 3
These are the only three conditions that is possible.

Now can anybody suggest a way to write a Unix script which do this task a shortest possible time,since it has huge number of entries in the file.
Please give me a best solution / example that will help me .

2

what about
A+Y
A+Z
B+X

a 2 by 3 matrix can have six ouputs
so, does the solution need to figure six or just your intitial three possibilities?

3

Only three possibilities as mentioned above by me

4

Try this:

sed -e 's/A,X/1/g' -e 's/B,Y/2/g' -e 's/B,Z/3/g' CM.txt

you can put this in a file.sed and run it like so:

sed -f sedfile file

Using sed

5

Expanding on what nj78 said:

sed 's/A,X/1/;s/B,Y/2/;s/B,Z/3/' CM.txt > CM1.txt
  1. Performance considerations

I suppose awk and sed to be about the same speed, so doing the same in awk will probably gain (or loose) some seconds per run. We found that out when dealing with other problems involving very large files here that sed's and awk's work speeds are about level and way ahead of the crowd. Other probable solutions like perl, python, shell scripts, what else, ... will be considerably slower, perhaps the slowest being shell script - it is simply not built for digesting huge loads of data.

For a discussion of this have a look here for instance.

  1. Security considerations

I in your place would not be so sure about what can be and what can't - strange things happen all the time. ;-)) In your case i would not rely on your knowledge that only three combinations are possible. Even if that means a slight performance degradation i would write the sed script that way:

sed 's/A,X/1/;s/B,Y/2/;s/B,Z/3/;s/,[^123].*$/ERROR/' CM.txt > CM1.txt

The last clause does the following: all your "legal" lines are by now changed and end in either "1", "2" or "3". If a line now ends in something else ([^123] means: "neither 1 nor 2 nor 3") it was not changed by the 3 clauses before and and is therefore not a legal (expected) combination, which i flag with a big "ERROR", so it will easily be found. In a second step i would invest the time to validate my output:

if [ $(grep -c "ERROR" CM1.txt) -eq 0 ] ; then 
     print - "File passed check, everything is ok"
else
     print - "Something has gone wrong, check your file"
fi

Bottom line: it is not a problem to deal with errors as long as you are prepared for it.

I hope this helps.

bakunin

6

Excellent addition of the error checking.

7

Its great and easy.

thanks for your reply.
:):):):):):slight_smile: