Hi,
I have a large number of data files each containing simple integers from 1 to around 25000 in ascending order. However, they are not in a specific progression; some numbers are missing in each file.
For ex. datfile1 may have the numbers in order 1 2 4 6 7 8 12 ... 24996 24999
while datfile2 may have them in order 1 3 5 6 8 9 ... 24999 25000
For each data file, I want to create a new one which has only the integers greater than 16000.
For ex. newdatfile1 may have numbers 16001 16003 16004 ... 24996 24999
The problem is I know very little about UNIX. So, any help will be highly appreciated.
This is not a homework assignment. The integers are the number of days after a certain date and the data is not available for some days, so some days are missing. Its actually for a project I'm working on. I am creating a regression model using the data via ampl. Now, I would like to analyse just the data after certain date. So, I thought there must be an easy way to separate out the data I want, using UNIX.
I am using bash and there is one integer on a single line.
#!/bin/bash
echo 'processing datfile1 using while loop:'
while read x
do if [ "$x" -gt 16000 ]
then printf "%d\n" "$x"
fi
done < datfile1
echo 'processing datfile2 using awk:'
awk '$1 > 16000' datfile2
If datfile1 contains:
1
2
4
6
7
8
12
24996
24999
and datfile2 contains:
1
3
5
6
8
9
12
24999
25000
the output produced is:
processing datfile1 using while loop:
24996
24999
processing datfile2 using awk:
24999
25000
If you are running this on a Solaris system, use /usr/xpg4/bin/awk , /usr/xpg6/bin/awk , or nawk instead of /bin/awk or /usr/bin/awk .
Although this was written using bash (as you requested), this script will work with any shell that recognizes basic Bourne shell syntax.
The scripts shown just write the extracted data to standard output. The original data (datfile1 and datfile2) are not modified. If you want to save the output in files, try something like:
#!/bin/bash
echo 'processing datfile1 using while loop:'
while read x
do if [ "$x" -gt 16000 ]
then printf "%d\n" "$x"
fi
done < datfile1 > datfile1.mod
echo 'processing datfile2 using awk:'
awk '$1 > 16000' datfile2 > datfile2.mod
And then look at the contents of the files datfile1.mod and datfile2.mod after it finishes. You could also pipe the output into another stream of processes to do whatever you want with the data without creating temporary files, replace the data in the original files, or millions of other choices depending on what you want to do. If you tell us what you are trying to do, we might be able to suggest saving the output in a manner that would be appropriate for your situation.