I have 4000 list files and 4000 sequence data files. Each list file contains a number of 'headers' and data file contains 'header and data'. I would like to extract data from the data file using the list file and write into a new file. As each of the files are quite large, an efficient piece of script(preferably bash) will be much appreciated. Example below:
The code produces exactly the similar 'Output' as the input 'DataFile'. The headers are not similar. List file contain less headers. For example 'contig00001 length=477 numreads=22' is not present in the 'list' file and should not be retrieved.
I've 4000 list and data files. Is there a way to loop over all the files in one run?
I'm sure there's a way to loop over your data files. However you've told us nothing about them other than you have a lot of them so it's a little hard to help. Do they have filenames? Do you have a list of them? Are they organized in any way?
[quote="r0h0n,post:2,topic:278377"]
listFile="/path/listFile.txt"
dataFile="/path/dataFile.txt"
>outputFile
cat $dataFile | while read line You've replaced the useless use of grep with a useless use of cat.
I suggest this instead:
while read RECORD LENGTH NUMREADS
do
IFS="=" read G LENGTH <<< "${LENGTH}"
IFS="=" read G NUMREADS <<< "${NUMREADS}"
...
done < listfile
Still working on the data processing. Should post in a while.
---------- Post updated at 09:52 AM ---------- Previous update was at 09:43 AM ----------
Does the output data have to be produced in the same order as the input data? That's going to be a royal pain because it's in random order, whereas the input file is sorted. That means starting over at the top of the datafile for every record instead of reading as you go.
---------- Post updated at 09:55 AM ---------- Previous update was at 09:52 AM ----------
Well, that turns this on its head. I hope you mean data file, because if the list file contains no listings that makes no sense at all... What does the data file look like then? What relationship do the list fields have with the data to be retrieved? Is length bytes? What does numreads mean? How do we calculate some sort of offset from this?
The list fiiles are as follows: List1.txt, List2.txt, ....., List4000.txt
And the corresponding data files are: DataFile1.txt, DataFile2.txt, ...., DataFile4000.txt
List1.txt will be used for DataFaile1.txt and so on.
Contents of both the files are not sorted. For example:
List1.txt
One way would be, read one header from the list file, say 'contig00002 length=653 numreads=34', search for it in the DataFile1.txt and retrieve the following:
'>contig00002 length=530 numreads=27
GGGCTGACGTGGCCGCTAATACGACTCACTATAGGGAGAGGAGGATAGGG
AGCTGAGCAGCCAGTGACAGGATCCAGCTCCAGGGGGTGAATGGGGATGG'
If nothing's sorted and it has to produce output in arbitrary order, there's no efficient solution, since there's no way but sheer brute force for finding the records.
for ((N=1; N<=4000; N++))
do
while read RECORD REST
do
# Open datafile into FD 5
exec 5<DataFile${N}.txt
# Find that entry in the file, leaving file pos at the next line
( if grep -m 1 "^\>${RECORD}"
then
# Print each line until we find the next record, or EOF
while read LINE
do
[ "${LINE:0:1}" == ">" ] && break
echo "$LINE"
done
fi ) <&5
# Close the file
exec 5<&-
done < List${N}.txt > output${N}.txt
done