Hello all,
I have 2 files, the first (indexFile1) contains start offset and length for each record inside the second file. The second file can be very large, each actual record start offset and length is defined by the entry in indexFile1. Since there are no records separators wc-l returns 0 for the second file, no matter how large its size actually is.
I want to gather all the records one at a time and write them out to a new file individually.
What is the best way to approach this processing?
I suspect I will have trouble reading a whole large file into a variable (using awk) and then use a cut command on the variable contents to collect my record in the form:
FileContent=$(awk '{print $0}' largeFile2) # this is where I think I have a problem :
# LINE contains start and offset identifying each record in largeFile2
while read LINE;do
pass=1
for results in $LINE; do
if [[ $pass -eq 1 ]];then
from=$results
pass=2
else to=$results
fi
done
(( from=$val1+1 ))
(( to=$val1+$val2 ))
newOut=$(echo $FileContent|cut -c $from-$to)
echo $newOut >> newfile
done < indexFile1
I have it working ok for small size of largeFile2. I can see a problem when the size of file2 gets large.
I hope you can give me some suggestions on how to do this better.
Thanks!