Hi all,
I have created a script which adding two columns and removing two columns for all files.
Filename: Cust_information_1200_201010.txt
Source Data:
"1","Cust information","123","106001","street","1-203 high street"
"1","Cust information","124","105001","street","1-203 high street"
2) Extract SID and PID from a file
Filename:Cust_information_1200_201010.txt
SID=1200
PID=201010
Output required - ( first and second columns removed and last two columns added)
"123","106001","street","1-203 high street","1200","201010"
"124","105001","street","1-203 high street","1200","201010"
Currently I am using the below script to execute 40 GB file. It is taking lot of time and also creating temporary files(lst files).
it is occupying double the space.
I am unable to tune the script using with in a single command can i add two columns and remove two columns in unix.
for file4 in `ls $TempDir|egrep '(.*Detail.*txt)'`
do
filename=`echo $file4 |cut -d "." -f1`
Var3=`echo $file4 |cut -d "." -f1 |awk -F[_-] '{print $(NF)}'`;Var4=`echo $file4 |cut -d "." -f1 |awk -F[_-] '{print $(NF-1)}'`; sed -e "s/^.*$/&,$Var3,$Var4/g" $TempDir/$file4 >$TempDir/$filename".lst"
awk -F, '{ OFS=","; $1=""; $2=""; print }' $TempDir/$filename".lst"|sed s/^,,//g >$TempDir/$filename".csv"
rm -f $file4
done
rm -f $TempDir/*Detail_*.lst
Any help greatly appreciated