Loop through text file > Copy Folder > Edit XML files in bulk?

biscuitcreek · April 27, 2011, 4:26pm

I have a text file which contains lines in this format - it contains 105 lines in total, but I'm just putting 4 here to keep it short:

58571,east_ppl_ppla_por
58788,east_pcy_hd_por
58704,east_pcy_ga_por
58697,east_pcy_pcybs_por

It's called id_key.txt

I have a sample folder called start_folder

The folder contains 2 xml files:

SearchBoxRN.xml
ShoppingHomePG.xml

Each xml file contains, amongst other things, the following text:

user:responsibilityKey="east_ppl_ppl_por"

I'd really like to be able to loop through the id_key.txt file, and for each line - e.g. the first line in id_key.txt:

Copy the start_folder and its contents

Rename the folder from "start_folder" to the first bit of the current line from the id_key.txt file - e.g. 58571
Update SearchBoxRN.xml and change user:responsibilityKey="east_ppl_ppl_por" to user:responsibilityKey="east_ppl_ppla_por"
Update ShoppingHomePG.xml and change user:responsibilityKey="east_ppl_ppl_por" to user:responsibilityKey="east_ppl_ppla_por"

This seems like a long shot to me, but I'm no expert.

I've tried looking at using a windows batch file on a PC, but we also have a unix environment at work, and if it's possible to do this task in UNIX, then that'd be great, because the alternative involves manually editing 525 XML files by hand, and creating 315 folders...

I'm afraid I am a real novice with unix. I can do simple stuff like copy files, delete files, but have never done anything with vi or other unix text editors.

Any advice much appreciated.

Thanks!

Corona688 · April 27, 2011, 5:42pm

If you have sed -i:

while IFS="," read N STR
do
        cp -R /path/to/start_folder /path/to/$N
        sed -i "s/east_ppl_ppl_por/$STR/g" /path/to/$N/SearchBoxRN.xml
        sed -i "s/east_ppl_ppl_por/$STR/g" /path/to/$N/ShoppingHomePG.xml
done < /path/to/id_key.txt

rdcwayx · April 27, 2011, 8:45pm

$ cat id_key.txt
58571,east_ppl_ppla_por
58788,east_pcy_hd_por
58704,east_pcy_ga_por
58697,east_pcy_pcybs_por

$ cat  SearchBoxRN.xml
58571:responsibilityKey="east_ppl_ppl_por"
58704:responsibilityKey="east_ppl_ppl_por"
58697:responsibilityKey="east_ppl_ppl_por"
58788:responsibilityKey="east_ppl_ppl_por"
58541:responsibilityKey="east_ppl_ppl_por"
58371:responsibilityKey="east_ppl_ppl_por"

$ awk -F , 'NR==FNR{a[$1]=$2;next} 
        /responsibilityKey/ {if ($1 in a) $2="responsibilityKey=\"" a[$1] "\""}1' id_key.txt FS=":" OFS=":" SearchBoxRN.xml

58571:responsibilityKey="east_ppl_ppla_por"
58704:responsibilityKey="east_pcy_ga_por"
58697:responsibilityKey="east_pcy_pcybs_por"
58788:responsibilityKey="east_pcy_hd_por"
58541:responsibilityKey="east_ppl_ppl_por"
58371:responsibilityKey="east_ppl_ppl_por"

biscuitcreek · April 28, 2011, 7:00am

Thanks for your help with this.

I created this .sh file:

while IFS="," read N STR
do
        cp -R /home/1320238/test1/start_folder /home/1320238/test1/$N
        sed -i "s/east_ppl_ppl_por/$STR/g" /home/1320238/test1/$N/SearchBoxRN.xml
        sed -i "s/east_ppl_ppl_por/$STR/g" /home/1320238/test1/$N/ShoppingHomePG.xml
done < /home/1320238/test1/id_key.txt

When I ran it, I got this output:

1320238:PLAY$ test.sh
sed: Not a recognized flag: i
Usage:  sed [-n] [-u] Script [File ...]
        sed [-n] [-u] [-e Script] ... [-f Script_file] ... [File ...]
sed: Not a recognized flag: i
Usage:  sed [-n] [-u] Script [File ...]
        sed [-n] [-u] [-e Script] ... [-f Script_file] ... [File ...]
sed: Not a recognized flag: i
Usage:  sed [-n] [-u] Script [File ...]
        sed [-n] [-u] [-e Script] ... [-f Script_file] ... [File ...]
sed: Not a recognized flag: i
Usage:  sed [-n] [-u] Script [File ...]
        sed [-n] [-u] [-e Script] ... [-f Script_file] ... [File ...]
sed: Not a recognized flag: i
Usage:  sed [-n] [-u] Script [File ...]
        sed [-n] [-u] [-e Script] ... [-f Script_file] ... [File ...]
sed: Not a recognized flag: i
Usage:  sed [-n] [-u] Script [File ...]
        sed [-n] [-u] [-e Script] ... [-f Script_file] ... [File ...]
1320238:PLAY$

The .sh file did do something though - it created a folder for each of the following entries from the id_key.txt file:

58571,east_ppl_ppla_por
58788,east_pcy_hd_por
58704,east_pcy_ga_por
58697,east_pcy_pcybs_por

Though not for the last line for some reason.

However, it didn't update the "east_ppl_ppl_por" with the value from the text file. Is that because the "i" flag isn't recognised by "sed"?

Does that mean the "sed" route won't work?

I also tried the method suggested by "rcdwayx", but got this output:

1320238:PLAY$ test2.sh
test2.sh: $:  not found.
test2.sh[2]: 58571,east_ppl_ppla_por:  not found.
test2.sh[3]: 58788,east_pcy_hd_por:  not found.
test2.sh[4]: 58704,east_pcy_ga_por:  not found.
test2.sh[5]: 58697,east_pcy_pcybs_por:  not found.
test2.sh[7]: $:  not found.
test2.sh[8]: 58571:responsibilityKey=east_ppl_ppl_por:  not found.
test2.sh[9]: 58704:responsibilityKey=east_ppl_ppl_por:  not found.
test2.sh[10]: 58697:responsibilityKey=east_ppl_ppl_por:  not found.
test2.sh[11]: 58788:responsibilityKey=east_ppl_ppl_por:  not found.
test2.sh[12]: 58541:responsibilityKey=east_ppl_ppl_por:  not found.
test2.sh[13]: 58371:responsibilityKey=east_ppl_ppl_por:  not found.
test2.sh[15]: $:  not found.
test2.sh[18]: 58571:responsibilityKey=east_ppl_ppla_por:  not found.
test2.sh[19]: 58704:responsibilityKey=east_pcy_ga_por:  not found.
test2.sh[20]: 58697:responsibilityKey=east_pcy_pcybs_por:  not found.
test2.sh[21]: 58788:responsibilityKey=east_pcy_hd_por:  not found.
test2.sh[22]: 58541:responsibilityKey=east_ppl_ppl_por:  not found.
test2.sh[23]: 58371:responsibilityKey=east_ppl_ppl_por:  not found.

I'm not sure what that means?

Any advice much appreciated.

Thanks!

palanisvr · April 28, 2011, 7:38am

for i in `cat id_key.txt`
do

#capture the values in the variables
first_bit=`echo $i | awk -F ',' '{print $1}'`
second_bit=`echo $i| awk -F ',' '{print $2}'`
#make sure that variables  are not null
if [ ! -z $first_bit  -a  ! -z $second_bit  ]
then
cp -R  ./start_folder $first_bit
sed s/east_ppl_ppl_por/$second_bit/ ./start_folder/SearchBoxRN.xml > $first_bit/SearchBoxRN.xml
sed s/east_ppl_ppl_por/$second_bit/ ./start_folder/ShoppingHomePG.xml > $first_bit/ShoppingHomePG.xml
fi

done

biscuitcreek · April 28, 2011, 7:46am

Thanks for the suggestion / advice!

I created a new "test3.sh" file:

for i in 'cat id_key.txt'
do

#capture the values in the variables
first_bit=`echo $i | awk -F ',' '{print $1}'`
second_bit=`echo $i| awk -F ',' '{print $2}'`
#make sure that variables  are not null
if [ ! -z $first_bit  -a  ! -z $second_bit  ]
then
cp -R  ./start_folder $first_bit
sed s/east_ppl_ppl_por/$second_bit/ ./start_folder/SearchBoxRN.xml > $first_bit/SearchBoxRN.xml
sed s/east_ppl_ppl_por/$second_bit/ ./start_folder/ShoppingHomePG.xml > $first_bit/ShoppingHomePG.xml
fi

But get this error:

test3.sh: 0403-057 Syntax error at line 1 : `for' is not matched.

Do you know what that means?

Thanks again for your help

palanisvr · April 28, 2011, 7:47am

I think you havent copied the code completly , you have missed copying "done" .

panyam · April 28, 2011, 7:49am

Hello,

for i in 'cat id_key.txt'

Here ' is not single quote ( ' ); instead it should be a back quote ( ` ).

 
for i in `cat id_key.txt`

biscuitcreek · April 28, 2011, 8:26am

Thanks palanisvr - your help is really, really, really appreciated. This will save so many hours of work. Thank you again!

---------- Post updated at 01:26 PM ---------- Previous update was at 01:07 PM ----------

I'm sorry to come back asking for more.

The script works really well if there are 2 XML files in the start_folder directory.

If there is only 1 XML file in the start_folder directory, the script copies the start_folder into each new directory - is there a way to prevent that from happening?

Any advice much appreciated.

Thanks

palanisvr · April 28, 2011, 11:55am

#Additional condition has been added to make sure script process the files unless there exist two files in the start folder.

#verify whether files "SearchBoxRN.xml" , "ShoppingHomePG.xml" exist in  "start_folder" Folder...

if [ `ls  ./start_folder/SearchBoxRN.xml |wc -l` -eq 1 -a  `ls  ./start_folder/ShoppingHomePG.xml |wc -l` -eq 1 ] 
then

for i in `cat id_key.txt`
do

#capture the values in the variables
first_bit=`echo $i | awk -F ',' '{print $1}'`
second_bit=`echo $i| awk -F ',' '{print $2}'`
#make sure that variables  are not null
if [ ! -z $first_bit  -a  ! -z $second_bit  ]
then
cp -R  ./start_folder $first_bit
sed s/east_ppl_ppl_por/$second_bit/ ./start_folder/SearchBoxRN.xml > $first_bit/SearchBoxRN.xml
sed s/east_ppl_ppl_por/$second_bit/ ./start_folder/ShoppingHomePG.xml > $first_bit/ShoppingHomePG.xml
fi

done

fi