Recursivley search files and replace the characters

Shell Programming and Scripting
1 
1st_FILE.cnf
Here is my sample test file [TST_FILE].
This will be replaced by [FILE_NAME].

2nd_FILE.cnf
This is my 2nd test file [2ND_TST_FILE]..This is agian a test file for [REASON] purpose.

variable.txt
TST_FILE=1st_FILE.txt
FILE_NAME=1st_FILE.txt
2ND_TST_FILE=2nd_FILE.txt
REASON=test

I have 2 sample config file under directory structure /tmp/test where variables are recognize by [].
So the task is, recursilvely go thru parent dir /tmp/test and find out the files with names *.cnf having placeholders for variables.
it's not necessary that all .cnf files got placeholders.

let's say

/tmp/test/confa/1st_FILE.cnf
/tmp/test/confa/readme.1stfile.txt
/tmp/test/confa/3rd_FILE.cnf 

/tmp/test/confb/2nd_FILE.cnf
/tmp/test/confb/readme.2ndfile.txt

so it will only list /tmp/test/confa/1st_FILE.cnf and /tmp/test/confb/2nd_FILE.cnf where placeholders are there.
not /tmp/test/confa/3rd_FILE.cnf ,
Then it will go each file one by one and replace each place holder to the actual ones based on file /tmp/test/variable.txt

e.g.

after replacement final look should look like

/tmp/test/confa/1st_FILE.cnf.out

Here is my sample test file 1st_FILE.txt.
This will be replaced by 1st_FILE.txt.

/tmp/test/confb/2nd_FILE.cnf.out

This is my 2nd test file 2nd_FILE.txt..This is agian a test file for test purpose.

I know, how to find the files recursively that matches [].
but my question is how can I bind variables from variable.txt to the filename that matched.

find . -type f -exec grep -lIi "\[" {} \; | while read name
> do
> echo $name
> done

--> this will display the files with []

now how can I loop these files with variable.txt and replace the variables with actual values.

any ideas????

2

If you have a list of those files, e.g. in a variable, you could try sth like (untested)

awk 'NR==FNR {R[$1]=$2; next} {for (r in R) gsub ("["r"]", R[r]); print > FILENAME".new"}' FS="=" variable.txt $filelist
3

Thanks Rudi.
however this is not working

my o/p looks like testing with one config file

cat 1st_FILE.cnf.new
iHere is my sample test file [1st1st2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt.txt1st2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt.txt1st2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt.txt1st2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt.txttest.txt1st1st2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt.txt1st2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt.txt1st2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt.txt1st2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt.txttest.txt1st1st2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt.txt1st2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt.txt1st2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt.txt1st2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt.txttest.txt1st1st2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt.txt1st2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt.txt1st2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt.txt1st2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt.txttest.txt1st1st2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt.txt1st2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt.txt1st2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt.txt1st2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt.txttest.txt1st1st2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt.txt1st2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt.txt1st2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt.txt1st2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt.txttest.txt1st1st2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt.txt1st2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt.txt1st2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt.txt1st2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt.txttest.txt1st1st2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt.txt1st2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt.txt1st2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt.txt1st2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt.txttest.txt].
1st1st2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt.txt1st2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt.txt1st2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt.txt1st2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt.txttest.txthis will be replaced by [1st1st2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt.txt1st2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt.txt1st2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt.txt1st2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt.txttest.txt1st1st2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt.txt1st2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt.txt1st2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt.txt1st2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt.txttest.txt1st1st2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt.txt1st2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt.txt1st2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt.txt1st2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt.txttest.txt1st1st2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt.txt1st2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt.txt1st2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt.txt1st2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt.txttest.txt1st1st2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt.txt1st2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt.txt1st2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt.txt1st2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt.txttest.txttesttest1st2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt.txt1st1st2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt.txt1st2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt.txt1st2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt.txt1st2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt2nd_FILE.txt.txttest.txt].

so it takes all the variables and replace it within [] for multiple times.
also not removing [].

cat ./1st_FILE.cnf
iHere is my sample test file [TST_FILE].
This will be replaced by [FILE_NAME].

cat config.txt
TST_FILE=1st_FILE.txt
FILE_NAME=1st_FILE.txt
2ND_TST_FILE=2nd_FILE.txt
REASON=test
4

I didn't think it was THAT picky. Try

awk 'NR==FNR {R[$1]=$2; next} {for (r in R) gsub ("\["r"\]", R[r]); print > FILENAME".new"}' FS="=" variable.txt $filelist
5

I'm sorry , I already tried "\[" in AWK, but it throws me below error

awk 'NR==FNR {R[$1]=$2; next} {for (r in R) gsub ("\["r"\]", R[r]); print > FILENAME".new"}' FS="=" config.txt 1st_FILE.cnf
awk: warning: escape sequence `\[' treated as plain `['
awk: warning: escape sequence `\]' treated as plain `]'

However,it creates the same file with all the variables (although it's showing warning.)

6

This is my result with above snippet and samples:

iHere is my sample test file 1st_FILE.txt.
This will be replaced by 1st_FILE.txt.

The problem is awk regards "[" and "]" as special chars to build a regex. You need to somehow persuade it to accept the escaped "[" and "]".

---------- Post updated at 16:45 ---------- Previous update was at 16:43 ----------

OK, got this one:

awk 'NR==FNR {R[$1]=$2; next} {for (r in R) gsub ("[[]"r"[]]", R[r]); print}' FS="=" file1 file2
iHere is my sample test file 1st_FILE.txt.
This will be replaced by 1st_FILE.txt.

Give it a shot...

1 Like
7

thanks, it's now working fine.

8

how to work this out if situation remains same only difference is now cnf file is a XML one instead flat file name=value

1st_FILE.cnf
<FILE_NAME> [TST_FILE]</FILE_NAME>
<FILE_VALUE> [FILE_NAME]</FILE_VALUE>

o/p should be

1st_FILE.cnf.new
<FILE_NAME>1st_FILE.txt</FILE_NAME>
<FILE_VALUE>1st_FILE.txt</FILE_VALUE>

Only tag name are going to same across different cnf files only variable names are different.
<FILE_NAME> and <FILE_VALUE>

9

Did you try to run that script on your new file? If I do, this is the result:

<FILE_NAME> 1st_FILE.txt</FILE_NAME>
<FILE_VALUE> 1st_FILE.txt</FILE_VALUE>
10

I tried, for me the o/p looks like

<FILE_NAME>[TST_FILE</FILE_NAME>
<FILE_VALUE>[FILE_NAME</FILE_VALUE>

--> removed only extreme right ], not even replacing it

---------- Post updated at 04:36 AM ---------- Previous update was at 04:17 AM ----------

no worries, I was doing some little crappy mistake. Now I corrected it and working fine. thanks for your heads up.

---------- Post updated at 08:31 AM ---------- Previous update was at 04:36 AM ----------

now if variable.txt changed then how to reflect the change in .cnf file.

variable.txt
TST_FILE=1st_FILE.txt
FILE_NAME={1st_FILE.txt==}
2ND_TST_FILE=2nd_FILE.txt
REASON=test

awk is not working as o/p now looks like

1st_FILE.cnf.new
<FILE_NAME>1st_FILE.txt</FILE_NAME>
<FILE_VALUE>{1st_FILE.txt</FILE_VALUE>

whereas it should look like

1st_FILE.cnf.new
<FILE_NAME>1st_FILE.txt</FILE_NAME>
<FILE_VALUE>{1st_FILE.txt==}</FILE_VALUE>

this is due to fact that FS is "="

11

Yes it is - any ideas from your side?

12

honestly I don't have any clue how to consider only one string as filed separator(=) when there are multiple same characters resides in the value.

so how can I instruct awk to only consider first character(=) as field separator later same characters can be considered as normal characters.

like

FILE_NAME={1st_FILE.txt==}

I tried

awk -F"{|}" '{print $2}'
awk -F"{|}|=" '{print $2}'

nothing doing well.....so stuck....

13

Well, I don't think there's a general solution for all possible scenarios. In this special case, try

awk 'NR==FNR {X=$0; gsub($1 FS,"",X);R[$1]=X;  next} {for (r in R) gsub ("[[]"r"[]]", R[r]); print}' FS="=" file1 file2
<FILE_NAME> 1st_FILE.txt</FILE_NAME>
<FILE_VALUE> {1st_FILE.txt==}</FILE_VALUE>
1 Like
14

thanks Rudi......it gives me more close to the result.