Hello everyone,
I need some assistance with what I thought would have been a very simple script.
Purpose of Script:
Script will parse through a source file and modify (search/replace) certain patterns and output to stdout or a file. Script will utilize a "control file" which will contain what to search for and what to replace it with. "Control file" will be used against the "source file".
Script Goals:
- Comments (lines beginning with #) and blank lines in the "control file" should be ignored by the script. [DONE]
- "Control file" will contain structured data. Deliminator with be initially a blank space, will be changed to a semi-colon. [DONE]
- The script works
[This is where I need help]
Example of "Control file"
#this is a comment, should be skipped
#blank lines should also be skipped
#data is structured as SEARCH REPLACE
#ok, here is some data
search1 replace1
search2 replace2
search3 replace3
#more data
search4 replace4
Example of "Source file"
search1
search2
search3
search4
Example of the script
#!/bin/bash
while read SEARCH REPLACE
do
sed -e 's/$SEARCH/$REPLACE/' sourcefile
done < <(awk '!/^#|^$/ { print $0}' controlfile)
Example of script output to stdout
search1
search2
search3
search4
search1
search2
search3
search4
search1
search2
search3
search4
search1
search2
search3
search4
Here I would have expected a bunch of replaces outputted (replace1, replace2, replace3, ...).
I can't seem to figure out what I am doing wrong. Any help would be greatly appreciated. Thanks in advanced.
Single quotes after sed do not allow variable expansion, use double. Also, -e not necessary.
1 Like
Thanks for the quick response DGPickett. I originally tried replacing the single quotes with double quotes for the sed portion of the script with no luck. I've gone ahead and replaced all single quotes with double quotes and still get the same exact output. Here is a copy of the updated scripted:
#!/bin/bash
while read SEARCH REPLACE
do
sed -e "s/$SEARCH/$REPLACE/" sourcefile
done < <(awk "!/^#|^$/ { print $0}" controlfile)
And here is a copy of the output it yields:
search1
search2
search3
search4
search1
search2
search3
search4
search1
search2
search3
search4
search1
search2
search3
search4
Hi,
This script is not homework.
awk 'NR==FNR && !/^#|^$/ {a[$1]=$2;next;} NR==FNR {next;} {if(a[$1]) $1=a[$1]; print;} Control_file Source_file
1 Like
The while read and sed look OK, but take your < < ( ... ) and hang it on cat -vet to see what the control inputs are.
I am a sed man, so I would have used sed to clean the control up like this (\t s/b a real tab):
sed '
s/#.*//
s/[ \t]*$//
/^$/d
' control_file | while read SEARCH REPLACE
.
.
.
.
Thanks anurag.singh!
Your awk example definitively gives me the output I was after. Think there was a typo between the a and the $1, had to change the '{' to a '['. Would you know why my script does not work? As per DGPickett's remark, it appears that sed is not able to see $SEARCH and $REPLACE. sed just sees the whole output of each line as $SEARCH. Almost like its not seeing the format of "SEARCH[space]REPLACE". I can't seem to figure out how to get sed to see the expanded/variablized values of $SEARCH and $REPLACE. Thanks again.
---------- Post updated at 05:18 PM ---------- Previous update was at 05:02 PM ----------
Thanks again DGPickett but I get the same output with the usage of sed to clean up the control file. Here is a copy of the updated script:
#!/bin/bash
sed '
s/#.*//
s/[ \t]*$//
/^$/d
' controlfile | while read SEARCH REPLACE
do
sed 's/$"SEARCH"/$"REPLACE"/' sourcefile
done
Here is its output:
search1
search2
search3
search4
search1
search2
search3
search4
search1
search2
search3
search4
search1
search2
search3
search4
I think you are correct with your statement that sed does not see the expanded values for $SEARCH and $REPLACE. If I do echo $SEACH, I get the whole line echoed (not what I would expect). If I do echo $"SEARCH" I only get the first column (what I would expect). Doing echo $"REPLACE" gives the second column. Doing just a echo $REPLACE gives me blank lines. I think the scripts logic is correct - figuring I am just missing some semantic in passing variables to sed.
The side effect of putting man sed (linux) inside of the while loop is that the sourcefile is processed for each search/replace pair in the control file. May I suggest:
# Args: Control-File Source-File
NAME=$(basename ${0})
FILE=/tmp/${NAME}.${$}
trap "rm -f '${FILE}'" EXIT
awk '/^#/ { next; } NF == 2 { print "s/" $1 "/" $2 "/"; }' "${1}" > "${FILE}"
sed -f "${FILE}" "${2}"
1 Like
In post #8, sed under while loop should be (-e not really needed here):
sed "s/$SEARCH/$REPLACE/" sourcefile
Also as mentioed in post #9, sed under while, will run for each search/replace pair in the control file (source file being processed that many times, that is why output is repeated that many times), so you need to use -i option if supported.
sed -i "s/$SEARCH/$REPLACE/" sourcefile
This will modify source file itself.
I believe in such scenarios (where fields in one file are being modified based on values of fields in another file), awk provides better solutions.
One side effect of using awk as shown in post #6:
is that whitespace would not be preserved. This may, or may not be, important. But the request was to replace string "pattern" with "replacement", not replace a field with a given value with a new value. Or was it implied?
IMHO, if the request was for word/field replacements, or with more complexity, I would probably have gone with PERL with \b in the regex. I also have to admit I am not a fan of in-place updates. Yes, it is very useful at times, but it has bit me in the posterior more times than I care to count.
Well, just take it in pieces, printing variables to stderr with delimiters so you can see what is up "'$xxxx'" ! Often an extra space or a tab not a space ruins the match, and then there is no translation.
---------- Post updated at 09:06 PM ---------- Previous update was at 09:04 PM ----------
sed "s/$SEARCH/$REPLACE/"
or
sed 's/'"$SEARCH"'/'"$REPLACE"'/'
Thanks everyone for your help. m.d.ludwig's combination of awk and sed in post #9 produces just what I was looking for.