Using sed to edit multiple files

Shell Programming and Scripting
1

Created a shell script to invoke sed to edit multiple files, but am missing something.
Here's the shell script:

oracle:$ cat edit_scripts.sh
#!/bin/sh
#------------------------------------------------------------------------------
# edit_scripts.sh
#
# This script executes sed to make global edits to the template files
#------------------------------------------------------------------------------
for i in *.sh; do
  sed -f script.sed < $i
done
#
for i in *.sql; do
  sed -f script.sed < $i
done

As you can see, I'm using a script file 'script.sed' to supply the actual editing commands:

oracle:$ cat script.sed
s/@yymmdd@/181016/g
s/@gi patch num@/12345678/g
s/@jvm patch num@/87654321/g

And for testing purposes, I have one .sh script to be edited:

oracle:$ ll *.sh
-rwxr-x--- 1 oracle oinstall 359 Jan  3 14:30 edit_scripts.sh
-rw-r--r-- 1 oracle oinstall 163 Jan  3 14:46 sed_sample.sh

oracle:$ cat sed_sample.sh
#-------------------
#  This should be the date: @yymmdd@
#  This should be the gi patch num:  @gi patch num@
#  This should be the jvm patch num: @jvm patch num@

But when I execute edit_scripts.sh, sed_sample.sh is not modified:

oracle:$ ./edit_scripts.sh
#!/bin/sh
#------------------------------------------------------------------------------
# edit_scripts.sh
#
# This script executes sed to make global edits to the template files
#------------------------------------------------------------------------------
for i in *.sh; do
  sed -f script.sed < $i
done
#
for i in *.sql; do
  sed -f script.sed < $i
done
#-------------------
#  This should be the date: 181016
#  This should be the gi patch num:  12345678
#  This should be the jvm patch num: 87654321
./edit_scripts.sh: line 12: *.sql: No such file or directory

oracle:$ cat sed_sample.sh
#-------------------
#  This should be the date: @yymmdd@
#  This should be the gi patch num:  @gi patch num@
#  This should be the jvm patch num: @jvm patch num@
2

Depending on your OS...
either:

sed -i -f script.sed $i

OR

{ rm $I; sed -f script.sed > $i; } < $i

OR
other multiple other ways...

1 Like
3

The problem is in this line:

sed -f script.sed < $i

First, sed doesn't need a redirection because it can work on files. Notice that i quoted "$i" because otherwise files with blanks in their names would break your script. It is generally a good idea to religiously quote whatever is quotable in shell scripts - it just comes with the impulse to convert the infidels, like i am doing to you right now. :wink:

sed -f script.sed  "$i"

Second, sed puts it output to <stdout> as per default. Probably it did do the changes but they were shown on your screen, not your file. To make the changes lasting you need to redirect this output into a (new!) file and only then move that file over the old one. As an additional measure of caution only do so if sed returned 0. This guards against i.e. disk-full-conditions where the output cannot be written (fully) to the disk and you move then a corrupt script over a perfectly fine template:

if sed -f script.sed  "$i" > "$i".tmp ; then
     mv "$i".tmp "$i"
fi

I hope this helps.

bakunin

4

The first example worked for me, even though the man page for sed on my system does not list the '-i' switch. My system is running Oracle Linux, which is derived from RHEL.

Thanks for the assistance.

5

It does so for a reason because it is not advisable to use the "-i" switch at all, even in the few versions of sed not adhering to the standards which have it, because the the standards also do NOT have the switch at all.

The reason is that "behind the curtain" the "-i" switch does the same as i said: creating a temporary file and then copying it over the original. It does NOT really do "in-place-editing" as the documentation says, it does in fact the same every script programmer does. This in itself is not bad but: if you do it yourself you have control over where the temporary file goes and that it is removed afterwards. With the "-i" switch you don't have this control and in case of an error it may well be that it is left stuck somewhere and such a leftover chunk of junk is filling up valuable disk space.

Furthermore it changes file permissions and ownership, which a real in-place-editing wouldn't do. Consider the following example:

$ id
uid=1000(edstevens) gid=1000(edstevens) groups=1000(edstevens),999(users)

$ ls -l
total 189960
-rw-rw-r-- 1 bakunin users       90 Nov 13 15:16 script.original

That means you can edit script.original because you are in the group "users" which has write privilege.

Now run this - and guess, what:

$ sed -i '...<whatever, it doesn't matter>...' script.original

$ ls -l
total 189960
-rw-rw-r-- 1 edstevens edstevens       99 Jan 04 18:00 script.original

Did you notice? The ownership is changed! Now, that happens when you create a temporary file too, but there you are aware that you are dealing with a new file. Here you have no reason to suspect because you have been (wrongly) told it will be the same file plus your changes. It isn't. It will have a new inode number, it will be located somewhere else on the disk physically, etc.. This is why you should stay away from this switch. Have i mentioned that it doesn't conform to the standard either? If you want to create portable script which will not work somewhere but mysteriously not so somewhere else you better adhere to the standards slavishly. They define what you can rightfully expect from a system. If you expect more than chances are you will be disappointed sooner or later.

I hope this helps.

bakunin

3 Likes