Replace a line with multiple lines

Hi

I have below line in multiple files
module load ibmdb2/client/10.1.3

I want to replace this line with below 3 lines.

INFA_CONFIG="$HOME/config/research_dw_rt_$ENV_VAR.cfg"
                DB_Ver=`grep -w 'DB_Version' $INFA_CONFIG | cut -d '=' -f 2`
                module load ibmdb2/client/$DB_Ver

Can you please help on this ?

Hello,

Welcome to the forum. We hope you find this a friendly and helpful place to get answers to your questions.

The first thing we'd normally need to know before proceeding any further is: what environment are you using here (e.g. An x86_64 system running Ubuntu 20.04 LTS, and using Bash as the shell) ? Knowing this is fairly fundamental, and would determine which solutions might be best in your particular case.

Also, if you could share any solutions you've attempted so far yourself (even if you only got part-way through finishing, or if you're just at the stage of thinking about what tools or solutions to use), that would be ideal also.

In this particular instance, I'd also ask how many files you're talking about here ? If it's in the single- or low-double digits, the easiest solution here is to copy the text you want to replace into your clipboard, and open each file in your editor of choice and just paste over the text you want to overwrite. Of course if you're talking many tens or some hundreds of files then automating the task becomes fair enough, but sometimes the simplest solution is the best, even if it is depending on doing things manually.

If you could let us know these points, we can then take things from there.

You might also want to verify the correct operation of the replacement snippet before you do any mass update. Assuming this is being inserted into some shell scripts, it does not do what you might be expecting: the grep section will not be executed in the revised files -- it is just plain text which will fail something like: bash: -w: command not found.

Thanks drysdalk for your response.
We have Red Hat Enterprise Linux Server release 6.10 using Bash as shell.
There are around 200 files and I need to use some automated way to replace these files.

I have tried below command but getting error.

$ sed 's/^10.1.3.*/

INFA_CONFIG="$HOME/config/research_dw_rt_$ENV_VAR.cfg" DB_Ver=grep -w 'DB_Version' $INFA_CONFIG | cut -d '=' -f 2 module load ibmdb2/client/$DB_Ver/g' file.txt
sed: -e expression #1, char 14: unterminated `s' command

Thanks Paul.
We have a team to test all the scripts. Are you saying it is impossible via a shell command/script ?

Hello,

Thank you for providing the requested information. The sed command you provide isn't a full command, as it stands. The s command is used to substitute one particular given string, very precisely, with another string. By default, the strings are separated by forward-slashes. So at the moment, you're telling sed you want to perform a substitution, but only providing a string to search for, and nothing to replace it with. Hence, the unterminated 's' command error.

So - you need to provide the exact string to search for, and what you want to replace it with. You can embed newlines in a sed substitution by representing newlines as \n in your script.

So, let's look at an example that expands on your solution to use a fully-qualified and completed sed substitution command.

Firstly, our input:

$ cat sample1.txt
The start
Some stuff
Some more stuff
Blah
Blah
module load ibmdb2/client/10.1.3
More stuff
Some more stuff
And now the end
$

This file includes the exact string - "module load ibmdb2/client/10.1.3" - that you want to replace.

And now, here's the script we'll use to do the substitution:

$ cat fix.sh
#!/bin/bash

for file in *txt
do
        /bin/sed -i 's%module load ibmdb2/client/10.1.3%INFA_CONFIG="$HOME/config/research_dw_rt_$ENV_VAR.cfg"\nDB_Ver=`grep -w 'DB_Version' $INFA_CONFIG | cut -d '=' -f 2`\nmodule load ibmdb2/client/$DB_Ver%g' "$file"
done
 
$

Now, there's one extra trick here that we have to use. The string we want to search for has a forward slash as part of it, as indeed do the strings we want to replace it with. Now, you'll recall my mentioning earlier that sed uses forward slashes as its field separator for the substitution command. So by default, sed would trip up over your strings, since it would think the slashes in them were intended as field separators, and you would get all kinds of errors as a result of sed thinking the number of fields wasn't correct.

To work around this, sed will in fact take the first character immediately after the substitution command as the field separator. You'll notice I went with a percent symbol here, since there is no occurrence of these in your strings. So the string we want to search for (one single line), and the one we want to replace it with (three separate lines, with newlines between them) can be fully and correctly specified.

And here's what we get when we run the script:

$ ./fix.sh
$ cat sample1.txt
The start
Some stuff
Some more stuff
Blah
Blah
INFA_CONFIG="$HOME/config/research_dw_rt_$ENV_VAR.cfg"
DB_Ver=`grep -w DB_Version $INFA_CONFIG | cut -d = -f 2`
module load ibmdb2/client/$DB_Ver
More stuff
Some more stuff
And now the end
$

Hope this helps ! If this doesn't quite get you there, or if any part of the above explanation doesn't quite make sense, please let us know and we can take things from there.

No -- you can do almost anything in a shell script, and maybe using a few basic utilities like awk.

What I am saying is that the assignment to DB_Ver was bad syntax. I see you edited it (after I posted) to insert back-ticks. These are actually deprecated in favour of the $( ... ) syntax.

It is good that you have a team to test all the scripts. But doing a bad edit through an automated process means you just gave them 200 buggy scripts to test and fix. And the next fix means a different edit, unless you reinstate the 200 files from an archive first.

I don't see where your sed 's/^10.1.3*/ came from. That has at least four errors: unbalanced quotes, ^ anchor to start of line, . being a wildcard, and * looking for zero or more 3s. You can just echo something on the command line to verify something like that, before you paste it into a script and run a full test.

Also, changing the db_version in the 200 scripts as a one-off is not what your original code did: that code puts the current version (at the time of its execution) into the final run. Just hacking the 200 scripts with today's version number means a fresh update every time you update the db itself.

Personally, I would design to have the 200 scripts just source a common .config file for any shared variables. DRY -- dont repeat yourself. You don't need 200 scripts to be smart when a one-liner is more controllable and more obvious to maintain.

Hello,

As a follow-up here, I've just spotted a problem with one of your three substitute lines. Let's take a closer look at your second line, the one you intend to use to create your $DB_Ver variable:

DB_Ver=`grep -w DB_Version $INFA_CONFIG | cut -d = -f 2`

Now, I think what you're attempting to do here is search the contents of the $INFA_CONFIG variable for the string "DB_Version", and then extract part of the result. However, that wont't actually work.

The reason for this is that the grep command is used to search the contents of files, which are either passed to it as standard input via a pipe, or supplied as an argument on the command line. You cannot use it to search the contents of a shell variable.

Rather, to search the contents of a variable, you'd want to do something like this:

echo $INFA_CONFIG | grep DB_Version | cut -d= -f2

Here we use echo to print out the contents of the variable $INFA_CONFIG, and then pipe that into our grep and cut sequence. In this way, you'll be able to get the data you want.

Sorry no, I think $INFA_CONFIG is a file name.
So the intended substitution is correct.

The proposed solution discards the ' characters.
They might not be needed at all, or can be replaced with " characters. If needed, each literal ' can be escaped as '\''
An any-characters text is problematic in a sed script, and the embedding in a shell script creates the extra problem with the '

Hi,

Ah, good catch - could very well be, yes. Well, one of the two approaches should work, depending on whether it is indeed a file, or a variable. In any case, we'll see what the OP says in terms of how it works for them (or not, as the case may be), and go from there.

Thanks drysdalk. This solution worked great !! sed command replaced the values perfectly

The following shell script allows all characters in the replacement text:

#!/bin/sh
# A simple assignment:
oldstr='module load ibmdb2/client/10.1.3'
# A here-doc assignment can take any characters:
newstr=$( cat << "_EOT"
INFA_CONFIG="$HOME/config/research_dw_rt_$ENV_VAR.cfg"
DB_Ver=`grep -w 'DB_Version' "$INFA_CONFIG" | cut -d '=' -f 2`
module load ibmdb2/client/"$DB_Ver"
_EOT
)

for file in *.txt
do
# Make a backup
  cp -p "$file" "$file.bak" || continue
# From the backup rewrite the file and do the changes
  while IFS= read -r line
  do
    case $line in ( *"$oldstr"* )
      line=$newstr
    esac
    printf "%s\n" "$line"
  done < "$file.bak" > "$file"
done

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.