File Processing: Handling spaces in a line

Hi All,

Iam trying to get a file processed and some lines have spaces...the below is not working

Want to remove empty line
Want to remove lines that start with #
Avoid line with substring WHOA
When trying to get the substring from the var also Iam having trouble

file is like VAR=VALUE, an fiel to set environment to be exact

for var in $(cat -r $file | sed '/^$/d' | grep -v -E ""^\#"" )
do
      subvar=`echo $var | cut -c1-4`
      if [ "$subvar" != "WHOA" ]
      then
                eval export $var
      fi
done

Sample Input

MONITOR_OFF=YES
DIR=/usr/as/brs
DATADIR=/usr/df/brs
SHELLDIR1=/usr/as/brs
SHELLDIR2=/usr/as/brs
WHOA= %s : %S - %C [%c xxxx]    --- This is the line Iam trying to avoid by substring logic, this has spaces in between, need logic to get only WHOA also from this line

I dont know how to pass the file using while and IFS as Iam doing some stuff before I use it

Regards

for x in `cat something` is basically always wrong because of the problem you have discovered among others. You could be doing a while read line ; do ... done < inputfile loop.

If your input file is of the form VAR=VALUE, you can do this:

while IFS="=" read VAR VALUE
do
        [ -z "$VAR" ] && continue # Ignore blank lines
        case "$VAR" in
        WHOA*)  ;;

        *)
                read $VAR <<EOF
$VALUE
EOF
                export $VAR
                ;;
        esac
done < $file

Which means that, if someone puts `rm -Rf ~/` in your configuration file, it will store that strong instead of running that string like eval would!

1 Like

Thanks a lot you seem to be making sense

Let me get you what Iam doing and you can put the whole thing may be, if that is not asking too much

Iam doing below
Want to remove empty line
Want to remove lines that start with #
When trying to get the substring from the var also Iam having trouble. I dont want to read a line with such substrings.

Read the file and process exactly as you said VAR=VALUE line only, an environment file to be exact

for var in $(cat -r $file | sed '/^$/d' | grep -v -E ""^\#"" )
do
      subvar=`echo $var | cut -c1-4`
      if [ "$subvar" != "WHOA" ]
      then
                eval export $var
      fi
done

Sample Input added as per CarloM request

MONITOR_OFF=YES
DIR=/usr/as/brs
DATADIR=/usr/df/brs
SHELLDIR1=/usr/as/brs
SHELLDIR2=/usr/as/brs
WHOA= %s : %S - %C [%c xxxx]    --- This is the line Iam trying to avoid by substring logic

Output is to set them as environment variables....

It would be easier if you provided some sample input & expected output/behaviour.

1 Like

I noticed that you are using way too many external commands when all of those tasks can be achieved using shell builtins.

Here is an example:

#!/bin/bash

while read line
do
        # Want to remove empty line
        [[ -z "$line" ]] && continue
        # Want to remove lines that start with #
        [[ "$line" =~ ^# ]] && continue
        # trying to get the substring
        subvar="${line:0:4}"
done < file
1 Like

As a first query, could we get rid of the sed part? It more simply written as:- egrep -v "^$|^#" $file

More importantly, do you have a file that contains your environment variables you wish to set? Does it look like this:-

# My variables
# Some other comment

WHOA1=Hello
WHOA2=Bye
Other=stuff

If so, you can simply source in the file (or an extract of it) rather than worrying about the eval and the very real risks that creates.

In your script, you would simply have:-

. file

If you just need the selection of variables starting WHOA, you could easily:-

grep "^WHOA" $file > /tmp/myenv.$$
. /tmp/myenv.$$
rm /tmp/myenv.$$

Does this help, or have I missed the point?

I would like to know more about where Corona688 is leading us, especially the here document bit. I'm a little puzzled and it would be nice to understand the suggestion.

Robin
Liverpool/Blackburn
UK

1 Like
subvar="${line:0:4}" 

gives bad substitution error on command line

Well Iam trying to get a substring from VAR which is WHOA and avoid those lines

how to get only WHOA from line

WHOA= %s : %S - %C [%c xxxx]

Would this come close to what you want to achieve:

while IFS="=" read A B
  do case $A in
       "#"*) printf "comment" ;;
         "") printf "empty" ;;
          *) read $A < <(echo $B) ;;
     esac
  eval echo "\$A="\$$A
  done <file

? That eval echo is just for controlling purposes.

1 Like

Can we see a sample input file and which ones you want to use and which to exclude?

Please past in CODE tags to make it easier to read.

Robin

I have already posted sample input with which ones I want to see, please see my earlier post

I want to remove only empty lines and lines starting with #
Also line with the substring WHOA

where

WHOA= %s : %S - %C [%c xxxx] 

I want to avoid

Just include a line like WHOA) ;; above to eliminate that variable...

Yes, I have seen that you were editing your post as I was writing mine. Oh well :rolleyes:

So, if you want to exclude all blank lines, comments (actually starting with a #) and those starting "WHOA=", then you can have all this in one grep thus:-

egrep -v "^$|^#|^WHOA=" file > tmp/myenv.$$
. /tmp/myenv.$$
rm /tmp/myenv.$$

You have various suggestions now. Which one is closest to what you need? We can concentrate on that route.

Robin

1 Like

I think I want to join if possible the below solution

while IFS="=" read VAR VALUE
do
        [ -z "$VAR" ] && continue # Ignore blank lines
        case "$VAR" in
        WHOA*)  ;;

        *)
                read $VAR <<EOF
$VALUE
EOF
                export $VAR
                ;;
        esac
done < $file

with yours

egrep -v "^$|^#|^WHOA=" file > tmp/myenv.$$
. /tmp/myenv.$$   -- This I dod not need as I only use these variables in another shell script to set the environment 
rm /tmp/myenv.$$

@Robin:
the here document is a trick to feed variables to read in a safe manner. The here document is pre-processed by the shell (and usually stored in a temp file) then opened as a stream. While processing the here document a `command` is executed and a $var is substituted, but not the contents of $var .
BTW the processing can be restricted by putting the EOF marker in quotes: <<"EOF" or <<'EOF' , then the here document is treated as a quoted string.

I am not convinced that one cannot use the evil eval here.

while IFS="=" read VAR VALUE
do
  case "$VAR" in
    WHOA*)
    ;;
    [A-Z]*)
    eval $VAR='$VALUE'
    export $VAR
    ;;
  esac
done < $file