reading in a file through

SkySmart · May 15, 2012, 7:19pm

I'm trying to use the below script to perform a certain task.

task is: i have a file that has data that is similar to what is pasted below.

we'll call this file host.conf

define host {
      address 10.11.11.14
      alias mail-server
      host_name        mail-server01.sky.net
      use generic-host
}

What I just pasted above is what i refer to as a chunk. Now, there are several of these chunks in the host.conf file.

now, i have a separate file that contains a list of host names. we'll call this separate file host.list.

the list of hostnames in host.list is one server per line.

example:

mail-server01.sky.net
mail-server02.sky.net
mail-server03.sky.net

i want to take each host that is in host.list, and run it against the host.conf file. what i'm looking to do is to get the IP of the host from the host.conf file.

in other words, every host that I have in host.list has a configuration set up in host.conf. the configuration set up for each host is the chunk i pasted above.

below is the code im attempting to use to help me pull out what i want.

awk 'BEGIN {
  while((getline < "host.list")>0)
     S[$0]

  FS="\n"; RS="}\n"
}

/define host/ && /}/ {

  for(X in D) delete D[X];

  for(N=2; N<=NF; N++)
  {
       split($N, A, " ");
       D[A[1]] = A[2]
       i = 3;
       while (i in A) 
          D[A[1]] = D[A[1]] "=" A[i++];
  }

  if (D["host_name"] in S) 
       printf("%20s -- %50s\n", D["host_name"], D["address"])

}' $1

the output of this script should be:

hostname ---- ip
ex:
mail-server01.sky.net  ------  10.11.11.14

my os is: redhat and sunos
shell: bash

Finja · May 15, 2012, 8:06pm

I'm no guru, and I'm sure there's a one-liner way to do it with awk or something, but if pressed for time, here's how I would do it in a shell script:

 
#!/bin/bash
 
#first, put each chunk from the host.conf file on a single line of a temp file
#and make sure all delimiters are single spaces, and there are none leading
echo -ne `cat host.conf` | sed 's/}/}\n/g' | sed 's/^ *//;s/  */ /g' > host.conf.searchable
 
#then read each line from the host.list file one at a time
while read HOSTNAME
do
  #check the temp file to see if there is a matching entry in the host.conf file
  if [[ -n grep $HOSTNAME host.conf.searchable ]] ; then
    #if there is, get the IP field of the matching line
    IP=`grep $HOSTNAME host.conf.searchable |cut -d" " -f4`
    #and then just spit it out in the required format
    echo "$HOSTNAME ----- $IP"
  fi
done<host.list
 
#my mama always told me to clean up after myself
rm host.conf.searchable
 
exit

agama · May 15, 2012, 10:40pm

I think the awk solution can be a wee bit more simple:

   awk '
    NR == FNR { seen[$1]; next }   # collect hosts from the conf file
    /}/ {                        # end of chunk, if name was in conf list, print data
        if( name in seen )
            printf( "%20s ---- %s\n", name, ip );
        name = "";
    }
    /host_name/ { name = $2; next; }
    /address/ { ip = $2; next; }
' host.conf chunk-file >output-file

Doesn't assume that elements in the block have a specific order; this is an advantage if chunk file is hand coded.

complex.invoke · May 15, 2012, 11:05pm

grep -f host.list <(sed -e '/address\|host_name/!d' host.conf  | sed -n 'h;n;H;x;s/\n/ /;p') | awk '{print $NF,"------",$2}'

SkySmart · May 16, 2012, 1:35pm

agama:

I think the awk solution can be a wee bit more simple:
   awk '
   NR == FNR { seen[$1]; next }   # collect hosts from the conf file
   /}/ {                        # end of chunk, if name was in conf list, print data
   if( name in seen )
   printf( "%20s ---- %s\n", name, ip );
   name = "";
   }
   /host_name/ { name = $2; next; }
   /address/ { ip = $2; next; }
' host.conf chunk-file >output-file
Doesn't assume that elements in the block have a specific order; this is an advantage if chunk file is hand coded.

thank you. this is the code i need. it works.

one question, where do you specify the "define host"? how does the script know where to begin?

i ask because in each chunks, there can be two ips set for one host. the ips will be specified by:

address  10.11.11.45
secondary_address 10.19.19.19

looks like the script only grabs any field that matches "address", when in fact, i need it to grab only the primary address, which is the "10.11.11.45".

---------- Post updated at 01:35 PM ---------- Previous update was at 09:54 AM ----------

It seems the script is grabbing the wrong IPs.

i modified the script to, instead of grabbing the primary ip that is defined in a chunk like this:

 define host {
      address 10.11.11.14
      alias mail-server
      host_name        mail-server01.sky.net
      use generic-host
}

i modified the script to grab the secondary IP instead. but its not working. it's grabbing ips that dont belong to the chunk of the particular host it is saying it belongs to.

 define host {
      alias mail-server
      host_name        mail-server01.sky.net
      _secondary_address 10.11.11.14
      use generic-host
}

here's my script, i'm sure its wrong:

  awk '
    NR == FNR { seen[$1]; next }   # collect hosts from the conf file
    /}/ {                        # end of chunk, if name was in conf list, print data
        if( name in seen )
            printf( "%20s ---- %s\n", name, ip );
        name = "F";
    }
    /host_name/ { name = $2; next; }
    / _secondary_address/ { ip = $2; next; }

' host.list host.conf

need help please

agama · May 16, 2012, 6:10pm

Small change should ignore the secondary address:

awk '
    NR == FNR { seen[$1]; next }   # collect hosts from the conf file
    /}/ {                        # end of chunk, if name was in conf list, print data
        if( name in seen )
            printf( "%20s ---- %s\n", name, ip );
        name = "";
    }
    /secondary_address/ { next; }
    /host_name/ { name = $2; next; }
    /address/ { ip = $2; next; }
' host.conf chunk-file >output-file

And to answer your original question, the programme doesn't look for define host tags. Given what you posted as the contents of your file it doesn't need to. If there are other 'chunks' that need to be ignored, then it might not do the right thing depending on the contents of thoese chunks.

something like this would help if there are multiple chunks to be ignored:

awk '
    NR == FNR { seen[$1]; next }   # collect hosts from the conf file
    /define host.*{/ { snarf = 1; next; }    # ok to capture data after this point
    !snarf { next; }              # skip record if not capturing
    /}/ {                        # end of chunk, if name was in conf list, print data
        if( name in seen )
            printf( "%20s ---- %s\n", name, ip );
        name = "";
        snarf = 0;             # end of chunk turn capture off
    }
    /secondary_address/ { next; }     # ignore secondary address edit: this might need to be /_secondary_address/
    /host_name/ { name = $2; next; }  
    /address/ { ip = $2; next; }
' host.conf chunk-file >output-file

I didn't test this, so it could have a typo, but it looks clean to me.

---------- Post updated at 18:10 ---------- Previous update was at 18:02 ----------

I just reread your last posts and am a bit confused. Is the secondary address marked with secondary_address or does the string have a leading underbar? Given that your programme is looking to capture with the leading underbar, and it's not working, I'm assuming it doesn't have it in the file. The problem might also have been leading whitespace. If the configuration file records don't have any whitespace, or it's tabs and not at least a space, the leading space in your pattern ( / _secondary_address/ could be your problem too. It's hard to tell since you only posted bits and pieces of your config file.

I hope this makes some sense, and that some of it gets you going again.

SkySmart · May 16, 2012, 6:43pm

agama:

Small change should ignore the secondary address:
awk '
   NR == FNR { seen[$1]; next }   # collect hosts from the conf file
   /}/ {                        # end of chunk, if name was in conf list, print data
   if( name in seen )
   printf( "%20s ---- %s\n", name, ip );
   name = "";
   }
   /secondary_address/ { next; }
   /host_name/ { name = $2; next; }
   /address/ { ip = $2; next; }
' host.conf chunk-file >output-file
And to answer your original question, the programme doesn't look for define host tags. Given what you posted as the contents of your file it doesn't need to. If there are other 'chunks' that need to be ignored, then it might not do the right thing depending on the contents of thoese chunks.

something like this would help if there are multiple chunks to be ignored:
awk '
   NR == FNR { seen[$1]; next }   # collect hosts from the conf file
   /define host.*{/ { snarf = 1; next; }    # ok to capture data after this point
   !snarf { next; }              # skip record if not capturing
   /}/ {                        # end of chunk, if name was in conf list, print data
   if( name in seen )
   printf( "%20s ---- %s\n", name, ip );
   name = "";
   snarf = 0;             # end of chunk turn capture off
   }
   /secondary_address/ { next; }     # ignore secondary address edit: this might need to be /_secondary_address/
   /host_name/ { name = $2; next; }  
   /address/ { ip = $2; next; }
' host.conf chunk-file >output-file
I didn't test this, so it could have a typo, but it looks clean to me.

---------- Post updated at 18:10 ---------- Previous update was at 18:02 ----------

I just reread your last posts and am a bit confused. Is the secondary address marked with secondary_address or does the string have a leading underbar? Given that your programme is looking to capture with the leading underbar, and it's not working, I'm assuming it doesn't have it in the file. The problem might also have been leading whitespace. If the configuration file records don't have any whitespace, or it's tabs and not at least a space, the leading space in your pattern ( / _secondary_address/ could be your problem too. It's hard to tell since you only posted bits and pieces of your config file.

I hope this makes some sense, and that some of it gets you going again.

thank you so much.

this works:

awk '
    NR == FNR { seen[$1]; next }   # collect hosts from the conf file
    /define host.*{/ { snarf = 1; next; }    # ok to capture data after this point
    !snarf { next; }              # skip record if not capturing
    /}/ {                        # end of chunk, if name was in conf list, print data
        if( name in seen )
            printf( "%20s ---- %s\n", name, ip );
        name = "";
        snarf = 0;             # end of chunk turn capture off
    }
    /_secondary_address/ { next; }     # ignore secondary address edit: this might need to be /_secondary_address/
    /host_name/ { name = $2; next; }
    /address/ { ip = $2; next; }

i just need one last thing. i need to modify this code to not spit out both the ips for address and _secondary address. so i dont want to ignore secondary address anymore. i want this code to spit out something like this:

sky.net ---- 10.11.11.11 ---- 10.12.12.12

in this case of course, the first ip will be the ip provided for "address", and the second ip will be the one provided for "_secondary_address".

thanks again!

bakunin · May 16, 2012, 6:45pm

A simple change to the regex would also do the trick (replace "<spc>" and "<tab>" with literal tabs and spaces):

awk '
    NR == FNR { seen[$1]; next }   # collect hosts from the conf file
    /}/ {                        # end of chunk, if name was in conf list, print data
        if( name in seen )
            printf( "%20s ---- %s\n", name, ip );
        name = "";
    }
    /^[<spc><tab>]*host_name[<spc><tab>]/ { name = $2; next; }
    /^[<spc><tab>]*address[<spc><tab>]/ { ip = $2; next; }
' host.conf chunk-file >output-file

How the script works: first the list of host names to search for is read into the array "seen". Then the file with the chunks is scanned.

The script reacts to three different type of lines in the chunk file:

1st type: line starts with "}". This is a line ending a chunk and if the variable "name" contains a hostname corresponding to one in the seen-array then the info (variables "name" and "ip") gathered before is printed out and "name" is cleared again.

2nd type: line starts with some optional spaces/tabs followed by the word "host_name" followed by a space or tab. The second part of this line is stored in the variable "name" for use when the next type-1-line is found. Then the rest of the script is skipped for this line and the next line of the chunk file is read.

3rd type: line starts with some optional spaces/tabs followed by the word "address" followed by a space or tab. The procedure is the same as with type-2-lines, but this time the second part of the line is stored in the variable "ip".

Notice that by surrounding the keywords with white space the ambiguity between "address" and "secondary_address" is automatically resolved.

I hope this helps.

bakunin

agama · May 16, 2012, 7:13pm

Capturing both IP addresses:

awk '
    NR == FNR { seen[$1]; next }   # collect hosts from the conf file
    /define host.*{/ { snarf = 1; next; }    # ok to capture data after this point
    !snarf { next; }              # skip record if not capturing
    /}/ {                        # end of chunk, if name was in conf list, print data
        if( name in seen )
            printf( "%20s ---- %s ---- %s\n", name, ip, ip2 );   # now prints both
        name = "";
        snarf = 0;             # end of chunk turn capture off
    }
    /_secondary_address/ { ip2 = $2; next}       # captures secondary too
    /host_name/ { name = $2; next; }
    /address/ { ip = $2; next; }

SkySmart · May 16, 2012, 7:30pm

agama:

Capturing both IP addresses:

awk '
   NR == FNR { seen[$1]; next }   # collect hosts from the conf file
   /define host.*{/ { snarf = 1; next; }    # ok to capture data after this point
   !snarf { next; }              # skip record if not capturing
   /}/ {                        # end of chunk, if name was in conf list, print data
   if( name in seen )
   printf( "%20s ---- %s ---- %s\n", name, ip, ip2 );   # now prints both
   name = "";
   snarf = 0;             # end of chunk turn capture off
   }
   /_secondary_address/ { ip2 = $2; next}       # captures secondary too
   /host_name/ { name = $2; next; }
   /address/ { ip = $2; next; }

unfortunately, this grabs IPs that belong to other server's chunks. there are some chunks for some servers that do not have a "_secondary_address" ip assigned. they just have ip for "address" assigned. for those, i expected the 3rd field to be empty since theres nothing to grab.

here's a script i was using but for some reason, this script doesn't read the entire host list i gave it, even though all the conditions are met. for instance, say there are 100 hosts in a list. it will for some reason not scan through all 100. it omits servers. i dont know which part of the script is making it do so.

awk 'BEGIN {
  while((getline < "host.list")>0)
     S[$0]

  FS="\n"; RS="}\n"; ORS="}\n";
}

/define host/ {

  for(X in D) delete D[X];

  for(N=2; N<=NF; N++)
  {
       split($N, A, " ");
       D[A[1]]=A[2];
  }

  if (D["host_name"] in S)
       printf("%s -------- %s -------- %s\n", D["host_name"]" " " ", D["address"], D["_secondary_address"])

}' host.conf

hosts.conf is the file that contains the "chunks".

agama · May 16, 2012, 7:41pm

Silly assumption on my part. Setting both address variables to empty strings after printing should be all that is needed:

awk '
    NR == FNR { seen[$1]; next }   # collect hosts from the conf file
    /define host.*{/ { snarf = 1; next; }    # ok to capture data after this point
    !snarf { next; }              # skip record if not capturing
    /}/ {                        # end of chunk, if name was in conf list, print data
        if( name in seen )
            printf( "%20s ---- %s ---- %s\n", name, ip, ip2 );   # now prints both
        name = = ip, ip2 = ""    # reset everything; 
        snarf = 0;             # end of chunk turn capture off
    }
    /_secondary_address/ { ip2 = $2; next}       # captures secondary too
    /host_name/ { name = $2; next; }
    /address/ { ip = $2; next; }
'  host.list host.conf

SkySmart · May 16, 2012, 7:47pm

agama:

Silly assumption on my part. Setting both address variables to empty strings after printing should be all that is needed:

awk '
   NR == FNR { seen[$1]; next }   # collect hosts from the conf file
   /define host.*{/ { snarf = 1; next; }    # ok to capture data after this point
   !snarf { next; }              # skip record if not capturing
   /}/ {                        # end of chunk, if name was in conf list, print data
   if( name in seen )
   printf( "%20s ---- %s ---- %s\n", name, ip, ip2 );   # now prints both
   name = = ip, ip2 = ""    # reset everything; 
   snarf = 0;             # end of chunk turn capture off
   }
   /_secondary_address/ { ip2 = $2; next}       # captures secondary too
   /host_name/ { name = $2; next; }
   /address/ { ip = $2; next; }
'  host.list host.conf

I ran this code, i got an error:

awk: cmd. line:8:         name = = ip, ip2 = ""    # reset everything; 
awk: cmd. line:8:                ^ syntax error

agama · May 16, 2012, 7:53pm

skysmart:

here's a script i was using but for some reason, this script doesn't read the entire host list i gave it, even though all the conditions are met. for instance, say there are 100 hosts in a list. it will for some reason not scan through all 100. it omits servers. i dont know which part of the script is making it do so.
awk 'BEGIN {
  while((getline < "host.list")>0)
   S[$0]

  FS="\n"; RS="}\n"; ORS="}\n";
}

/define host/ {

  for(X in D) delete D[X];

  for(N=2; N<=NF; N++)
  {
   split($N, A, " ");
   D[A[1]]=A[2];
  }

  if (D["host_name"] in S)
   printf("%s -------- %s -------- %s\n", D["host_name"]" " " ", D["address"], D["_secondary_address"])

}' host.conf
hosts.conf is the file that contains the "chunks".

I cut/pasted your code and for my small set of data it does just fine. I gave it 100 dummy names in addition to the ones you had in your example. Can you post a list of what seems not being captured from your list?

SkySmart · May 16, 2012, 7:57pm

i guess it would help to point out that the chunks are all entered into the file by hand, and sometimes by an automated script.

so for some of the chunks, there are tabs in there, or spaces, or both.

didn't think that would matter.

i wish i can post more information about the files in question but i cant due to security reasons.

however, i would prefer to use your latest updated script instead. the one i said i got an error on. i think what may be missing in that script may just be something very simple. can you please look it over?

agama · May 16, 2012, 8:06pm

We crossed posts and I didn't see the error. In the statement name = = ip, ip2 = "" the double = = should just be a single, and one added between ip and ip2; looks like I cut and pasted a comma in there too.-- happy fingers. Sorry about that!

name = ip = ip2 = "";