awk out an unknown column ?

i9300 · August 2, 2008, 10:18pm

I need a solution to awk out an unknown column. I am unable to say '{print $x}' because the location changes. I would like to find a perl or awk solution to this. I do not know either very well but am trying to delve deeper into both.
I am looking for the version of pkg8 in this example. Please keep in mind the location of pkg8 will change depending...

Here is the line I am looking at :

# grep pkg8 /tmp/trash
XXX Packages             : XXXpkg1(6.2.3.1) XXXpkg2(1.0.0.3) XXXpkg3(4.0.0.5) XXXpkg4(2.0.0.1) XXXpkg5(A0)  XXXpkg6(2.0.1.1)  XXXpkg8(5.7.0.10) XXXpkg9(1.5) XXXpkg10(2.1.2.3)

I was able to find a UGLY solution which includes a for loop and if statement. But this is very slow. I am looking at this file on 200 + systems, so the execution does matter.

Here is the entire script for this :


#!/bin/ksh
HOSTNAME=`uname -n`
OS_VER=`uname -r`

PKG=XXXpkg8
CORRECT_VER=5.7.0.10
COLLECT=/tmp/trash
for x in `grep $PKG $COLLECT|awk -F: '{print $2}'`;do
    if [ `echo $x|grep "XXXpkg8("` ];then
        PKG_VER=`echo $x|grep "XXXpkg8("`
    else
        continue
    fi
done
VER=`echo $PKG_VER|awk -F"(" '{print $2}'|sed -e 's/)//'`



if [[ $VER = $CORRECT_VER ]];then
        if [[ $STATUS = NO ]];then
            continue
        else
            STATUS=YES
        fi
else
    STATUS=NO
    MESSAGE="$MESSAGE XXXpkg8:$VER"
fi

echo "$STATUS $MESSAGE"

I would like to replace the entire for loop with a perl/awk one liner. I have searched for days and frankly my head hurts already. I have seen grep and for loop used inside of awk but I really don't understand what it is doing.
problems in grep inside awk - Computing.Net
[url=http://www.linuxquestions.org/questions/programming-9/can-i-use-grep-inside-awk-174287

If anyone can offer some help I would appreciate it.

ghostdog74 · August 2, 2008, 10:59pm

awk '{
  for( ......) #go through all fields
  {
   if( field contains what i want ){
     print field
   }
  }

}' file

aigles · August 3, 2008, 2:12pm

Replace the following lines :

for x in `grep $PKG $COLLECT|awk -F: '{print $2}'`;do
    if [ `echo $x|grep "XXXpkg8("` ];then
        PKG_VER=`echo $x|grep "XXXpkg8("`
    else
        continue
    fi
done
VER=`echo $PKG_VER|awk -F"(" '{print $2}'|sed -e 's/)//'`

by :

VER=$( sed -n 's/.*'"$PKG"'(\([^)]*\)).*/\1/p'  $COLLECT )

Jean-Pierre.

i9300 · August 4, 2008, 12:28am

ghostdog74,

While I appreciate your post, I am not an expert with awk. I was unable to get your example to work. This is what I tried.

awk '{
  for( `grep $PKG $COLLECT|awk -F: '{print $2}'` ) #go through all fields
  {
   if( XXXpkg8 ){
     print field
   }
  }

}' $COLLECT

I am not sure how this will populate the $VER which is what I am after.

Aigles,

Thanks for the example. I was able to drop the one line in and it worked perfectly. Since I am no expert with sed/awk/perl, I really appreciate you providing an example that I could just drop in. However it turned out to be very slow. Since I will be running this within another for loop (about 200 times), I need it to run faster.

I ran your sed in ./check_pkg and my code was in check_pkg2.

root@smsdr2# time ./check_pkg
YES

real    0m1.737s
user    0m1.500s
sys     0m0.000s
root@smsdr2# time ./check_pkg2
YES

real    0m0.365s
user    0m0.040s
sys     0m0.060s

root@smsdr2# diff check_pkg check_pkg2
8,16c8,15
< #for x in `grep $PKG $COLLECT|awk -F: '{print $2}'`;do
< #     if [ `echo $x|grep "XXXpkg8("` ];then
< #             PKG_VER=`echo $x|grep "XXXpkg8("`
< #     else
< #             continue
< #     fi
< #done
< #VER=`echo $PKG_VER|awk -F"(" '{print $2}'|sed -e 's/)//'`
< VER=$( sed -n 's/.*'"$PKG"'(\([^)]*\)).*/\1/p'  $COLLECT )
---
> for x in `grep $PKG $COLLECT|awk -F: '{print $2}'`;do
>       if [ `echo $x|grep "XXXpkg8("` ];then
>               PKG_VER=`echo $x|grep "XXXpkg8("`
>       else
>               continue
>       fi
> done
> VER=`echo $PKG_VER|awk -F"(" '{print $2}'|sed -e 's/)//'`
17a17,18
>
>

Any other ideas?

era · August 4, 2008, 12:46am

Why do you need to run it in another loop? Perhaps you could show us the whole code so we can see what to optimize. (I'm thinking maybe you could run the whole loop in sed.)

aigles · August 4, 2008, 4:27am

i9300:

ghostdog74,

While I appreciate your post, I am not an expert with awk. I was unable to get your example to work. This is what I tried.
awk '{
  for( `grep $PKG $COLLECT|awk -F: '{print $2}'` ) #go through all fields
  {
   if( XXXpkg8 ){
   print field
   }
  }

}' $COLLECT
I am not sure how this will populate the $VER which is what I am after.

Aigles,

Thanks for the example. I was able to drop the one line in and it worked perfectly. Since I am no expert with sed/awk/perl, I really appreciate you providing an example that I could just drop in. However it turned out to be very slow. Since I will be running this within another for loop (about 200 times), I need it to run faster.

I ran your sed in ./check_pkg and my code was in check_pkg2.
root@smsdr2# time ./check_pkg
YES

real    0m1.737s
user    0m1.500s
sys     0m0.000s
root@smsdr2# time ./check_pkg2
YES

real    0m0.365s
user    0m0.040s
sys     0m0.060s
root@smsdr2# diff check_pkg check_pkg2
8,16c8,15
< #for x in `grep $PKG $COLLECT|awk -F: '{print $2}'`;do
< #     if [ `echo $x|grep "XXXpkg8("` ];then
< #             PKG_VER=`echo $x|grep "XXXpkg8("`
< #     else
< #             continue
< #     fi
< #done
< #VER=`echo $PKG_VER|awk -F"(" '{print $2}'|sed -e 's/)//'`
< VER=$( sed -n 's/.*'"$PKG"'($[^)]*$).*/\1/p'  $COLLECT )
---
> for x in `grep $PKG $COLLECT|awk -F: '{print $2}'`;do
>       if [ `echo $x|grep "XXXpkg8("` ];then
>               PKG_VER=`echo $x|grep "XXXpkg8("`
>       else
>               continue
>       fi
> done
> VER=`echo $PKG_VER|awk -F"(" '{print $2}'|sed -e 's/)//'`
17a17,18
>
>
Any other ideas?

A little modification to the sed command will increase performances :

VER=$( sed -n "/$PKG/"'s/.*'"$PKG"'(\([^)]*\)).*/\1/p'  $COLLECT )

I have tested the three solutions with a file containing 6900 lines (package versions on last line) on my PC (cygwin) and on my AIX box.

The initial version run faster under AIX only (i am very surprise by the difference between Cygwin and AIX results).
The second sed version is the fastest in the two case.

Cygwin times :

---- Initial version ---
YES  
real    0m3.140s
user    0m1.047s
sys     0m0.719s
---- sed version -------
YES  
real    0m0.406s
user    0m0.215s
sys     0m0.137s
---- sed version 2 -----
YES  

real    0m0.360s
user    0m0.231s
sys     0m0.090s

AIX times :

---- Initial version ---
YES  
r�el    0m0,14s
util    0m0,00s
sys     0m0,09s
---- sed version -------
YES  
r�el    0m1,55s
util    0m1,52s
sys     0m0,03s
---- sed version 2 -----
YES  

r�el    0m0,03s
util    0m0,02s
sys     0m0,01s

Jean-Pierre.

ghostdog74 · August 4, 2008, 5:42am

@OP
to go through the fields in awk

awk '
{
 for(i=1;i<=NF;i++ ){
   if ( $i == "what i want" ) {
      # do something.
   } 
 } 

}

' file

i9300 · August 5, 2008, 9:44am

aigles,

Works perfectly!! Very quick! This is exactly what I needed -- Thanks!

Now for the really difficult question how does it work?
About all that I can figure out from the man page is -n will "Suppresses the default output". I know the standard sed functions but I can't look at this and tell what it is doing.

aigles · August 5, 2008, 1:36pm

$PKG=XXXpkg8
VER=$( sed -n "/XXXpkg8/s/.*XXXpkg8\([^)]*).*/\1/p' )

sed -n                      Suppress automatic print, output must be explicit (p command)
/XXXpkg8/                   Select lines containing XXXpkg8
s/.*XXXpkg8\([^)]*).\).*/   Substitute all the line, memorize version XXXpkg8(version)
/\1/                        with memorized version
p                           Print the result if a substitution have been made

Jean-Pierre.

i9300 · August 5, 2008, 4:45pm

Thanks JP!

sudhamacs · August 5, 2008, 5:13pm

grep "XXXpkg8(5.7.0.10)" /tmp/trash
echo $?
if its = 0 then u found the pattern and the file.

Is this is what u r looking for?

aigles · August 6, 2008, 3:28am

No! If file exist and if pattern is found, the exit status is 0.

Jean-Pierre.

i9300 · September 16, 2008, 10:29am

aigles:

$PKG=XXXpkg8
VER=$( sed -n "/XXXpkg8/s/.*XXXpkg8\([^)]*).*/\1/p' )

sed -n                      Suppress automatic print, output must be explicit (p command)
/XXXpkg8/                   Select lines containing XXXpkg8
s/.*XXXpkg8\([^)]*).\).*/   Substitute all the line, memorize version XXXpkg8(version)
/\1/                        with memorized version
p                           Print the result if a substitution have been made

Jean-Pierre.

Aigles,

Thanks for providing the breakdown but I still can't translate this to not look for the version in XXXpkg8(version)....
IE how would I do this same task if all I wanted in the output was XXpkg8(version) ....

Thanks!

era · September 17, 2008, 1:26am

The parentheses $...$ select what is being memorized as \1. The parentheses don't affect what is being matched (in this particular case) so you can move them around in the expression. If you have multiple parenthesized expressions, they will be remembered in \1 \2 etc according to the order of the opening parentheses.

The "select" part of the script is actually redundant, since the substitution obviously can only take place on lines where the left-hand side of the substitution expression matches.

aigles · September 17, 2008, 8:07am

The "select" part had been introduced for performance purposes.
See See Post #6

Jean-Pierre.

era · September 17, 2008, 8:13am

Ah, sorry, missed that. (I hate how this site breaks up threads over many pages.) Good stuff.

Franklin52 · September 17, 2008, 9:57am

You can change the number of post showed per thread, go to User CP -> Edit Options -> Thread Display Options -> Number of Posts to Show Per Page.

Regards

shamrock · September 17, 2008, 1:13pm

TIMTOWTDI...

awk -F'pkg8[(]' '/pkg8/{split($2,a,")");print a[1]}' file