Perl script help

bombcan · January 23, 2013, 8:58am

Hi

I'm pretty new to PERL so please bear with me:D

I have a df output which I need to add the used of each filesystem

Example:

anmtl004	/dev/hd4	320	37%	4253	8%	/	rootvg
anmtl004	/dev/fslv01	2048	10%	71	1%	/IBMperfpmr	rootvg
anmtl004	/dev/hd11admin	128	1%	5	1%	/admin	rootvg
anmtl004	/dev/fslv00	256	43%	371	2%	/appl/cyb_agent	rootvg
anmtl004	mtlhqnasa:/vol/nonprod01/trd/rec	943718	77%	18817636	60%	/appl/rec	
anmtl004	mtlhqnasa:/vol/nonprod01/trd/bwarchive	943718	77%	18817636	60%	/archive	
anmtl004	/dev/lvaudit	512	5%	368	1%	/audit	rootvg
anmtl004	mtlhqnasa:/vol/nonprod01/cn_dev	943718	77%	18817636	60%	/cn_dev	
anmtl004	/dev/lvSAPdb2	1024	1%	15	1%	/db2	vgSAP
anmtl004	mtlhqnasa.cn.ca:/vol/fixes	209141	83%	51782	1%	/fixes	
anmtl004	/dev/hd1	512	5%	1403	2%	/home	rootvg
anmtl004	/dev/lvlogs	256	12%	194	1%	/logs	rootvg
anmtl004	/dev/hd10opt	256	66%	3201	14%	/opt	rootvg
anmtl004	/proc	-	-	-	-	/proc	
anmtl004	mtlhqnasa:/vol/nonprod00/sapmnt_bd3	943718	74%	4928053	16%	/sapmnt/BD3	
anmtl004	mtlhqnasa:/vol/nonprod00/trd/sapmnt_bx2	943718	74%	4928053	16%	/sapmnt/BX2	
anmtl004	mtlhqnasa:/vol/nonprod00/trd/sapmnt_ed1	943718	74%	4928053	16%	/sapmnt/ED1	
anmtl004	mtlhqnasa:/vol/nonprod00/trd/sapmnt_ex1	943718	74%	4928053	16%	/sapmnt/EX1	
anmtl004	mtlhqnasa:/vol/nonprod00/trd/sapmnt_wx1	943718	74%	4928053	16%	/sapmnt/WX1	
anmtl004	/dev/hd3	2048	1%	227	1%	/tmp	rootvg
anmtl004	/dev/hd2	4416	80%	47078	16%	/usr	rootvg
anmtl004	/dev/lvSAPusr	5120	4%	215	1%	/usr/sap	vgSAP
anmtl004	/dev/lvSAPDBD3usr	128	1%	21	1%	/usr/sap/BD3	vgSAPD
anmtl004	/dev/lvSAPDBD3ascs	1024	13%	109	1%	/usr/sap/BD3/ASCS08	vgSAPD
anmtl004	/dev/lvSAPDBD3aers	1024	24%	125	1%	/usr/sap/BD3/ERS09	vgSAPD
anmtl004	/dev/lvSAPDBD3wd	3072	7%	557	1%	/usr/sap/BD3/W07	vgSAPD
anmtl004	/dev/lvSAPDBX2usr	128	1%	22	1%	/usr/sap/BX2	vgSAPX
anmtl004	/dev/lvSAPDBX2ascs	1024	7%	67	1%	/usr/sap/BX2/ASCS13	vgSAPX
anmtl004	/dev/lvSAPDBX2aers	1024	21%	91	1%	/usr/sap/BX2/ERS14	vgSAPX
anmtl004	/dev/lvSAPDBX2wd	3072	10%	378	1%	/usr/sap/BX2/W12	vgSAPX
anmtl004	/dev/lvSAPDAD	5120	16%	3480	1%	/usr/sap/DAD	vgSAP
anmtl004	/dev/lvSAPXED1usr	128	1%	19	1%	/usr/sap/ED1	vgSAPD
anmtl004	/dev/lvSAPXED1ers	1024	19%	79	1%	/usr/sap/ED1/ERS04	vgSAPD
anmtl004	/dev/lvSAPXED1scs	1024	44%	178	1%	/usr/sap/ED1/SCS03	vgSAPD
anmtl004	/dev/lvSAPXED1wd	1024	32%	392	1%	/usr/sap/ED1/W02	vgSAPD
anmtl004	/dev/lvSAPXEX1usr	128	1%	19	1%	/usr/sap/EX1	vgSAPX
anmtl004	/dev/lvSAPXEX1ers	1024	14%	76	1%	/usr/sap/EX1/ERS19	vgSAPX
anmtl004	/dev/lvSAPXEX1scs	1024	35%	177	1%	/usr/sap/EX1/SCS18	vgSAPX
anmtl004	/dev/lvSAPXEX1wd	1024	32%	406	1%	/usr/sap/EX1/W17	vgSAPX
anmtl004	/dev/lvSAPXWX1usr	128	1%	19	1%	/usr/sap/WX1	vgSAPX
anmtl004	/dev/lvSAPXWX1ers	1024	20%	42	1%	/usr/sap/WX1/ERS81	vgSAPX
anmtl004	/dev/lvSAPXWX1scs	1024	61%	194	1%	/usr/sap/WX1/SCS80	vgSAPX
anmtl004	/dev/lvSAPXWX1wd	1024	31%	380	1%	/usr/sap/WX1/W79	vgSAPX
anmtl004	mtlhqnasa:/vol/nonprod02/nasd_hostagent_depot	943718	18%	643102	3%	/usr/sap/hostagent_depot	
anmtl004	mtlhqnasa:/vol/prod00/saptrans_bw	322560	86%	2384871	9%	/usr/sap/transBW	
anmtl004	mtlhqnasa:/vol/prod00/saptrans_bwj	322560	86%	2384871	9%	/usr/sap/transBWJ	
anmtl004	/dev/hd9var	896	49%	1727	2%	/var	rootvg
anmtl004	/dev/livedump	256	1%	4	1%	/var/adm/ras/livedump	rootvg

This is what I have

#Build a lookup table
open (F,"report_df.csv") || die ("File could not be open");     #Read file in
while ( <F> ) {                                                 #for every line do

        my @line = split (",");                                 #Put each line in an array and split it at every ","
        my $host = $line[0];                                    #Every field needed is assigned a variable
        my $filesystem = $line[1];
        my $alloc_mb = $line[2];
        my $pct_used = $line[3];
        my $vg = $line[7];

$fs{$host}{$vg}{$filesystem}{"ALLOC_MB"} = $alloc_mb;           #Build the hash
$fs{$host}{$vg}{$filesystem}{"PCT_USED"} = $pct_used;
}
close F;
my @list_of_hosts = (sort keys %fs);                                    #Build an array of every host

foreach my $host (@list_of_hosts) {
        my @list_of_vg = (keys %{$fs{$host}});

        foreach my $vg (@list_of_vg) {
        my @list_of_fs = (keys %{$fs{$host}{$vg}});

                foreach my $filesystem (@list_of_fs) {
                my $alloc_mb = $fs{$host}{$vg}{$filesystem}{"ALLOC_MB"};
                my $pct_used = $fs{$host}{$vg}{$filesystem}{"PCT_USED"};
                my $free_space = (100 - $pct_used) * $alloc_mb;


                               print "$host, $filesystem, $vg\n";
}
}
}

So I need to add them now..

Thanks

Corona688 · January 23, 2013, 9:12am

Can you show the output you want?

bombcan · January 23, 2013, 9:36am

Hi

Sure

anmtl004	rootvg {TOTALSUM}

---------- Post updated at 09:36 AM ---------- Previous update was at 09:36 AM ----------

and so on for all VGs

Corona688 · January 23, 2013, 10:13am

I meant representative output. That output's not even enough for me to properly guess what you want a total of. Neither is a broken program that doesn't do what you want...

So, a wild guess:

# Total in MB would be sum third column times fourth column divided by 100.
awk '{ sub(/%/, "", $4); A[$1] += ($3 * $4)/100; } END { for(X in A) printf("%s, %d MB USED\n", X, A[X]) }' report_df.csv

which would print

anmtl004, 6579862 MB USED

bombcan · January 23, 2013, 10:25am

Thanks for the code however I need it in perl.

And the output would have to add the MB used of all the filesystems in the particular VG (ex:rootvg) and then have a report reporting on

HOST VG TOTAL

I got as far as getting the total used of each filesystems
with this

my $free_space = (100 - $pct_used) * $alloc_mb;

Thanks

Corona688 · January 23, 2013, 10:54am

Even though it did in a simple one-line program what yours couldn't do in over 30? all right...

Think I see, now. Here is the awk version:

$ awk '{ sub(/%/, "", $4); A[$7] += ($3 * (100-$4))/100; } END { for(X in A) printf("%s, %d MB USED\n", X, A[X]) }' datafile

/usr/sap/transBWJ, 45158 MB USED
/usr/sap/ED1, 126 MB USED
/logs, 225 MB USED
/fixes, 35553 MB USED
/usr/sap/BX2/ASCS13, 952 MB USED
/usr/sap/EX1/ERS19, 880 MB USED
/usr/sap/hostagent_depot, 773848 MB USED
/var/adm/ras/livedump, 253 MB USED
/usr/sap/BD3/ASCS08, 890 MB USED
/tmp, 2027 MB USED
/usr/sap/EX1/W17, 696 MB USED
/usr/sap/WX1/ERS81, 819 MB USED
/proc, 0 MB USED
/var, 456 MB USED
/usr/sap/ED1/ERS04, 829 MB USED
/db2, 1013 MB USED
/sapmnt/BD3, 245366 MB USED
/archive, 217055 MB USED
/usr/sap/BD3/ERS09, 778 MB USED
/usr/sap/DAD, 4300 MB USED
/sapmnt/WX1, 245366 MB USED
/usr/sap/BD3, 126 MB USED
/, 201 MB USED
/sapmnt/EX1, 245366 MB USED
/cn_dev, 217055 MB USED
/IBMperfpmr, 1843 MB USED
/appl/rec, 217055 MB USED
/usr/sap/WX1, 126 MB USED
/usr, 883 MB USED
/usr/sap/EX1, 126 MB USED
/audit, 486 MB USED
/usr/sap, 4915 MB USED
/home, 486 MB USED
/usr/sap/BX2/ERS14, 808 MB USED
/opt, 87 MB USED
/admin, 126 MB USED
/usr/sap/EX1/SCS18, 665 MB USED
/usr/sap/WX1/W79, 706 MB USED
/appl/cyb_agent, 145 MB USED
/usr/sap/transBW, 45158 MB USED
/usr/sap/WX1/SCS80, 399 MB USED
/sapmnt/BX2, 245366 MB USED
/usr/sap/ED1/SCS03, 573 MB USED
/usr/sap/BX2/W12, 2764 MB USED
/usr/sap/BD3/W07, 2856 MB USED
/usr/sap/BX2, 126 MB USED
/sapmnt/ED1, 245366 MB USED
/usr/sap/ED1/W02, 696 MB USED

$

I'll get to work on converting that into perl...

bombcan · January 23, 2013, 11:07am

Ok thanks but it still does not add the filesystems in the VG. This is doing each filesystem.
The ones that don't have a VG at the end you can skip.

So just to resume

anmtl004	/dev/lvSAPusr	5120	4%	215	1%	/usr/sap	vgSAP

This would look like
anmtl004 vgSAP 4.79GB

Thanks again for your efforts and time!

Corona688 · January 23, 2013, 11:19am

[edit] I see new requirements. Starting over again.

What about things without a VG?

Corona688 · January 23, 2013, 11:55am

$ awk '{ sub(/%/,""); A[$1,$8]+=($3*(100-$4))/100; } END { for(X in A) printf("%s %.1f\n", X, A[X]/1000) }' SUBSEP=" " filename

anmtl004  2777.7
anmtl004 vgSAPD 6.9
anmtl004 vgSAP 10.2
anmtl004 rootvg 7.2
anmtl004 vgSAPX 9.1

$  perl -ane '$t{$F[0]." ".$F[7]} += $F[2]*((100-substr($F[3], 0, -1))/100); END { foreach $k ( keys %t ) { printf("%s %.1f\n", $k, $t{$k}/1000); } }' filename

anmtl004 vgSAP 10.2
anmtl004 vgSAPX 9.1
anmtl004 rootvg 7.2
anmtl004 vgSAPD 6.9
anmtl004  2777.7

$

bombcan · January 24, 2013, 8:18am

This is good but I think you just added the allocated space. Example for the vgSAP you just added the 5120*2. It's actually 49% of that so 5GB.

Corona688 · January 24, 2013, 10:36am

I don't think so:

$F[2]*((100-substr($F[3], 0, -1))/100)

I add the third column ($F[2]) times 100-the fourth column($F[3]). substr removes the % from $F[3] so I can do math on it.

So for

anmtl004	/dev/lvSAPusr	5120	4%	215	1%	/usr/sap	vgSAP

it does (5120 * (100-4))/100 = 4195.2 megabytes.

BUT:

You asked me to group them by name, and never provided a complete example of output, so if there's more than one vgSAP in the same host they all get added together.

If that's not actually what you want:

Please show complete output for the input you gave so we can stop playing 20 questions

bombcan · January 24, 2013, 11:17am

BTW I really appreciate your time on this..Thanks

You have it right however 5120 is what is assigned then 4% is what is used.
So you have to calculate 4% of 5120 . As for the rest all is good. You are correct when you say "You asked me to group them by name, and never provided a complete example of output, so if there's more than one vgSAP in the same host they all get added together."

Corona688 · January 24, 2013, 11:32am

Ahhh. I was looking from your original program where you used 100-%, instead of just the percent. You want the opposite.

$ awk '{ sub(/%/,""); A[$1,$8]+=($3*$4)/100; } END { for(X in A) printf("%s %.1f\n", X, A[X]/1000) }' SUBSEP=" " filename

anmtl004  6570.0
anmtl004 vgSAPD 1.6
anmtl004 vgSAP 1.0
anmtl004 rootvg 4.7
anmtl004 vgSAPX 2.6

$ perl -ane '$t{$F[0]." ".$F[7]} += $F[2]*(substr($F[3], 0, -1)/100); END { foreach $k ( keys %t ) { printf("%s %.1f\n", $k, $t{$k}/1000); } }' rootvg

anmtl004 vgSAP 1.0
anmtl004 vgSAPX 2.6
anmtl004 rootvg 4.7
anmtl004 vgSAPD 1.6
anmtl004  6570.0

$