Parsing a log file and creating a report script

amir07 · August 10, 2017, 5:34pm

[/ICODE]The log file is huge and lot of information, i would like to parse and make a report .

below is the log file looks like:

REPORT DATE: Mon Aug  10 04:16:17 CDT 2017
SYSTEN VER: v1.3.0.9
TERMINAL TYPE: prod
SYSTEM: nb11cu51
UPTIME:   04:16AM   up 182 days 57 mins min
MODEL, TYPE, and SN: IBM,9026-P70,01100699F
PROCESSOR TYPE: PowerPC_RS64-II
NUMBER OF PROCESSORS: 4





rmt0             P2/Z1-A5                 SCSI 8mm Tape Drive (20000 MB)

        Manufacturer................EXABYTE
        Machine Type and Model......IBM-20GB
        Device Specific.(Z1)........41eA
        Serial Number...............60171713
        Device Specific.(LI)........A0000004
        Part Number.................59H4117
        FRU Number..................59H4120
        EC Level....................E30396
        Device Specific.(Z0)........0180020283000030
        Device Specific.(Z3)........


rmt1             P2-I6/Z1-A0              IBM Magstar MP Tape Subsystem

        Manufacturer................IBM     
        Machine Type and Model......03570C11        
        Serial Number...............0000000A6844
        Device Specific.(FW)........544D
        Loadable Microcode Level....A0B00E27

  tmscsi1          P2-I6/Z1-A6              SCSI I/O Controller Initiator Device

        Device Specific.(Z0)........0300
        Manufacturer................IBM     
        Machine Type and Model......PCI SCSI-2UW-0  

  pci2             P2                       PCI Bus

        Device Specific.(YL)........P2

        Device Specific.(YL)........P2-I7/Q1

  ssa0             P2-I7/Q1                 IBM SSA 160 SerialRAID Adapter (14109100)

        Part Number................. 09L569B
        FRU Number.................. 34L5318
        Serial Number...............S0237219
        EC Level....................    F23699
        Manufacturer................IBM053
        ROS Level and ID............B800    0000
        Loadable Microcode Level....05
        Device Driver Level.........00
        Displayable Message.........SSA-ADAPTER
        Device Specific.(Z0)........SDRAM=064
        Device Specific.(Z1)........CACHE=00
        Device Specific.(Z2)........UID=006094BE000037F4
        Device Specific.(YL)........P2-I7/Q1

  ssa1             P2-I8/Q1                 IBM SSA 160 SerialRAID Adapter (14109100)

        Part Number................. 09L5695
        FRU Number.................. 34L5388
        Serial Number...............S0270187
        EC Level....................    F23699
        Manufacturer................IBM053
        ROS Level and ID............B800    0000
        Loadable Microcode Level....05
        Device Driver Level.........00
        Displayable Message.........SSA-ADAPTER
        Device Specific.(Z0)........SDRAM=064
        Device Specific.(Z1)........CACHE=



  rmt1             P2-I6/Z1-A0              IBM Magstar MP Tape Subsystem

        Manufacturer................IBM
        Machine Type and Model......03570C11
        Serial Number...............0000000A6844
        Device Specific.(FW)........544D
        Loadable Microcode Level....A0B00E27

  tmscsi1          P2-I6/Z1-A6              SCSI I/O Controller Initiator Device

        Device Specific.(Z0)........0300
        Manufacturer................IBM
        Machine Type and Model......PCI SCSI-2UW-0

  pci2             P2                       PCI Bus

        Device Specific.(YL)........P2

  ssa0             P2-I7/Q1                 IBM SSA 160 SerialRAID Adapter (14109100)






 hdisk0           P2/Z1-A9                 16 Bit SCSI Disk Drive (9100 MB)

        Manufacturer................IBM     
        Machine Type and Model......DNES-309170W    
        FRU Number..................25L3101     
        ROS Level and ID............53414755
        Serial Number...............AJJ55889
        EC Level....................F42017    
        Part Number.................25L1861     
        Device Specific.(Z0)........000003029F00013A
        Device Specific.(Z1)........25L2871     
        Device Specific.(Z2)........0933
        Device Specific.(Z3)........00176
        Device Specific.(Z4)........0001
        Device Specific.(Z5)........22
        Device Specific.(Z6)........F42036    

  hdisk1           P2/Z1-Aa                 16 Bit LVD SCSI Disk Drive (9100 MB)

        Manufacturer................SEAGATE     
        Machine Type and Model......DPSS-309170N    
        FRU Number..................07N3675     
        ROS Level and ID............53393347
        Serial Number...............ZD11B560
        EC Level....................F79851    
        Part Number.................07N3721     
        Device Specific.(Z0)........000003029F00013A
        Device Specific.(Z1)........07N4912     
        Device Specific.(Z2)........0933
        Device Specific.(Z3)........00231
        Device Specific.(Z4)........0001
        Device Specific.(Z5)........22
        Device Specific.(Z6)........F79851    

  pdisk0          USSAB540-D4              SSA160 Physical Disk Drive (9100 MB)

        Manufacturer................IBM     
        Machine Type and Model......DRVC09B
        Part Number.................34L8483     
        ROS Level and ID............0023
        Serial Number...............680BA636SA
        EC Level....................E32094    
        Device Specific.(Z2)........CUSHA023  
        Device Specific.(Z3)........34L8483     
        Device Specific.(Z4)........00068


pdisk01          USSAB540-D4              SSA160 Physical Disk Drive (9100 MB)

        Manufacturer................MAC     
        Machine Type and Model......DRVC09A
        Part Number.................34L8483     
        ROS Level and ID............0023
        Serial Number...............680BA636TT
        EC Level....................E32094    
        Device Specific.(Z2)........CUSHA023  
        Device Specific.(Z3)........34L8483     
        Device Specific.(Z4)........00068

The output should like this :

SYSTEM: nb11cu51
MODEL: IBM
TYPE: 7026-H70
SN:0110068BF
NUMBER OF PROCESSORS: 4
Total pdisk:2
Total hdisk:2
Total Tape Drive (rmt0,1): 2
SearialRAID adapeter (ssa0,1): 2


Device  Manufacturer   Machine Type Model     FRU Number     Serial Number     Part Number
hdisk0    IBM           DNES-309170W        25L3101        AJJ55889    25L1861
hdisk1  SEAGATE           DPSS-309170N        07N3675        ZD11B560    07N3721
pdisk0  IBM            DRVC09B            NA        680BA636SA      NA
pdisk01 MAC           DRVC09A            NA        680BA636TT    NA
rmt0    EXABYTE           IBM-20GB            59H4120        A0000004        59H4117
ssa0    09L569A           IBM053            34L5318     S0237219    09L569A

Don_Cragun · August 10, 2017, 6:36pm

First...
Please note that learning to correctly use CODE tags is not that difficult..

Use Code Tags when you post any code or data samples so others can easily read your code. You can easily do this by highlighting your code and then clicking on the # in the editing menu. (You can also type code tags

```text
 and 
```

by hand.) The following tutorial demonstrates how to do this.

And then...

What operating system and shell are you using?
What have you tried to solve this problem on your own?
Where are you stuck?
Do the devices in your output have to be sorted, or can the output be in the same order as your input?

amir07 · August 14, 2017, 3:30pm

IBM AIX 5.3
shell script
device needs to be sorted
i will put my code what I have

Don_Cragun · August 14, 2017, 3:36pm

Please only put sample input, sample output, and code segments in CODE tags; not everything in your post.

We look forward to seeing what code you currently have, the output it produces from your sample input, and a description of where you are stuck trying to get the output you want.

RudiC · August 14, 2017, 5:15pm

How are devices selected for the report? rmt1 exists twice but is listed once only, ssa0 is reported, but ssa1 is not. Why should we ignore tmscsi1 ?

bakunin · August 15, 2017, 4:34am

This is a very old system (if i had to guess i'd say 15+ years) and i am astonished that it even runs AIX 5.3. My old 43P won't run anything newer than 5.2. SSA (IBMs idea of obsoleting SCSI) went out of fashion about the turn of the century. (Btw.: i don't know the "9026-P70" in your source report but the "7026-H70" from your sample output. Might it be that your cut&paste didn't work well?)

AS it seems this an output from either the lscfg -vp command or from prtconf (not sure if this was already present in 5.3). Wouldn't it be easier - instead of parsing this output - to use the lsdev and lsattr commands to get exactly the specifications you want to know about every individual device?

For this, of course, one would need to know what exactly you want to see in your report. Right now this is only vaguely defined.

I hope this helps.

bakunin

amir07 · August 16, 2017, 10:31am

The reason tmscsi ignored because there is no manufacturer name, serial number and part number.
ssa1 also need to be reported.
all devices with manufacturer name, seriall number and part number need to be reported!

---------- Post updated at 09:30 AM ---------- Previous update was at 09:11 AM ----------

for h in nb11cu51
do
ssh root@$h lsattr -Elsys0 | awk -v h=$h '
echo { print "Device  Manufacturer   Machine_Type Model     FRU_Number     Serial_Number     Part_Number" }
$1=="Manufacturer   "{s=$2}
$1=="Manufacturer"{m=$2}
$1=="Model"{n=$2}
$1=="Fru_Number"{o=$2}
$1=="Serial_number"{p=$2}
$1=="Part_Number"{q=$2}
END {print h,m,s,n,o,q}'
done

I could not figure out the heading detail of the report and counts.

---------- Post updated at 09:31 AM ---------- Previous update was at 09:30 AM ----------

for h in nb11cu51
do
ssh root@$h lsattr -Elsys0 | awk -v h=$h '
echo { print "Device  Manufacturer   Machine_Type Model     FRU_Number     Serial_Number     Part_Number" }
$1=="Manufacturer   "{s=$2}
$1=="Manufacturer"{m=$2}
$1=="Model"{n=$2}
$1=="Fru_Number"{o=$2}
$1=="Serial_number"{p=$2}
$1=="Part_Number"{q=$2}
END {print h,m,s,n,o,q}'
done

I could not figure out the heading detail of the report and counts.

RudiC · August 16, 2017, 11:06am

How about

awk -F"\.\.+"   '
BEGIN           {HD = "Device	Manufacturer	Machine Type and Model	FRU Number	Serial Number	Part Number"
                 for (MX=n=split (HD, HDArr, "\t"); n>0; n--)   {SRCH[HDArr[n]]
                                                                 LNG[n] = length(HDArr[n])
                                                                }
                }

/^(SYSTEM|MODEL|PROCESS|NUMBER)/ 

!LEADINDONE &&
!NF             {LEADINDONE    = 1
                 print
                 print HD
                 gsub (/[^\t]/, "-", HD)
                 print HD
                }
!LEADINDONE     {next
                }

NF == 1         {TMPL = RES[HDArr[4]] RES[HDArr[5]] RES[HDArr[6]]
                 if (TMPL != "") for (i=1; i<=MX; i++) printf "%-*s%s", LNG, RES[HDArr]?RES[HDArr]:"NA", (i == MX)?ORS:OFS
#                       else     delete CNT[TMPCNT]
                 split ("", RES)
                 split ($0, T, " ")
                 RES[HDArr[1]] = T[1]
                 TMPCNT = T[1]
                 sub (/[0-9]*$/, _, TMPCNT)
                 CNT[TMPCNT]++
                }

NF < 2          {next
                }

                {sub (/^ */, "", $1)
                }

$1 in SRCH      {RES[$1] = $NF
                }

END             {for (i=1; i<=MX; i++) printf "%-*s%s", LNG, RES[HDArr]?RES[HDArr]:"NA", (i == MX)?ORS:OFS
                 printf RS
                 for (c in CNT) print "Total", c, ":", CNT[c]
                }

' OFS="\t" file
SYSTEM: nb11cu51
MODEL, TYPE, and SN: IBM,9026-P70,01100699F
PROCESSOR TYPE: PowerPC_RS64-II
NUMBER OF PROCESSORS: 4

Device    Manufacturer    Machine Type and Model    FRU Number    Serial Number    Part Number
------    ------------    ----------------------    ----------    -------------    -----------
rmt0  	EXABYTE     	IBM-20GB              	59H4120   	60171713     	59H4117    
rmt1  	IBM         	03570C11              	NA        	0000000A6844 	NA         
ssa0  	IBM053      	NA                    	 34L5318  	S0237219     	 09L569B   
ssa1  	IBM053      	NA                    	 34L5388  	S0270187     	 09L5695   
rmt1  	IBM         	03570C11              	NA        	0000000A6844 	NA         
hdisk0	IBM         	DNES-309170W          	25L3101     	AJJ55889     	25L1861     
hdisk1	SEAGATE     	DPSS-309170N          	07N3675     	ZD11B560     	07N3721     
pdisk0	IBM         	DRVC09B               	NA        	680BA636SA   	34L8483     
pdisk01	MAC         	DRVC09A               	NA        	680BA636TT   	34L8483     

Total	rmt	:	3
Total	pdisk	:	2
Total	ssa	:	3
Total	pci	:	2
Total	hdisk	:	2
Total	tmscsi	:	2

The total counts are at the end as they are available only then; the reason pci and tmscsi counts are shown is that your input file does NOT stick to a reasonable structure (if you remove the # from the delete CNT line, ssa count will disappear as well...).

bakunin · August 22, 2017, 8:37am

Instead of working hard to parse the output of the generic lsattr command why don't you use its abilities to display only the things you want to know?

From the man page of lsattr :

-F Format Displays the output in a user-specified format, where the Format parameter is a quoted list of column names separated by nonalphanumeric characters or white space. If white space is used as the separator, the lsattr command displays the output in aligned columns. Only column names from the Predefined Attributes (PdAt), Customized Attributes (CuAt), and the Extended Predefined Attributes (PdAtXtd) object classes can be specified. In addition to the column names, there are two special purpose names that can be used: the name description can be used to obtain a display of attribute descriptions and user_settable can be used to determine whether an attribute can be changed. This flag cannot be used with the -E, -D, -O, or -R flag.

-a Attribute Displays information for the specified attributes of a specific device or type of device. You can use one -a flag for each attribute name or multiple attribute names. If you use one -a flag for multiple attribute names, the list of attribute names must be enclosed in quotation marks with spaces between the names. If you use the -R flag, you must specify only one -a flag with only one attribute name. If you do not specify either the -a or -R flag, the lsattr command displays all information for all attributes of the specified device. The -a Attribute flag cannot be used with the -o Operation flag. This combination of flags causes the lsattr command to exit with an error message.

I hope this helps.

bakunin