Hi, a couple of T2000 servers did not show the component status and the output was too short. There was a patch that needed to be installed. I also updated to the latest recommended patch cluster and OBP patch.
After reboot/restart of picl, it worked ok for a couple of weeks. Then suddenly it went back to its original state. prtdiag -v output not correct, but /usr/platform/`uname -i`/sbin/prtdiag -v is correct. Even this happens intermittently. What could be the problem. need to resolve asap
can you run this:
which prtdiag
your system should just use the correct version but i feel that it is not.....
what version you talking abt?
SunFire T2000, now free with any purchase over $10. (while supplies last)
How about some output from prtdiag? ...or error messages? ...or telling us what patch was installed. ...or tell us about any hardware attached to the server such as a storage array? ...or maybe anything that would actually help troubleshoot the problem?
Boot device: disk File and args:
ufs-file-system
Loading: /platform/SUNW,Sun-Fire-T200/boot_archive
Loading: /platform/sun4v/boot_archive
ramdisk-root hsfs-file-system
Loading: /platform/SUNW,Sun-Fire-T200/kernel/sparcv9/unix
Loading: /platform/sun4v/kernel/sparcv9/unix
SunOS Release 5.10 Version Generic_138888-03 64-bit
Copyright 1983-2008 Sun Microsystems, Inc. All rights reserved.
Use is subject to license terms.
os-io sorry, variable 'xmit_hiwat' is not defined in the 'udp' module
sorry, variable 'recv_hiwat' is not defined in the 'udp' module
NOTICE: nxge1: xcvr addr:0x0c - link is down
Hostname: domainwer12
NOTICE: nxge1: xcvr addr:0x0c - link is up 1000 Mbps full duplex
/dev/md/rdsk/d5 is clean
/dev/md/rdsk/d3 is clean
/dev/md/rdsk/d100 is clean
/dev/rdsk/c2t600A0B800029CE40000021D847868CA3d0s0 is clean
Reading ZFS config: done.
Authorized uses only. All activity may be monitored and reported.
domainwer12 console login: root
Password: starting Sun StorEdge EBS daemons:
nsrexecd
Apr 5 08:18:05 domainwer12 root: Oracle Cluster Ready Services starting up automatically.
Apr 5 08:18:11 domainwer12 root: Cluster Ready Services completed waiting on dependencies.
Apr 5 08:18:11 domainwer12 last message repeated 2 times
Apr 5 08:18:11 domainwer12 root: Running CRSD with TZ = Singapore
Apr 5 08:18:14 domainwer12 login: ROOT LOGIN /dev/console
Last login: Sun Apr 5 08:11:12 on console
Authorized uses only. All activity may be monitored and reported.
Sourcing //.profile-EIS.....
root@domainwer12 #
root@domainwer12 #
root@domainwer12 #
root@domainwer12 #
root@domainwer12 #
root@domainwer12 #
root@domainwer12 # bash
root@domainwer12 # prtdiag -v
System Configuration: Sun Microsystems sun4v Sun Fire T200
Memory size: 16256 Megabytes
================================ Virtual CPUs ================================
CPU ID Frequency Implementation Status
------ --------- ---------------------- -------
0 1200 MHz SUNW,UltraSPARC-T1 on-line
1 1200 MHz SUNW,UltraSPARC-T1 on-line
2 1200 MHz SUNW,UltraSPARC-T1 on-line
3 1200 MHz SUNW,UltraSPARC-T1 on-line
4 1200 MHz SUNW,UltraSPARC-T1 on-line
5 1200 MHz SUNW,UltraSPARC-T1 on-line
6 1200 MHz SUNW,UltraSPARC-T1 on-line
7 1200 MHz SUNW,UltraSPARC-T1 on-line
8 1200 MHz SUNW,UltraSPARC-T1 on-line
9 1200 MHz SUNW,UltraSPARC-T1 on-line
10 1200 MHz SUNW,UltraSPARC-T1 on-line
11 1200 MHz SUNW,UltraSPARC-T1 on-line
12 1200 MHz SUNW,UltraSPARC-T1 on-line
13 1200 MHz SUNW,UltraSPARC-T1 on-line
14 1200 MHz SUNW,UltraSPARC-T1 on-line
15 1200 MHz SUNW,UltraSPARC-T1 on-line
16 1200 MHz SUNW,UltraSPARC-T1 on-line
17 1200 MHz SUNW,UltraSPARC-T1 on-line
18 1200 MHz SUNW,UltraSPARC-T1 on-line
19 1200 MHz SUNW,UltraSPARC-T1 on-line
20 1200 MHz SUNW,UltraSPARC-T1 on-line
21 1200 MHz SUNW,UltraSPARC-T1 on-line
22 1200 MHz SUNW,UltraSPARC-T1 on-line
23 1200 MHz SUNW,UltraSPARC-T1 on-line
24 1200 MHz SUNW,UltraSPARC-T1 on-line
25 1200 MHz SUNW,UltraSPARC-T1 on-line
26 1200 MHz SUNW,UltraSPARC-T1 on-line
27 1200 MHz SUNW,UltraSPARC-T1 on-line
28 1200 MHz SUNW,UltraSPARC-T1 on-line
29 1200 MHz SUNW,UltraSPARC-T1 on-line
30 1200 MHz SUNW,UltraSPARC-T1 on-line
31 1200 MHz SUNW,UltraSPARC-T1 on-line
========================= IO Configuration =========================
IO
Location Type Slot Path Name Model
----------- ----- ---- --------------------------------------------- ------------------------- ---------
IOBD/NET0 PCIE IOBD /pci@780/pci@0/pci@1/network@0 network-pciex8086,105e
IOBD/NET1 PCIE IOBD /pci@780/pci@0/pci@1/network@0,1 network-pciex8086,105e
IOBD/PCIE PCIE IOBD /pci@780/pci@0/pci@9/scsi@0 scsi-pciex1000,56 LSI,1064E
IOBD/NET2 PCIE IOBD /pci@7c0/pci@0/pci@2/network@0 network-pciex8086,105e
IOBD/NET3 PCIE IOBD /pci@7c0/pci@0/pci@2/network@0,1 network-pciex8086,105e
IOBD/PCIE1 PCIE 1 /pci@7c0/pci@0/pci@8/network@0 network-pciex108e,abcd SUNW,pcie+
IOBD/PCIE1 PCIE 1 /pci@7c0/pci@0/pci@8/network@0,1 network-pciex108e,abcd SUNW,pcie+
IOBD/PCIE1 PCIE 1 /pci@7c0/pci@0/pci@8/network@0,2 network-pciex108e,abcd SUNW,pcie+
IOBD/PCIE1 PCIE 1 /pci@7c0/pci@0/pci@8/network@0,3 network-pciex108e,abcd SUNW,pcie+
IOBD/PCIE2 PCIE 2 /pci@7c0/pci@0/pci@9/network@0 network-pciex108e,abcd SUNW,pcie+
IOBD/PCIE2 PCIE 2 /pci@7c0/pci@0/pci@9/network@0,1 network-pciex108e,abcd SUNW,pcie+
IOBD/PCIE2 PCIE 2 /pci@7c0/pci@0/pci@9/network@0,2 network-pciex108e,abcd SUNW,pcie+
IOBD/PCIE2 PCIE 2 /pci@7c0/pci@0/pci@9/network@0,3 network-pciex108e,abcd SUNW,pcie+
IOBD/PCIE0 PCIE 0 /pci@780/pci@0/pci@8/pci@0/TSI,mko@0 TSI,mko XVR200
IOBD/PCIX PCIX IOBD /pci@7c0/pci@0/pci@1/pci@0/isa@2 isa
IOBD/PCIX PCIX IOBD /pci@7c0/pci@0/pci@1/pci@0/usb@5 usb-pciclass,0c0310
IOBD/PCIX PCIX IOBD /pci@7c0/pci@0/pci@1/pci@0/usb@6 usb-pciclass,0c0310
IOBD/PCIX PCIX IOBD /pci@7c0/pci@0/pci@1/pci@0/ide@8 ide-pci10b9,5229
IOBD/PCIX1 PCIX 1 /pci@7c0/pci@0/pci@1/pci@0,2/SUNW,qlc@1 SUNW,qlc-pci1077,2422 QLA2460
IOBD/PCIX0 PCIX 0 /pci@7c0/pci@0/pci@1/pci@0,2/SUNW,qlc@2 SUNW,qlc-pci1077,2422 QLA2460
========================= HW Revisions =======================================
System PROM revisions:
----------------------
OBP 4.27.7 2007/10/24 08:24
IO ASIC revisions:
------------------
Location Path Device Revision
-------------------- --------------------------------------------- ------------------------------ ---------
IOBD/SAS-SATA-HBA /pci@780 SUNW,sun4v-pci 0
IOBD/SAS-SATA-HBA /pci@780/pci@0 pciex10b5,8532.bc 188
IOBD/SAS-SATA-HBA /pci@780/pci@0/pci@9 pciex10b5,8532.bc 188
IOBD/IO-BRIDGE /pci@7c0 SUNW,sun4v-pci 0
IOBD/PCI-SWITCH1 /pci@7c0/pci@0 pciex10b5,8532.bc 188
IOBD/PCI-BRIDGE /pci@7c0/pci@0/pci@1/pci@0,2 pciex8086,341.9 9
IOBD/GBE0 /pci@780/pci@0/pci@1/network@0 pciex8086,105e.108e.105e.6 6
IOBD/GBE0 /pci@780/pci@0/pci@1/network@0,1 pciex8086,105e.108e.105e.6 6
IOBD/GBE1 /pci@7c0/pci@0/pci@2/network@0 pciex8086,105e.108e.105e.6 6
IOBD/GBE1 /pci@7c0/pci@0/pci@2/network@0,1 pciex8086,105e.108e.105e.6 6
IOBD/SAS-SATA-HBA /pci@780/pci@0/pci@9/scsi@0 pciex1000,56.1000.1000.2 2
root@domainwer12 # cat /etc/system
*ident "@(#)system 1.18 97/06/27 SMI" /* SVR4 1.5 */
*
* SYSTEM SPECIFICATION FILE
*
* moddir:
*
* Set the search path for modules. This has a format similar to the
* csh path variable. If the module isn't found in the first directory
* it tries the second and so on. The default is /kernel /usr/kernel
*
* Example:
* moddir: /kernel /usr/kernel /other/modules
* root device and root filesystem configuration:
*
* The following may be used to override the defaults provided by
* the boot program:
*
* rootfs: Set the filesystem type of the root.
*
* rootdev: Set the root device. This should be a fully
* expanded physical pathname. The default is the
* physical pathname of the device where the boot
* program resides. The physical pathname is
* highly platform and configuration dependent.
*
* Example:
* rootfs:ufs
* rootdev:/sbus@1,f8000000/esp@0,800000/sd@3,0:a
*
* (Swap device configuration should be specified in /etc/vfstab.)
* exclude:
*
* Modules appearing in the moddir path which are NOT to be loaded,
* even if referenced. Note that `exclude' accepts either a module name,
* or a filename which includes the directory.
*
* Examples:
* exclude: win
* exclude: sys/shmsys
* forceload:
*
* Cause these modules to be loaded at boot time, (just before mounting
* the root filesystem) rather than at first reference. Note that
* forceload expects a filename which includes the directory. Also
* note that loading a module does not necessarily imply that it will
* be installed.
*
* Example:
* forceload: drv/foo
* set:
*
* Set an integer variable in the kernel or a module to a new value.
* This facility should be used with caution. See system(4).
*
* Examples:
*
* To set variables in 'unix':
*
* set nautopush=32
* set maxusers=40
*
* To set a variable named 'debug' in the module named 'test_module'
*
* set test_module:debug = 0x13
* Begin MDD root info (do not edit)
rootdev:/pseudo/md@0:0,0,blk
* End MDD root info (do not edit)
* ORACLE SETTINGS
set noexec_user_stack=1
set shmsys:shminfo_shmmax=4294967295
set shmsys:shminfo_shmmin=1
set shmsys:shminfo_shmmni=100
set shmsys:shminfo_shmseg=10
set semsys:seminfo_semmni=4096
set semsys:seminfo_semmsl=100
set semsys:seminfo_semmns=25000
set semsys:seminfo_semopm=100
set semsys:seminfo_semvmx=32767
set udp:xmit_hiwat=65536
set udp:recv_hiwat=65536
* ORACLE SETTINGS
* Require NFS clients to use privileged ports
set nfssrv:nfs_portmon=1
root@domainwer12 # Apr 5 08:19:05 domainwer12 sshd[2946]: Failed none for snsravi from 172.20.2.200 port 2772 ssh2
root@domainwer12 #
Now the OBP is 4.30, not 4.27 as you see above. But same prtdiag output
This is not a problem at all.
Almost all the Niagra servers (T200, T5120, T5xxx, T1xxx, T2xxx). gives the same brief output.
Prtdiag is dependent on the hardware, so basically it is how much your hardware is willing to give to the OS.
unfortunately, what I've noticed is that these servers don't give much information in the prtdiag command.
your prtdiag command is perfectly correct, and there's no way you're going to get more information.
For more hardware information, you can always check hardware components in ILOM.
Sun Fire, you're wrong. As I told you earlier, I have 2 similar systems and after upgrading the firmware the output came out with all components as "okay" etc.. streaming up to 3 to 4 pages. This is not normal.
thats interesting...you are right, I have checked the outputs of systems I set up, and looks like after firmware upgrade it gives more comprehensive information.
this is the output of one T5220, and firmware is up to date
System Configuration: Sun Microsystems sun4v SPARC Enterprise T2000
Memory size: 8064 Megabytes
================================ Virtual CPUs ================================
CPU ID Frequency Implementation Status
------ --------- ---------------------- -------
0 1000 MHz SUNW,UltraSPARC-T1 on-line
1 1000 MHz SUNW,UltraSPARC-T1 on-line
2 1000 MHz SUNW,UltraSPARC-T1 on-line
3 1000 MHz SUNW,UltraSPARC-T1 on-line
4 1000 MHz SUNW,UltraSPARC-T1 on-line
5 1000 MHz SUNW,UltraSPARC-T1 on-line
6 1000 MHz SUNW,UltraSPARC-T1 on-line
7 1000 MHz SUNW,UltraSPARC-T1 on-line
8 1000 MHz SUNW,UltraSPARC-T1 on-line
9 1000 MHz SUNW,UltraSPARC-T1 on-line
10 1000 MHz SUNW,UltraSPARC-T1 on-line
11 1000 MHz SUNW,UltraSPARC-T1 on-line
12 1000 MHz SUNW,UltraSPARC-T1 on-line
13 1000 MHz SUNW,UltraSPARC-T1 on-line
14 1000 MHz SUNW,UltraSPARC-T1 on-line
15 1000 MHz SUNW,UltraSPARC-T1 on-line
======================= Physical Memory Configuration ========================
Segment Table:
--------------------------------------------------------------
Base Segment Interleave Bank Contains
Address Size Factor Size Modules
--------------------------------------------------------------
0x0 8 GB 4 2 GB MB/CMP0/CH0/R0/D0
MB/CMP0/CH0/R0/D1
2 GB MB/CMP0/CH1/R0/D0
MB/CMP0/CH1/R0/D1
2 GB MB/CMP0/CH2/R0/D0
MB/CMP0/CH2/R0/D1
2 GB MB/CMP0/CH3/R0/D0
MB/CMP0/CH3/R0/D1
========================= IO Configuration =========================
IO
Location Type Slot Path Name Model
----------- ----- ---- --------------------------------------------- ------------------------- ---------
IOBD/NET0 PCIE IOBD /pci@780/pci@0/pci@1/network@0 network-pciex8086,105e
IOBD/NET1 PCIE IOBD /pci@780/pci@0/pci@1/network@0,1 network-pciex8086,105e
IOBD/PCIE PCIE IOBD /pci@780/pci@0/pci@9/scsi@0 scsi-pciex1000,56 LSI,1064E
IOBD/NET2 PCIE IOBD /pci@7c0/pci@0/pci@2/network@0 network-pciex8086,105e
IOBD/NET3 PCIE IOBD /pci@7c0/pci@0/pci@2/network@0,1 network-pciex8086,105e
IOBD/PCIX PCIX IOBD /pci@7c0/pci@0/pci@1/pci@0/isa@2 isa
IOBD/PCIX PCIX IOBD /pci@7c0/pci@0/pci@1/pci@0/usb@5 usb-pciclass,0c0310
IOBD/PCIX PCIX IOBD /pci@7c0/pci@0/pci@1/pci@0/usb@6 usb-pciclass,0c0310
IOBD/PCIX PCIX IOBD /pci@7c0/pci@0/pci@1/pci@0/ide@8 ide-pci10b9,5229
========================= HW Revisions =======================================
System PROM revisions:
----------------------
OBP 4.29.0.a 2008/09/15 12:01
IO ASIC revisions:
------------------
Location Path Device Revision
-------------------- --------------------------------------------- ------------------------------ ---------
MB/SAS-SATA-HBA /pci@780 SUNW,sun4v-pci 0
MB/SAS-SATA-HBA /pci@780/pci@0 pciex10b5,8532.bc 188
MB/SAS-SATA-HBA /pci@780/pci@0/pci@9 pciex10b5,8532.bc 188
MB/IO-BRIDGE /pci@7c0 SUNW,sun4v-pci 0
MB/PCI-BRIDGE /pci@7c0/pci@0 pciex10b5,8532.bc 188
MB/PCI-BRIDGE /pci@7c0/pci@0/pci@8 pciex10b5,8532.bc 188
MB/SAS-SATA-HBA /pci@780/pci@0/pci@9/scsi@0 pciex1000,56.1000.1000.2 2
============================ Environmental Status ============================
Fan sensors:
------------------------------------------------------------
Location Sensor Status
------------------------------------------------------------
NNL084303T:CH/FT0/FM0 RS ok
NNL084303T:CH/FT0/FM1 RS ok
NNL084303T:CH/FT0/FM2 RS ok
NNL084303T:CH/FT2 RS ok
Temperature sensors:
------------------------------------------------------------
Location Sensor Status
------------------------------------------------------------
NNL084303T:CH/IOBD/IOB T_CORE ok
NNL084303T:CH/IOBD T_AMB ok
NNL084303T:CH/MB/CMP0 T_TCORE ok
NNL084303T:CH/MB/CMP0 T_BCORE ok
NNL084303T:CH/MB T_AMB ok
NNL084303T:CH/PDB T_AMB ok
Current sensors:
------------------------------------------------------------
Location Sensor Status
------------------------------------------------------------
NNL084303T:CH/MB I_VCORE ok
NNL084303T:CH/MB I_VMEML ok
NNL084303T:CH/MB I_VMEMR ok
Current indicators:
------------------------------------------------------------
Location Indicator Condition
------------------------------------------------------------
NNL084303T:CH/IOBD I_USB0 ok
NNL084303T:CH/IOBD I_USB1 ok
NNL084303T:CH/FIOBD I_USB ok
Voltage sensors:
------------------------------------------------------------
Location Sensor Status
------------------------------------------------------------
NNL084303T:CH/SC/BAT V_BAT ok
NNL084303T:CH/IOBD V_+1V5 ok
NNL084303T:CH/IOBD V_+1V8 ok
NNL084303T:CH/IOBD V_+3V3MAIN ok
NNL084303T:CH/IOBD V_+3V3STBY ok
NNL084303T:CH/IOBD V_+1V ok
NNL084303T:CH/IOBD V_+1V2 ok
NNL084303T:CH/IOBD V_+5V ok
NNL084303T:CH/IOBD V_-12V ok
NNL084303T:CH/IOBD V_+12V ok
NNL084303T:CH/MB V_+1V5 ok
NNL084303T:CH/MB V_VMEML ok
NNL084303T:CH/MB V_VMEMR ok
NNL084303T:CH/MB V_VTTL ok
NNL084303T:CH/MB V_VTTR ok
NNL084303T:CH/MB V_+3V3STBY ok
NNL084303T:CH/MB V_VCORE ok
LEDs:
------------------------------------------------------------
Location LED State
------------------------------------------------------------
NNL084303T:CH/FT0/FM0 SERVICE off
NNL084303T:CH/FT0/FM1 SERVICE off
NNL084303T:CH/FT0/FM2 SERVICE off
NNL084303T:CH/FT2 SERVICE off
NNL084303T:CH/SYS ACT steady
NNL084303T:CH/SYS LOCATE off
NNL084303T:CH/SYS SERVICE off
NNL084303T:CH/SYS REAR_FAULT off
NNL084303T:CH/SYS TEMP_FAULT off
NNL084303T:CH/SYS TOP_FAN_FAULT off
NNL084303T:CH/HDD0 SERVICE off
NNL084303T:CH/HDD0 OK2RM off
NNL084303T:CH/HDD1 SERVICE off
NNL084303T:CH/HDD1 OK2RM off
NNL084303T:CH/HDD2 SERVICE off
NNL084303T:CH/HDD2 OK2RM off
NNL084303T:CH/HDD3 SERVICE off
NNL084303T:CH/HDD3 OK2RM off
============================ FRU Status ============================
Location Name Status
------------------------------------------------------
NNL084303T:CH/FT0/FM0 FAN enabled
NNL084303T:CH/FT0/FM1 FAN enabled
NNL084303T:CH/FT0/FM2 FAN enabled
NNL084303T:CH/FT2 FAN enabled
NNL084303T:CH SC enabled
NNL084303T:CH IOBD enabled
NNL084303T:CH MB enabled
NNL084303T:CH/MB/CMP0/CH0/R0/D0 DIMM enabled
NNL084303T:CH/MB/CMP0/CH0/R0/D1 DIMM enabled
NNL084303T:CH/MB/CMP0/CH1/R0/D0 DIMM enabled
NNL084303T:CH/MB/CMP0/CH1/R0/D1 DIMM enabled
NNL084303T:CH/MB/CMP0/CH2/R0/D0 DIMM enabled
NNL084303T:CH/MB/CMP0/CH2/R0/D1 DIMM enabled
NNL084303T:CH/MB/CMP0/CH3/R0/D0 DIMM enabled
NNL084303T:CH/MB/CMP0/CH3/R0/D1 DIMM enabled
NNL084303T:CH PDB enabled
NNL084303T:CH FIOBD enabled
NNL084303T:CH SASBP enabled
NNL084303T:CH/PS0 PS enabled
NNL084303T:CH/PS1 PS enabled
============================ FW Version ============================
Version
------------------------------------------------------------
System Firmware 6.6.7 2008/09/29 09:36
====================== System PROM revisions =======================
Version
------------------------------------------------------------
OBP 4.29.0.a 2008/09/15 12:01
Chassis Serial Number
---------------------
NNL084303T
Exactly..
So why in my system its not happening 90% of the time. remaining 10% its intermittent or hangs at the prtdiag command. Kernel patch and firmware is LATEST
Assuming the prtdiag you run is /usr/sbin/prtdiag, that one is simply a shell script that calls "/usr/platform/`uname -i`/sbin/prtdiag" which you claim works fine ...
Its inconsistent most of the time. 2 similar systems behave very weirdly.Just assume this.. my prtdiag -v not showing detailed info as what Sun Fire posted earlier. What you see in his is what I want. It came for 1 week, then went back to the "useless" output. Case with SUN was escalated and they required the latest OBP and patch 138282-02 to be installed. I thought it was ok, then closed the case with them.Now it came back. Thinking of resolving it as soon w/o escalating again. That's y I need your help guys! 
incredible - i know of some similar issues regarding hardware query commands that either crashed a system or hung up the session. picl seemed to have restored order but patches were applied as this was recommended by sun. whether these were patches applied directly from sun or from a latest cluster, i'm not sure.
i think you should open up another case with sun as i think this is a code issue and nothing much we can do to help. until then, maybe a cron of /usr/platform/`uname -i`/sbin/prtdiag can help you along the way. run it once a day or maybe twice... see if you can get at least 1 of the 2 good outputs.
just me typing out out loud
i'm wondering if prtpicl shows different output from a good prtdiag result versus a bad prtdiag result.