SunFire V440 Quad gigabit ce1 won't run Gig

fixit9660 · July 6, 2009, 10:48am

Hi there. My first time posting so apologies for any indescretions in advance. :o I've got two SunFire V440 servers running Solaris 5v10. Both have a quad Gigabit ethernet card fitted, both have the same symptom; that is, I can't get the first port ce1 to run Gigabit. Motherboard ports are ce0 and ce5. the quad card ports are ce1 - ce4 inclusive. Port ce2 runs Gig fine. (Haven't tried ports ce3 or ce4). When I configure ce1 for Gig it doesn't even show a green l.e.d. when I plug in the patch cable. The switch and patch cables are definitely good. Port ce1 work perfectly at 100Mb/s FDX on both servers.
I guess I'm missing a patch or something but all help appreciated. By the way I'm a Network Engineer, not a Solaris Admin so anticipate a high degree of noobness relating to Solaris and Servers. Apologies in advance. Thanks, Andy.

jlliagre · July 6, 2009, 12:19pm

Perhaps someone has configured ce1 to stick at 100Mb/s instead of auto-negotiating the speed/duplex mode.

What says:

ndd -set /dev/ce instance 1
ndd /dev/ce adv_autoneg_cap
grep ce1 /etc/system
grep ndd /etc/rc*.d/*

DukeNuke2 · July 6, 2009, 1:09pm

also make sure your switch is configured for autoneg! call your network guys and make sure it works. as a test you can maybe use the port which ce2 uses (login through console!).

good luck,
DN2

fixit9660 · July 6, 2009, 8:07pm

Hi jlliagre, output as requested from my server:

datanetnms4 > ndd -set /dev/ce instance 1
datanetnms4 > ndd /dev/ce adv_autoneg_cap
0
datanetnms4 > grep ce1 /etc/system
datanetnms4 > grep ndd /etc/rc*.d/*
/etc/rc0.d/K34ncalogd: /usr/sbin/ndd -set /dev/nca nca_log_file $i
/etc/rc0.d/K34ncalogd: /usr/sbin/ndd -set /dev/nca nca_log_size $logd_file_size
/etc/rc0.d/K34ncalogd: /usr/sbin/ndd -set /dev/nca nca_logging_on 1
/etc/rc0.d/K34ncalogd: /usr/sbin/ndd -set /dev/nca nca_logging_on 0
/etc/rc1.d/K34ncalogd: /usr/sbin/ndd -set /dev/nca nca_log_file $i
/etc/rc1.d/K34ncalogd: /usr/sbin/ndd -set /dev/nca nca_log_size $logd_file_size
/etc/rc1.d/K34ncalogd: /usr/sbin/ndd -set /dev/nca nca_logging_on 1
/etc/rc1.d/K34ncalogd: /usr/sbin/ndd -set /dev/nca nca_logging_on 0
/etc/rc2.d/S42ncakmod: # Set the default HTTPD door in NCA via ndd
/etc/rc2.d/S42ncakmod: /usr/sbin/ndd -set /dev/nca httpd_door_path \
/etc/rc2.d/S69inet:ndd -set /dev/ce instance 0
/etc/rc2.d/S69inet:ndd -set /dev/ce adv_1000fdx_cap 0
/etc/rc2.d/S69inet:ndd -set /dev/ce adv_1000hdx_cap 0
/etc/rc2.d/S69inet:ndd -set /dev/ce adv_100fdx_cap 1
/etc/rc2.d/S69inet:ndd -set /dev/ce adv_100hdx_cap 0
/etc/rc2.d/S69inet:ndd -set /dev/ce adv_10fdx_cap 0
/etc/rc2.d/S69inet:ndd -set /dev/ce adv_10hdx_cap 0
/etc/rc2.d/S69inet:ndd -set /dev/ce adv_autoneg_cap 0
/etc/rc2.d/S69inet:ndd -set /dev/ce instance 1
/etc/rc2.d/S69inet:ndd -set /dev/ce adv_1000fdx_cap 0
/etc/rc2.d/S69inet:ndd -set /dev/ce adv_1000hdx_cap 0
/etc/rc2.d/S69inet:ndd -set /dev/ce adv_100fdx_cap 1
/etc/rc2.d/S69inet:ndd -set /dev/ce adv_100hdx_cap 0
/etc/rc2.d/S69inet:ndd -set /dev/ce adv_10fdx_cap 0
/etc/rc2.d/S69inet:ndd -set /dev/ce adv_10hdx_cap 0
/etc/rc2.d/S69inet:ndd -set /dev/ce adv_autoneg_cap 0
/etc/rc2.d/S94ncalogd: /usr/sbin/ndd -set /dev/nca nca_log_file $i
/etc/rc2.d/S94ncalogd: /usr/sbin/ndd -set /dev/nca nca_log_size $logd_file_size
/etc/rc2.d/S94ncalogd: /usr/sbin/ndd -set /dev/nca nca_logging_on 1
/etc/rc2.d/S94ncalogd: /usr/sbin/ndd -set /dev/nca nca_logging_on 0
/etc/rcS.d/K34ncalogd: /usr/sbin/ndd -set /dev/nca nca_log_file $i
/etc/rcS.d/K34ncalogd: /usr/sbin/ndd -set /dev/nca nca_log_size $logd_file_size
/etc/rcS.d/K34ncalogd: /usr/sbin/ndd -set /dev/nca nca_logging_on 1
/etc/rcS.d/K34ncalogd: /usr/sbin/ndd -set /dev/nca nca_logging_on 0
datanetnms4 >

---------- Post updated at 01:07 AM ---------- Previous update was at 12:59 AM ----------

Hi DukeNuke2,

The switch isn't configured for auto-neg, we set all our switches to FDX where the server interfaces are multi-speed capable. It stops them re-negotiating and messing up the comms. By the way I AM the Network Guy and yes, (as per my original post) the switch port and patch cable are known good. I have tried the network connection to ce2 in port ce1 with the same, non-operating, results. The network connection for ce1 works perfectly in ce2.
The server interface doesn't even show a green (physical) l.e.d. when I connect up the patch cable, when it's configured for Gig, which is what I find most confusing.

achenle · July 6, 2009, 10:32pm

These lines look like your problem:

/etc/rc2.d/S69inet:ndd -set /dev/ce instance 0
/etc/rc2.d/S69inet:ndd -set /dev/ce adv_1000fdx_cap 0
/etc/rc2.d/S69inet:ndd -set /dev/ce adv_1000hdx_cap 0
/etc/rc2.d/S69inet:ndd -set /dev/ce adv_100fdx_cap 1  <-----
/etc/rc2.d/S69inet:ndd -set /dev/ce adv_100hdx_cap 0
/etc/rc2.d/S69inet:ndd -set /dev/ce adv_10fdx_cap 0
/etc/rc2.d/S69inet:ndd -set /dev/ce adv_10hdx_cap 0
/etc/rc2.d/S69inet:ndd -set /dev/ce adv_autoneg_cap 0

Of interest when setting up copper gigE:

The IEEE 802.3 standard states that you must support and test autonegotiation
enabled to certify a product IEEE 802.3 compliant, and for multivendor
interoperability (for example, testing at the UNH Interoperability Laboratory). There
are no requirements in the standard to support locked down or forced configurations
using autonegotiation disabled. As a result, there are no requirements for vendors to
test multivendor interoperability between products with autonegotiation disabled.

The IEEE 802.3ab specification does not allow for forced mode 1000BASE-T with
autonegotiation disabled running at 1000 Mbps. As a result, many switch vendors do
not support forced mode. Although the transceiver used in the Sun� Gigaswift
Ethernet UTP adapter 1.0 is configurable for the 1000 Mbps forced mode and the ce
driver allows this mode be aware that it does not work under certain circumstances.

I'm willing to bet that the number of vendors who pay to have labs test their gear outside of the IEEE specs is pretty close to zero.

incredible · July 7, 2009, 2:51am

Put the above into a script file to force autoneg from server end too.

jlliagre · July 7, 2009, 4:03am

That wouldn't allow Gb to work.

Simply comment out the very same lines and reboot.

fixit9660 · July 7, 2009, 9:16am

From the CLI, why does this work:

ndd -set /dev/ce instance 2
ndd -set /dev/ce adv_1000fdx_cap 1
ndd -set /dev/ce adv_1000hdx_cap 0
ndd -set /dev/ce adv_100fdx_cap 0
ndd -set /dev/ce adv_100hdx_cap 0
ndd -set /dev/ce adv_10fdx_cap 0
ndd -set /dev/ce adv_10hdx_cap 0
ndd -set /dev/ce adv_autoneg_cap 0

and this doesn't?

ndd -set /dev/ce instance 1
ndd -set /dev/ce adv_1000fdx_cap 1
ndd -set /dev/ce adv_1000hdx_cap 0
ndd -set /dev/ce adv_100fdx_cap 0
ndd -set /dev/ce adv_100hdx_cap 0
ndd -set /dev/ce adv_10fdx_cap 0
ndd -set /dev/ce adv_10hdx_cap 0
ndd -set /dev/ce adv_autoneg_cap 0

Perderabo · July 7, 2009, 10:09am

Because you didn't screw up instance 2 with your rc2.d script. Please try jlliagre's advice "Simply comment out the very same lines and reboot. "

fixit9660 · July 7, 2009, 10:20am

Ah right, so you're saying that the rc2.d scripts override the CLI commands?

Perderabo · July 7, 2009, 2:07pm

Not really, no. Here are a few quotes from the book:

Note According to IEEE 802.3 specifications, when using 1000 Mbps speed over
copper media, autonegotiation is not optional. In the absence of autonegotiation, (for
example, while using Forced mode), link syncing between link partners may not
happen and the link may not come up.

To Set Autonegotiation to off for Other Noncompliant Network Equipment
If your network equipment does not support autonegotiation, you can set
autonegotiation to off on the GigaSwift Ethernet (ce) device.

Set the following GigaSwift Ethernet driver parameters to values according to the
documentation that shipped with your switch:

adv-1000fdx-cap

adv-1000hdx-cap

adv-ampause-cap

adv-pause-cap

Set the adv-autoneg-cap parameter to 0.

When link-up is attempted with autonegotiation enabled (the default), the link parameter capabilities (such as speed in megabits/second, duplex and link-clock mastership only for 1000 Mbps) are automatically negotiated between the device and its link partner.

The concept of link clock mastership was introduced with gigabit twisted-pair technology. One side of the link is the master that provides the link clock and the other side is the slave that uses the link clock. Once this relationship is established, the link is up, and data can be communicated. Two physical layer parameters control whether your side is the master or the slave, or whether mastership is negotiated with the link partner....

I gather that following the documentation is not your style, you don't seem too interested in the advice to turn on auto-negotiation. When I see language like the above, I do turn it on. I don't know what happens when you run without it. Good luck with the pause and clock mastership stuff. Since you're a network dude I assume you have it all under control.

Anyway here is another quote from the book...

So what happens if you do? Gosh, Andy, I don't know. But based on your report, I'm not missing much.