Supported but not advertised 1000baseT/Full only in one node of RH cluster

Hello all,

I have two twins Proliant DL580 G3 with RHEL4U4 clustered with Oracle RAC with two nics working in each one.

All nics have the same firmware, driver and type of physical cable connected (Cat 5E). All ports of the switch have the same configuration but only one of the nics of only one server can negociate to 1Gbps.

The main difference between nic working 1Gbps and the rest is the others nics Support 1000baseT/Full but this mode is NOT Advertised.

I have readed similar posts but in this case, I have two twin boxes which only work one interface of one box... It's very rare.

Configuration
------------------------------
Options: autoneg on
driver: tg3 -> version 3.92n 04B8459653BDAA6D1C7B798
version: 3.92n
firmware-version: 5704-v3.27b, ASFIPMIc v2.36
Broadcom Corporation NetXtreme BCM5704 Gigabit Ethernet (rev 10)
Kernel: 2.6.9-78.0.17.ELsmp

Thanks and regards.

Last night we change the cable in server is working Gigabit and use it for the server can't negotiate Gbit and we used the same switch port with no results....

�Any suggestion why this nic can't negotiate Gigabit?

Hi Rubick,

for testing switch the cable of one of the ports where your adapter is not negotiating with the cable in the port that is negotiating ... if the other adapter works, it is a problem with the switch / switchport, if not it's an adapter problem. Did you try different switchport settings ? Do you experience any difference when setting the switchport to 1000/full rather than auto?

Rgds
zxmaus

We did a lot of test about 3 hours with service lost because is heartbeat interface, we tried many many differents configuracions in switch and adapter side (auto,full, etc...) with no results... We use the same cable and switch port with adapter working properly and adapter never shows advertisment 1000.

In this case, we have more servers and only one adapter of one server supports and advertisment one card in 1000/full others, only supports without advertisment.

Support tell us upgrade of all components... but I don't want to make this upgrade because one server is working with the same level of firmwares/drivers and I wan't to know the root problem not only "customer please, upgrade and maybe works"

By the other hand, maybe I will ask for mother board change as these interfaces are integrated.

I will upgrade this "X-file" when I come back of my holidays

Thanks.

Have you used ethtool to see statistics, settings, etc? See LXR / The Linux Cross Reference for lots of information about this driver internals.

Yes, and ethool shows I commented before:

# ethtool eth1
Settings for eth1:
        Supported ports: [ TP ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Half 1000baseT/Full ----> Suppports Gbit
        Supports auto-negotiation: Yes
        Advertised link modes:  10baseT/Full
                                100baseT/Half 100baseT/Full
                                --- Where is Advertised Gbit mode?

Looking at the source code, if a NIC is set in low power mode, it will not autonegotiate 1000baseT.

Suggest you dump the NVRAM of this NIC and a "good" NIC and see if they are the same.

You surprised me!

I have just made dump of working gigabit interface (1P) and the other not working interface (2P) and there are many differences. I show you only a part because is a bit long :

# diff eth1.1P.dump eth1.2P.dump
130,134c130,134
< 0x0000007f    0x17
< 0x00000080    0x08
< 0x00000081    0x5d
< 0x00000082    0x2e
< 0x00000083    0x0a
---
> 0x0000007f    0x16
> 0x00000080    0x35
> 0x00000081    0x81
> 0x00000082    0x1b
> 0x00000083    0x50
210,214c210,214
< 0x000000cf    0x17
< 0x000000d0    0x08
< 0x000000d1    0x5d
< 0x000000d2    0x2e
< 0x000000d3    0x0b
---
> 0x000000cf    0x16
> 0x000000d0    0x35
> 0x000000d1    0x81
> 0x000000d2    0x1b
> 0x000000d3    0x51

I tried to read this hex code but I don't understand matrix languaje

Thanks for your help!

Have a look at tg3.h. This gives you all the information you need.

This is out of my scope.. it's too hard for me.

Maybe I could download NVRAM of working nic and upload to non working nic... this should be work.

Next week support will change the mother board. I will wait for this change.

Thanks