Centos 4 32 bit - New kernel ethX MAC address order issue

I have compiled a new kernel (3.2.9) for centos 4/5/6 servers. There is an issue with the centos 4, 32 bit servers. The kernel changes the order in which the MAC address is determined and because of this the server network does not come up as the wrong MAC address are assigned. Even if we specify it correctly in /etc/sysconfig/network-scripts/ifcfg-eth* , kernel does not read it in the case of centos 4 during boot time. We passed using the ndev= parameter, that contain MAC address in the grub.conf file, this is also not doing good. Below is the way the old kernel identifies the MAC and new kernel identifies the MAC. My question is, Is there a way to change the order in which the kernel identifies the Card, so that it follows always the same sequence as in the Old kernel itself. Ie 00:xx:xx:xx:xx:2e will be always identified as eth0 and others also. You can see in the first case the lspci device �0000:03:08.0� is identified as eth0 and in the second case, �0000:03:08.0� is identified as eth2. I need 0000:03:08.0 always be identified as eth0 itself.

I have used the following in the file /etc/udev/rules.d/50-udev.rules Upon rebooting kernel seems to ignore it.

SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?", ATTR{address}=="00:xx:xx:xx:xx:2e", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth", NAME="eth0"
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?", ATTR{address}=="00:xx:xx:xx:xx:20", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth", NAME="eth1"
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?", ATTR{address}=="00:xx:xx:xx:xx:21", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth", NAME="eth2"

PS: The same kernel works fine in centos 5/6 servers. Issue seen only in centos 4 servers.

First case: How old kernel identifies MAC in /var/log/messages during boot up.

---------------------------------------
Mar 6 07:34:03 server kernel: [ 7.800794] e100 0000:03:08.0: eth0: addr 0xfeafb000, irq 18, MAC addr 00:xx:xx:xx:xx:2e

Mar 6 07:34:03 server kernel: [ 7.817023] tg3 0000:02:09.0: eth1: Tigon3 [partno(BCM95704A7) rev 2003] (PCIX:100MHz:64-bit) MAC address 00:xx:xx:xx:xx:20

Mar 6 07:34:03 server kernel: [ 7.825107] tg3 0000:02:09.1: eth2: Tigon3 [partno(BCM95704A7) rev 2003] (PCIX:100MHz:64-bit) MAC address 00:xx:xx:xx:xx:21
---------------------------------------

Second case: How new kernel identifies MAC in /var/log/messages during boot up.

---------------------------------------
Mar 6 07:23:42 server kernel: tg3 0000:02:09.0: eth0: Tigon3 [partno(BCM95704A7) rev 2003] (PCIX:100MHz:64-bit) MAC address 00:xx:xx:xx:xx:20

Mar 6 07:23:42 server kernel: tg3 0000:02:09.1: eth1: Tigon3 [partno(BCM95704A7) rev 2003] (PCIX:100MHz:64-bit) MAC address 00:xx:xx:xx:xx:21

Mar 6 07:23:42 server kernel: e100 0000:03:08.0: eth2: addr 0xfeafb000, irq 18, MAC addr 00:xx:xx:xx:xx:2e

do not use the ATTR{address} construct.

Use the ID== instead and use the bus ID of the system. This way, regardless of whether you change cards or not, you will be fine.

Set your rule to look like this:

SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ID=="0000:01:00.0", KERNEL=="eth*", NAME="eth0"

That is an example. 0000:01:00.0 is a reference point. Find your own bus ID out by using:

ethtool -i eth0

Do this for all your devices, and then you can simply do a udevadm trigger or udevtrigger assuming your devices are not in use.

You may also want to modify your /boot/grub/menu.lst to include the construct pci=nobfsort on your kernel line. This will ensure that devices scan in order.

It can be difficult to force a device into being eth0 when there's already another eth0. For this reason, when I need to force ethernet adapters into a specific order, I name them other things -- 'wan' or 'lan0' for instance.

When -- not if -- something new autodefaults to eth0, it won't "eat" the names you're using, making the order you want impossible from the get-go.

Corona,

A trick I have found, especially since the network is usually already fubar, is to modprobe -r the module for the NIC, and then make the udev change. Then modprobe the module back and all should work fine.

Certainly, but what happens next reboot, replacement, or reordering of PCI slots?

Using custom names means the conflict simply doesn't happen.

The next reboot, the udev rules are loaded appropriately based on PCI Bus ID, forcing the names to exist as intended.

PCI slots do not reorder themselves. This is especially an important distinction when dealing with enterprises that have a custom configuration, that for the purposes of homogony and manageability require this level of consistency.

I'm not pulling this out of my rump, it is what the last few companies I have worked for have done, specifically to make sure that consistent device names were kept.

Using the PCI Bus ID will not change so long as the controller is not damaged or the slot is not changed, and it should not be. If it is, it is as simple as changing the persistent naming rule in udev, which would have been changed in the case of a failed card if you used MAC addresses instead of BUS ID, but this saves that effort if the slot is reused, so long as you don't add a controller with a different number of physical ports (like a 2 port NIC to a 4 port NIC). However, most field services reps will replace a failed card in the open slot the previous card had occupied, preventing an outage or problem.

Of course not; humans do it.

Or kernel upgrades.

You know the saying, once bitten, twice shy? I have attempted to do as you say. Identical software would work on some systems but not others, and the ordering or the ability to reorder eth0 names broke every other udev upgrade. If it works for you, don't let anyone sneeze on it, because that could just be a happy accident.

Renaming one device overtop another just isn't trivial, it's full of pitfalls and race conditions; the best way to avoid them IMHO is to just not do it. Rename them something the kernel doesn't claim, and you won't have anything getting there first.