Respected Members,
We encountered an issue during node reboot when eth1 and eth5 of bond1 were behaving unusual(both eth1 and eth5 were in unknown states and ifdown and ifup were used to rectify). Please find the messages and configuration files as below and please let me know for any other input:
more ifcfg-bond1
# Please read /usr/share/doc/initscripts-*/sysconfig.txt
# for the documentation of these parameters.
IPV6INIT=no
ONBOOT=yes
DEVICE=bond1
BOOTPROTO=static
NETMASK=255.255.255.252
NM_CONTROLLED=no
IPADDR=192.168.0.1
USERCTL=no
BONDING_OPTS='mode=1 miimon=100 primary=eth1 primary_reselect=always'
more ifcfg-eth1
ONBOOT=yes
DEVICE=eth1
MASTER=bond1
BOOTPROTO=none
NM_CONTROLLED=no
ETHTOOL_OPTS='speed 100 duplex full autoneg off'
USERCTL=no
IPV6INIT=no
SLAVE=yes
more ifcfg-eth5
ONBOOT=yes
DEVICE=eth5
MASTER=bond1
BOOTPROTO=none
NM_CONTROLLED=no
ETHTOOL_OPTS='speed 100 duplex full autoneg off'
USERCTL=no
IPV6INIT=no
SLAVE=yes
cat /proc/net/bonding/bond1
Ethernet Channel Bonding Driver: v3.6.0 (September 26, 2009)
Bonding Mode: fault-tolerance (active-backup)
Primary Slave: eth1 (primary_reselect always)
Currently Active Slave: eth1
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0
Slave Interface: eth1
MII Status: up
Speed: 100 Mbps
Duplex: full
Link Failure Count: 8
Permanent HW addr: ac:16:2d:8a:36:a5
Slave queue ID: 0
Slave Interface: eth5
MII Status: up
Speed: 100 Mbps
Duplex: full
Link Failure Count: 8
Permanent HW addr: ac:16:2d:86:ab:79
Slave queue ID: 0
ifconfig eth1
eth1 Link encap:Ethernet HWaddr AC:16:2D:8A:36:A5
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:144320945 errors:0 dropped:0 overruns:0 frame:0
TX packets:149469477 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:59204967529 (55.1 GiB) TX bytes:83875856927 (78.1 GiB)
Interrupt:36
ifconfig eth5
eth5 Link encap:Ethernet HWaddr AC:16:2D:8A:36:A5
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:1 errors:0 dropped:0 overruns:0 frame:0
TX packets:10 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:64 (64.0 b) TX bytes:640 (640.0 b)
Interrupt:28
ifconfig bond1
bond1 Link encap:Ethernet HWaddr AC:16:2D:8A:36:A5
inet addr:192.168.0.1 Bcast:192.168.0.3 Mask:255.255.255.252
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
RX packets:144347534 errors:0 dropped:0 overruns:0 frame:0
TX packets:149497311 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:59213542863 (55.1 GiB) TX bytes:83893330646 (78.1 GiB)
ethtool -i eth1
driver: tg3
version: 3.132
firmware-version: 5719-v1.38 NCSI v1.2.46.0
bus-info: 0000:03:00.1
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no
ethtool -i eth5
driver: tg3
version: 3.132
firmware-version: 5719-v1.38 NCSI v1.2.46.0
bus-info: 0000:04:00.1
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no
/var/log/messages
Dec 18 03:03:09 kernel: tg3 0000:03:00.1: eth1: Link is down
Dec 18 03:03:09 kernel: bonding: bond1: link status definitely down for interface eth1, disabling it
Dec 18 03:03:09 kernel: bonding: bond1: making interface eth5 the new active one.
Dec 18 03:03:10 kernel: tg3 0000:04:00.1: eth5: Link is down
Dec 18 03:03:10 kernel: bonding: bond1: link status definitely down for interface eth5, disabling it
Dec 18 03:03:10 kernel: bonding: bond1: now running without any active interface !
Dec 18 03:03:11 kernel: tg3 0000:03:00.1: eth1: Link is up at 100 Mbps, full duplex
Dec 18 03:03:11 kernel: tg3 0000:03:00.1: eth1: Flow control is on for TX and on for RX
Dec 18 03:03:11 kernel: tg3 0000:03:00.1: eth1: EEE is disabled
Dec 18 03:03:11 kernel: bond1: link status definitely up for interface eth1, 100 Mbps full duplex.
Dec 18 03:03:11 kernel: bonding: bond1: making interface eth1 the new active one.
Dec 18 03:03:11 kernel: bonding: bond1: first active interface up!
Dec 18 03:03:12 kernel: tg3 0000:04:00.1: eth5: Link is up at 100 Mbps, full duplex
Dec 18 03:03:12 kernel: tg3 0000:04:00.1: eth5: Flow control is on for TX and on for RX
Dec 18 03:03:12 kernel: tg3 0000:04:00.1: eth5: EEE is disabled
Dec 18 03:03:12 kernel: bond1: link status definitely up for interface eth5, 100 Mbps full duplex.
Dec 18 03:03:18 cmanicd[3252]: Entering log_iml_event(slot: 0, port: 2, code: (Other,5))
Dec 18 03:03:18 cmanicd[3252]: Entering get_event_id(slot: 0, port: 2
Dec 18 03:03:21 kernel: tg3 0000:04:00.1: eth5: Link is down
Dec 18 03:03:21 kernel: tg3 0000:03:00.1: eth1: Link is down
Dec 18 03:03:21 kernel: bonding: bond1: link status definitely down for interface eth1, disabling it
Dec 18 03:03:21 kernel: bonding: bond1: now running without any active interface !
Dec 18 03:03:21 kernel: bonding: bond1: link status definitely down for interface eth5, disabling it
Dec 18 03:03:22 kernel: tg3 0000:03:00.1: eth1: Link is up at 100 Mbps, full duplex
Dec 18 03:03:22 kernel: tg3 0000:03:00.1: eth1: Flow control is on for TX and on for RX
Dec 18 03:03:22 kernel: tg3 0000:03:00.1: eth1: EEE is disabled
Dec 18 03:03:23 kernel: tg3 0000:04:00.1: eth5: Link is up at 100 Mbps, full duplex
Dec 18 03:03:23 kernel: tg3 0000:04:00.1: eth5: Flow control is on for TX and on for RX
Dec 18 03:03:23 kernel: tg3 0000:04:00.1: eth5: EEE is disabled
Dec 18 03:03:23 kernel: bond1: link status definitely up for interface eth1, 100 Mbps full duplex.
Dec 18 03:03:23 kernel: bonding: bond1: making interface eth1 the new active one.
Dec 18 03:03:23 kernel: bonding: bond1: first active interface up!
Dec 18 03:03:23 kernel: bond1: link status definitely up for interface eth5, 100 Mbps full duplex.
Dec 18 03:03:24 cmanicd[3252]: Existing event id(0) found for the slot and port.
Dec 18 03:03:24 cmanicd[3252]: Entering read_iml_event(slot: 0, port: 2, eventid: 0)
Dec 18 03:03:24 cmanicd[3252]: Calling ioctl() to read event id: 0)
Dec 18 03:03:24 cmanicd[3252]: Successfully read the event id: 0)
Dec 18 03:03:24 cmanicd[3252]: Trying to modify the existing IML Event.
Dec 18 03:03:24 cmanicd[3252]: Successfully updated the existing IML Event.
Dec 18 03:03:24 cmanicd[3252]: Returning from log_iml_event().
Dec 18 03:03:25 kernel: tg3 0000:03:00.1: eth1: Link is down
Dec 18 03:03:25 kernel: tg3 0000:04:00.1: eth5: Link is down
Dec 18 03:03:25 kernel: bonding: bond1: link status definitely down for interface eth1, disabling it
Dec 18 03:03:25 kernel: bonding: bond1: now running without any active interface !
Dec 18 03:03:25 kernel: bonding: bond1: link status definitely down for interface eth5, disabling it
Dec 18 03:03:26 cmanicd[3252]: Entering log_iml_event(slot: 1, port: 2, code: (Other,5))
Dec 18 03:03:26 cmanicd[3252]: Entering get_event_id(slot: 1, port: 2
Dec 18 03:03:27 cmanicd[3252]: Existing event id(7) found for the slot and port.
Dec 18 03:03:27 cmanicd[3252]: Entering read_iml_event(slot: 1, port: 2, eventid: 7)
Dec 18 03:03:27 cmanicd[3252]: Calling ioctl() to read event id: 7)
Dec 18 03:03:27 cmanicd[3252]: Successfully read the event id: 7)
Dec 18 03:03:27 cmanicd[3252]: Trying to modify the existing IML Event.
Dec 18 03:03:27 cmanicd[3252]: Successfully updated the existing IML Event.
Dec 18 03:03:27 cmanicd[3252]: Returning from log_iml_event().
Dec 18 03:03:27 kernel: tg3 0000:04:00.1: eth5: Link is up at 100 Mbps, full duplex
Dec 18 03:03:27 kernel: tg3 0000:04:00.1: eth5: Flow control is on for TX and on for RX
Dec 18 03:03:27 kernel: tg3 0000:04:00.1: eth5: EEE is disabled
Dec 18 03:03:27 kernel: bond1: link status definitely up for interface eth5, 100 Mbps full duplex.
Dec 18 03:03:27 kernel: bonding: bond1: making interface eth5 the new active one.
Dec 18 03:03:27 kernel: tg3 0000:03:00.1: eth1: Link is up at 100 Mbps, full duplex
Dec 18 03:03:27 kernel: tg3 0000:03:00.1: eth1: Flow control is on for TX and on for RX
Dec 18 03:03:27 kernel: tg3 0000:03:00.1: eth1: EEE is disabled
Dec 18 03:03:27 kernel: bonding: bond1: first active interface up!
Dec 18 03:03:27 kernel: bond1: link status definitely up for interface eth1, 100 Mbps full duplex.
Dec 18 03:03:27 kernel: bonding: bond1: making interface eth1 the new active one.
Dec 18 03:03:28 cmanicd[3252]: Entering get_event_id(slot: 0, port: 2
Dec 18 03:03:28 cmanicd[3252]: Entering read_iml_event(slot: 0, port: 2, eventid: 0)
Dec 18 03:03:28 cmanicd[3252]: Calling ioctl() to read event id: 0)
Dec 18 03:03:28 cmanicd[3252]: Successfully read the event id: 0)
Dec 18 03:03:28 cmanicd[3252]: Entering get_event_id(slot: 1, port: 2
Dec 18 03:03:30 cmanicd[3252]: Entering read_iml_event(slot: 1, port: 2, eventid: 7)
Dec 18 03:03:30 cmanicd[3252]: Calling ioctl() to read event id: 7)
Dec 18 03:03:30 cmanicd[3252]: Successfully read the event id: 7)
Dec 18 03:03:30 cmanicd[3252]: Entering get_event_id(slot: 0, port: 2
Dec 18 03:03:30 cmanicd[3252]: Entering read_iml_event(slot: 0, port: 2, eventid: 0)
Dec 18 03:03:30 cmanicd[3252]: Calling ioctl() to read event id: 0)
Dec 18 03:03:30 cmanicd[3252]: Successfully read the event id: 0)
Dec 18 03:03:30 cmanicd[3252]: Entering iml_log_link_up(slot: 0, port: 2)
Dec 18 03:03:30 cmanicd[3252]: Entering get_event_id(slot: 0, port: 2
Dec 18 03:03:30 cmanicd[3252]: Existing event id(0) found for the slot and port.
Dec 18 03:03:30 cmanicd[3252]: Entering repair_iml_event(slot: 0, port: 2, event: 0)
Dec 18 03:03:30 cmanicd[3252]: Entering read_iml_event(slot: 0, port: 2, eventid: 0)
Dec 18 03:03:30 cmanicd[3252]: Calling ioctl() to read event id: 0)
Dec 18 03:03:30 cmanicd[3252]: Successfully read the event id: 0)
Dec 18 03:03:30 cmanicd[3252]: Trying to repair the existing IML Event.
Dec 18 03:03:31 cmanicd[3252]: Successfully repaired the IML Event.
Dec 18 03:03:31 cmanicd[3252]: Returning from repair_iml_event().
Dec 18 03:03:32 cmanicd[3252]: Entering get_event_id(slot: 1, port: 2
Dec 18 03:03:33 cmanicd[3252]: Entering repair_iml_event(slot: 1, port: 2, event: 7)
Dec 18 03:03:33 cmanicd[3252]: Entering read_iml_event(slot: 1, port: 2, eventid: 7)
Dec 18 03:03:33 cmanicd[3252]: Calling ioctl() to read event id: 7)
Dec 18 03:03:33 cmanicd[3252]: Successfully read the event id: 7)
Dec 18 03:03:33 cmanicd[3252]: Trying to repair the existing IML Event.
Dec 18 03:03:34 cmanicd[3252]: Successfully repaired the IML Event.
Dec 18 03:03:34 cmanicd[3252]: Returning from repair_iml_event().
Dec 18 03:03:35 cmanicd[3252]: Entering log_iml_event(slot: 1, port: 2, code: (Other,5))
Dec 18 03:03:35 cmanicd[3252]: Entering get_event_id(slot: 1, port: 2
Dec 18 03:03:37 cmanicd[3252]: Existing event id(7) found for the slot and port.
Dec 18 03:03:37 cmanicd[3252]: Entering read_iml_event(slot: 1, port: 2, eventid: 7)
Dec 18 03:03:37 cmanicd[3252]: Calling ioctl() to read event id: 7)
Dec 18 03:03:37 cmanicd[3252]: Successfully read the event id: 7)
Dec 18 03:03:37 cmanicd[3252]: Trying to modify the existing IML Event.
Dec 18 03:03:38 cmanicd[3252]: Successfully updated the existing IML Event.
Dec 18 03:03:38 cmanicd[3252]: Returning from log_iml_event().
Dec 18 03:03:39 cmanicd[3252]: Entering log_iml_event(slot: 0, port: 2, code: (Other,5))
Dec 18 03:03:39 cmanicd[3252]: Entering get_event_id(slot: 0, port: 2
Dec 18 03:03:39 cmanicd[3252]: Existing event id(0) found for the slot and port.
Dec 18 03:03:39 cmanicd[3252]: Entering read_iml_event(slot: 0, port: 2, eventid: 0)
Dec 18 03:03:39 cmanicd[3252]: Calling ioctl() to read event id: 0)
Dec 18 03:03:40 cmanicd[3252]: Successfully read the event id: 0)
Dec 18 03:03:40 cmanicd[3252]: Trying to modify the existing IML Event.
Dec 18 03:03:40 cmanicd[3252]: Successfully updated the existing IML Event.
Dec 18 03:03:40 cmanicd[3252]: Returning from log_iml_event().
Dec 18 03:03:41 cmanicd[3252]: Entering get_event_id(slot: 0, port: 2
Dec 18 03:03:42 cmanicd[3252]: Entering repair_iml_event(slot: 0, port: 2, event: 0)
Dec 18 03:03:42 cmanicd[3252]: Entering read_iml_event(slot: 0, port: 2, eventid: 0)
Dec 18 03:03:42 cmanicd[3252]: Calling ioctl() to read event id: 0)
Dec 18 03:03:42 cmanicd[3252]: Successfully read the event id: 0)
Dec 18 03:03:42 cmanicd[3252]: Trying to repair the existing IML Event.
Dec 18 03:03:43 cmanicd[3252]: Successfully repaired the IML Event.
Dec 18 03:03:43 cmanicd[3252]: Returning from repair_iml_event().
Dec 18 03:03:44 cmanicd[3252]: Entering get_event_id(slot: 1, port: 2
Dec 18 03:03:46 cmanicd[3252]: Entering repair_iml_event(slot: 1, port: 2, event: 7)
Dec 18 03:03:46 cmanicd[3252]: Entering read_iml_event(slot: 1, port: 2, eventid: 7)
Dec 18 03:03:46 cmanicd[3252]: Calling ioctl() to read event id: 7)
Dec 18 03:03:46 cmanicd[3252]: Successfully read the event id: 7)
Dec 18 03:03:47 cmanicd[3252]: Trying to repair the existing IML Event.
Dec 18 03:03:47 cmanicd[3252]: Successfully repaired the IML Event.
Dec 18 03:03:47 cmanicd[3252]: Returning from repair_iml_event().
Dec 18 03:03:48 cmanicd[3252]: Entering log_iml_event(slot: 0, port: 2, code: (Other,5))
Dec 18 03:03:48 cmanicd[3252]: Entering get_event_id(slot: 0, port: 2
Dec 18 03:03:49 cmanicd[3252]: Existing event id(0) found for the slot and port.
Dec 18 03:03:49 cmanicd[3252]: Entering read_iml_event(slot: 0, port: 2, eventid: 0)
Dec 18 03:03:49 cmanicd[3252]: Calling ioctl() to read event id: 0)
Dec 18 03:03:49 cmanicd[3252]: Successfully read the event id: 0)
Dec 18 03:03:49 cmanicd[3252]: Trying to modify the existing IML Event.
Dec 18 03:03:49 cmanicd[3252]: Successfully updated the existing IML Event.
Dec 18 03:03:49 cmanicd[3252]: Returning from log_iml_event().
Dec 18 03:03:50 cmanicd[3252]: Entering log_iml_event(slot: 1, port: 2, code: (Other,5))
Dec 18 03:03:50 cmanicd[3252]: Entering get_event_id(slot: 1, port: 2
Dec 18 03:03:51 cmanicd[3252]: Existing event id(7) found for the slot and port.
Dec 18 03:03:51 cmanicd[3252]: Entering read_iml_event(slot: 1, port: 2, eventid: 7)
Dec 18 03:03:51 cmanicd[3252]: Calling ioctl() to read event id: 7)
Dec 18 03:03:51 cmanicd[3252]: Successfully read the event id: 7)
Dec 18 03:03:51 cmanicd[3252]: Trying to modify the existing IML Event.
Dec 18 03:03:51 cmanicd[3252]: Successfully updated the existing IML Event.
Dec 18 03:03:51 cmanicd[3252]: Returning from log_iml_event().
Dec 18 03:03:53 cmanicd[3252]: Entering get_event_id(slot: 1, port: 2
Dec 18 03:03:54 cmanicd[3252]: Entering repair_iml_event(slot: 1, port: 2, event: 7)
Dec 18 03:03:54 cmanicd[3252]: Entering read_iml_event(slot: 1, port: 2, eventid: 7)
Dec 18 03:03:54 cmanicd[3252]: Calling ioctl() to read event id: 7)
Dec 18 03:03:54 cmanicd[3252]: Successfully read the event id: 7)
Dec 18 03:03:54 cmanicd[3252]: Trying to repair the existing IML Event.
Dec 18 03:03:54 cmanicd[3252]: Successfully repaired the IML Event.
Dec 18 03:03:54 cmanicd[3252]: Returning from repair_iml_event().
Dec 18 03:03:55 cmanicd[3252]: Entering get_event_id(slot: 0, port: 2
Dec 18 03:03:59 cmanicd[3252]: Entering repair_iml_event(slot: 0, port: 2, event: 0)
Dec 18 03:03:59 cmanicd[3252]: Entering read_iml_event(slot: 0, port: 2, eventid: 0)
Dec 18 03:03:59 cmanicd[3252]: Calling ioctl() to read event id: 0)
Dec 18 03:04:01 cmanicd[3252]: Successfully read the event id: 0)
Dec 18 03:04:01 cmanicd[3252]: Trying to repair the existing IML Event.
Dec 18 03:04:06 cmanicd[3252]: Successfully repaired the IML Event.
Dec 18 03:04:06 cmanicd[3252]: Returning from repair_iml_event().
Dec 18 03:06:35 kernel: tg3 0000:03:00.1: eth1: Link is down
Dec 18 03:06:35 kernel: bonding: bond1: link status definitely down for interface eth1, disabling it
Dec 18 03:06:35 kernel: bonding: bond1: making interface eth5 the new active one.
Dec 18 03:06:36 kernel: tg3 0000:04:00.1: eth5: Link is down
Dec 18 03:06:36 kernel: bonding: bond1: link status definitely down for interface eth5, disabling it
Dec 18 03:06:36 kernel: bonding: bond1: now running without any active interface !
Dec 18 03:06:37 cmanicd[3252]: Entering iml_log_link_down(slot: 0, port: 2)
Dec 18 03:06:37 cmanicd[3252]: Entering log_iml_event(slot: 0, port: 2, code: (Down,2))
Dec 18 03:06:37 cmanicd[3252]: Entering get_event_id(slot: 0, port: 2
Dec 18 03:06:37 cmanicd[3252]: Existing event id(0) found for the slot and port.
Dec 18 03:06:37 cmanicd[3252]: Entering read_iml_event(slot: 0, port: 2, eventid: 0)
Dec 18 03:06:37 cmanicd[3252]: Calling ioctl() to read event id: 0)
Dec 18 03:06:37 cmanicd[3252]: Successfully read the event id: 0)
Dec 18 03:06:37 cmanicd[3252]: Trying to modify the existing IML Event.
Dec 18 03:06:38 cmanicd[3252]: Successfully updated the existing IML Event.
Dec 18 03:06:38 cmanicd[3252]: Returning from log_iml_event().
Dec 18 03:06:39 cmanicd[3252]: Entering iml_log_link_down(slot: 1, port: 2)
Dec 18 03:06:39 cmanicd[3252]: Entering log_iml_event(slot: 1, port: 2, code: (Down,2))
Dec 18 03:06:39 cmanicd[3252]: Entering get_event_id(slot: 1, port: 2
Dec 18 03:06:40 cmanicd[3252]: Existing event id(7) found for the slot and port.
Dec 18 03:06:40 cmanicd[3252]: Entering read_iml_event(slot: 1, port: 2, eventid: 7)
Dec 18 03:06:40 cmanicd[3252]: Calling ioctl() to read event id: 7)
Dec 18 03:06:40 cmanicd[3252]: Successfully read the event id: 7)
Dec 18 03:06:40 cmanicd[3252]: Trying to modify the existing IML Event.
Dec 18 03:06:40 cmanicd[3252]: Successfully updated the existing IML Event.
Dec 18 03:06:40 cmanicd[3252]: Returning from log_iml_event().
Dec 18 03:19:22 kernel: tg3 0000:03:00.1: eth1: Link is up at 100 Mbps, full duplex
Dec 18 03:19:22 kernel: tg3 0000:03:00.1: eth1: Flow control is on for TX and on for RX
Dec 18 03:19:22 kernel: tg3 0000:03:00.1: eth1: EEE is disabled
Dec 18 03:19:22 kernel: bond1: link status definitely up for interface eth1, 100 Mbps full duplex.
Dec 18 03:19:22 kernel: bonding: bond1: making interface eth1 the new active one.
Dec 18 03:19:22 kernel: bonding: bond1: first active interface up!
Dec 18 03:19:22 kernel: tg3 0000:04:00.1: eth5: Link is up at 100 Mbps, full duplex
Dec 18 03:19:22 kernel: tg3 0000:04:00.1: eth5: Flow control is on for TX and on for RX
Dec 18 03:19:22 kernel: tg3 0000:04:00.1: eth5: EEE is disabled
Dec 18 03:19:22 kernel: bond1: link status definitely up for interface eth5, 100 Mbps full duplex.
Dec 18 03:19:25 kernel: tg3 0000:03:00.1: eth1: Link is down
Dec 18 03:19:25 kernel: bonding: bond1: link status definitely down for interface eth1, disabling it
Dec 18 03:19:25 kernel: bonding: bond1: making interface eth5 the new active one.
Dec 18 03:19:26 kernel: tg3 0000:04:00.1: eth5: Link is down
Dec 18 03:19:26 kernel: bonding: bond1: link status definitely down for interface eth5, disabling it
Dec 18 03:19:26 kernel: bonding: bond1: now running without any active interface !
To come out from this situation(at Dec 18 03:03), ifdown and ifup were performed. Please suggest what may be done here to resolve the issue.