Packet going out of wrong interface due to OS automatically added cache route with lower metric

msr1981 · October 9, 2017, 7:34am

RHEL 7.0, IPV6

Scenario:
I have routed specific network using network scripts.

"ip -6 route show" shows that route has been added. ( with metric 1024)
Ping of the specific IP through that route is successful.
Now after few days, for some reason, we see that cache route appears for that given destination IP with lower metric ( which is actually trying to take it through interface/bond of the default route.)Ping to that destination IP fails then because packet is going out of wrong interface.
If I delete that cache route, again the packet starts going out of the correct interface ( which is not interface of the default route) and ping is successful.

Has anyone faced similar issue with REHL. I am using RHEL 7
Questions:
Q1. In what situation OS will try to route through default, even if static route for specific IP/network exists.
Q2. How to configure the route so that OS does not fall back on default route if specific interface is assigned for a specific ip/network by static configuration. Please note, we can not delete the default route.
Q3. Even if OS is creating cache route for some temporary situation, why is it not detecting that situation has normalized and why it is not deleting the automatically created cache route then. ( assuming that some temporary network side failure caused this issue )

Please note, we are not facing this issue for IPV4. We are facing this with IPV6
Can anyone suggest on how to permanently resolve this situation ?

rbatte1 · October 9, 2017, 10:03am

So to save us from guessing and trying to interpret your description only, can you paste the output from your various network display commands so we can actually see the configuration. It will be far easier to work it out from that.

Thanks, in advance,
Robin

msr1981 · October 10, 2017, 4:02am

Please see below output of ip -6 route show.

default route is

default via 2405:200:370:237::1 dev bond1  metric 1

specific route for network that was added:

2405:200:370:10::/64 via 2405:200:370:236::1 dev bond0.236  metric 1024

cache route added automatically by OS with metric 0 was as below.

2405:200:370:10::20 via 2405:200:370:237::1 dev bond1  metric 0  cache

This cache route caused packet to go out of default route instead of the specific route. Destination IP is not reachable from the default route. cache route had to be deleted manually for normal traffic to resume.
--------------------------------------------------------------------------

ip -6 route show 
unreachable ::/96 dev lo  metric 1024  error -101
unreachable ::ffff:0.0.0.0/96 dev lo  metric 1024  error -101
unreachable 2002:a00::/24 dev lo  metric 1024  error -101
unreachable 2002:7f00::/24 dev lo  metric 1024  error -101
unreachable 2002:a9fe::/32 dev lo  metric 1024  error -101
unreachable 2002:ac10::/28 dev lo  metric 1024  error -101
unreachable 2002:c0a8::/32 dev lo  metric 1024  error -101
unreachable 2002:e000::/19 dev lo  metric 1024  error -101
2405:200:201:400:172:16:21:156 via 2405:200:370:243::1 dev bond3.243  metric 1024 
2405:200:201:400:172:16:21:157 via 2405:200:370:243::1 dev bond3.243  metric 1024 
2405:200:310:15::2a via 2405:200:370:240::1 dev bond2.240  metric 0  cache 
2405:200:310:15::56 via 2405:200:370:240::1 dev bond2.240  metric 0  cache 
2405:200:310:15::/64 via 2405:200:370:240::1 dev bond2.240  metric 1024 
2405:200:311:230::11 via 2405:200:370:230::1 dev bond3.230  metric 0  cache 
2405:200:311:230::/64 via 2405:200:370:230::1 dev bond3.230  metric 1024 
2405:200:311:247::11 via 2405:200:370:243::1 dev bond3.243  metric 0  cache 
2405:200:311:247::12 via 2405:200:370:243::1 dev bond3.243  metric 0  cache 
2405:200:311:247::13 via 2405:200:370:243::1 dev bond3.243  metric 0  cache 
2405:200:311:247::/64 via 2405:200:370:243::1 dev bond3.243  metric 1024 
2405:200:314:170::/64 via 2405:200:370:236::1 dev bond0.236  metric 1024 
2405:200:360:10::/64 via 2405:200:370:236::1 dev bond0.236  metric 1024 
2405:200:370:10::20 via 2405:200:370:237::1 dev bond1  metric 0  cache 
2405:200:370:10::/64 via 2405:200:370:236::1 dev bond0.236  metric 1024 
2405:200:370:16::2a via 2405:200:370:240::1 dev bond2.240  metric 0  cache 
2405:200:370:16::56 via 2405:200:370:240::1 dev bond2.240  metric 0  cache 
2405:200:370:16::/64 via 2405:200:370:240::1 dev bond2.240  metric 1024 
2405:200:370:228::/64 dev bond3.228  proto kernel  metric 256 
2405:200:370:230::/64 dev bond3.230  proto kernel  metric 256 
2405:200:370:231::/64 dev bond4  proto kernel  metric 256 
2405:200:370:234::/64 dev bond3.234  proto kernel  metric 256 
2405:200:370:236::/64 dev bond0.236  proto kernel  metric 256 
2405:200:370:237::/64 dev bond1  proto kernel  metric 256 
2405:200:370:238::/64 dev bond0.238  proto kernel  metric 256 
2405:200:370:239::/64 dev bond0.239  proto kernel  metric 256 
2405:200:370:240::/64 dev bond2.240  proto kernel  metric 256 
2405:200:370:242::/64 dev bond3.242  proto kernel  metric 256 
2405:200:370:243::13 dev bond3.243  metric 0  cache 
2405:200:370:243::/64 dev bond3.243  proto kernel  metric 256 
2405:200:370:244::/64 dev bond2.244  proto kernel  metric 256 
2405:200:a10:fc82::/64 via 2405:200:370:243::1 dev bond3.243  metric 1024 
unreachable 3ffe:ffff::/32 dev lo  metric 1024  error -101
fe80::/64 dev bond0  proto kernel  metric 256 
fe80::/64 dev bond1  proto kernel  metric 256 
fe80::/64 dev bond2  proto kernel  metric 256 
fe80::/64 dev bond3  proto kernel  metric 256 
fe80::/64 dev bond4  proto kernel  metric 256 
fe80::/64 dev eno4  proto kernel  metric 256 
fe80::/64 dev ens1f3  proto kernel  metric 256 
fe80::/64 dev bond0.236  proto kernel  metric 256 
fe80::/64 dev bond0.238  proto kernel  metric 256 
fe80::/64 dev bond0.239  proto kernel  metric 256 
fe80::/64 dev bond2.240  proto kernel  metric 256 
fe80::/64 dev bond2.244  proto kernel  metric 256 
fe80::/64 dev bond3.228  proto kernel  metric 256 
fe80::/64 dev bond3.230  proto kernel  metric 256 
fe80::/64 dev bond3.234  proto kernel  metric 256 
fe80::/64 dev bond3.242  proto kernel  metric 256 
fe80::/64 dev bond3.243  proto kernel  metric 256 
fe80::/64 dev ens6f3  proto kernel  metric 256 
default via 2405:200:370:237::1 dev bond1  metric 1

----------------------------------------------------------
Configuration for the bond is as follows.

 /etc/sysconfig/network-scripts ] cat route6-bond0.236
  2405:200:370:10::/64 via 2405:200:370:236::1 dev bond0.236
  2405:200:314:170::/64 via 2405:200:370:236::1 dev bond0.236
  2405:200:360:10::/64 via 2405:200:370:236::1 dev bond0.236

 /etc/sysconfig/network-scripts ] cat ifcfg-bond0.236 
  DEVICE=bond0.236
  BOOTPROTO=none
  ONBOOT=yes
  VLAN=yes
  IPV6INIT=yes
  IPV6_AUTOCONF=no
  IPV6ADDR=2405:200:370:236::12/64

 /etc/sysconfig/network-scripts ] cat route6-bond1 
   via 2405:200:370:237::1 dev bond1

  /etc/sysconfig/network-scripts ] cat ifcfg-bond1 
  DEVICE=bond1
  BONDING_MASTER=yes
  BONDING_OPTS="miimon=100 mode=1"
  BOOTPROTO=none
  ONBOOT=yes
  IPADDR=10.71.167.102
  NETMASK=255.255.255.240
  IPV6INIT=yes
  IPV6_AUTOCONF=no
  IPV6ADDR=2405:200:370:237::12/64
  IPV6_DEFAULTGW=2405:200:370:237::1

msr1981 · October 23, 2017, 4:54am

Friends,

No one has observed similar problem in RHEL7 for IPV6 ?
No guesses as to why this problem happened ? Let me know if more information is required.

Thanks in anticipation