iptables setup for two different lxc containers inside the same host

Hello out there. A month ago I started to deal with this problem and until now I couldn't cope with it. The quick story is that I'm trying to setup two different lxc containers inside the same host machine running debian linux. One of the containers is running a VPN server, while the second one is running a WEB server. Both of them are connected to the internet through a bridge (br0) interface on the host. In order to route the traffic between these two containers I used iptables.

Let's now see this procedure more closely and step by step.

SETUP THE FIRST CONTAINER (VPN)

@HOST [ /etc/network/interfaces ]

# interfaces(5) file used by ifup(8) and ifdown(8)
# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet dhcp

# Bridge interface
auto br0
iface br0 inet static
address 192.168.1.1
netmask 255.255.255.0
broadcast 192.168.1.255
bridge_ports none
bridge_fd 2.0
bridge_maxwait 1

@HOST [ /var/lib/lxc/VPN/config ]

# Template used to create this container: /usr/share/lxc/templates/lxc-debian
# Parameters passed to the template: -r jessie
# For additional config options, please look at lxc.container.conf(5)
#lxc.network.type = empty
lxc.rootfs = /var/lib/lxc/VPN/rootfs

# Common configuration
lxc.include = /usr/share/lxc/config/debian.common.conf

# Container specific configuration
lxc.mount = /var/lib/lxc/VPN/fstab
lxc.utsname = VPN
lxc.arch = amd64
lxc.autodev = 1
lxc.kmsg = 0
lxc.start.auto = 1

lxc.network.type = veth
lxc.network.veth.pair = vethVPN
lxc.network.name = veth0
lxc.network.flags = up
lxc.network.hwaddr = 00:16:3e:42:1d:a7
lxc.network.link = br0

## for openvpn
lxc.mount.entry = /dev/net dev/net none bind,create=dir
lxc.cgroup.devices.allow = c 10:200 rwm

@GUEST VPN [ /etc/network/interfaces ]

auto lo
iface lo inet loopback

#auto eth0
#iface eth0 inet dhcp

auto veth0
iface veth0 inet static
address 192.168.1.2
netmask 255.255.255.0
gateway 192.168.1.1
dns-nameservers 192.168.1.1

After some more configuration inside the container for the vpn server (not interesting for this post) i add the following iptables rules in the HOST machine:

# Generated by iptables-save v1.4.21 on Fri Apr 28 16:07:58 2017
*filter
:INPUT ACCEPT [1189211:150089991]
:FORWARD ACCEPT [902865:826112449]
:OUTPUT ACCEPT [1324099:212970374]
COMMIT
# Completed on Fri Apr 28 16:07:58 2017
# Generated by iptables-save v1.4.21 on Fri Apr 28 16:07:58 2017
*nat
:PREROUTING ACCEPT [36:1998]
:INPUT ACCEPT [17:858]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
-A PREROUTING -p udp -m udp --dport 1194 -j DNAT --to-destination 192.168.1.2:1194
-A POSTROUTING -o eth0 -j MASQUERADE
COMMIT

Until that step everything is working as expected. VPN container can ping the outside world, can apt-get update correctly and VPN clients find their way to the outside world as expected.

The next step was to add another container for the WEB server.

@HOST [ /var/lib/lxc/WEB/config ]

# Template used to create this container: /usr/share/lxc/templates/lxc-debian
# Parameters passed to the template: -r jessie
# For additional config options, please look at lxc.container.conf(5)
#lxc.network.type = empty
lxc.rootfs = /var/lib/lxc/WEB/rootfs

# Common configuration
lxc.include = /usr/share/lxc/config/debian.common.conf

# Container specific configuration
lxc.mount = /var/lib/lxc/WEB/fstab
lxc.utsname = WEB
lxc.arch = amd64
lxc.autodev = 1
lxc.kmsg = 0
lxc.start.auto = 0

# Network config
lxc.network.type = veth
lxc.network.veth.pair = vethWEB
lxc.network.name = veth0
lxc.network.flags = up
lxc.network.hwaddr = 00:14:2e:42:1d:a7
lxc.network.link = br0

## for openvpn
lxc.mount.entry = /dev/net dev/net none bind,create=dir
lxc.cgroup.devices.allow = c 10:200 rwm

@GUEST [ /etc/network/interfaces ]

auto lo
iface lo inet loopback

auto veth0
iface veth0 inet static
address 192.168.1.4
netmask 255.255.255.0
gateway 192.168.1.1
dns-nameservers 192.168.1.1

For http traffic to be routed in the WEB container I add the following iptables rule

iptables -t nat -A PREROUTING -p tcp -m tcp --dport 80 -j DNAT --to-destination 192.168.1.4:80

and thus now the final rules are :

# Generated by iptables-save v1.4.21 on Wed May 17 08:13:33 2017
*nat
:PREROUTING ACCEPT [5132:301425]
:INPUT ACCEPT [5124:300824]
:OUTPUT ACCEPT [95:6546]
:POSTROUTING ACCEPT [55:3052]
-A PREROUTING -p udp -m udp --dport 1194 -j DNAT --to-destination 192.168.1.2:1194
-A PREROUTING -p tcp -m tcp --dport 80 -j DNAT --to-destination 192.168.1.4:80
-A POSTROUTING -o eth0 -j MASQUERADE
COMMIT
# Completed on Wed May 17 08:13:33 2017
# Generated by iptables-save v1.4.21 on Wed May 17 08:13:33 2017
*filter
:INPUT ACCEPT [1227003:153631244]
:FORWARD ACCEPT [3344156:3204894200]
:OUTPUT ACCEPT [1377802:229812203]
COMMIT

This is where problems start.

  1. containers can ping the outside world
PING www.google.com (216.58.208.228) 56(84) bytes of data.
64 bytes from par10s22-in-f228.1e100.net (216.58.208.228): icmp_seq=1 ttl=52 time=12.4 ms
  1. containers can not apt-get update
Err http://http.debian.net jessie InRelease         
  
Err http://http.debian.net jessie Release.gpg       
  Cannot initiate the connection to http.debian.net:80 (2605:bc80:3010:b00:0:deb:166:202). - connect (101: Network is unreachable) [IP: 2605:bc80:3010:b00:0:deb:166:202 80]
Reading package lists... Done
W: Failed to fetch http://http.debian.net/debian/dists/jessie/InRelease  

W: Failed to fetch http://http.debian.net/debian/dists/jessie/Release.gpg  Cannot initiate the connection to http.debian.net:80 (2605:bc80:3010:b00:0:deb:166:202). - connect (101: Network is unreachable) [IP: 2605:bc80:3010:b00:0:deb:166:202 80]

W: Some index files failed to download. They have been ignored, or old ones used instead.
  1. vpn clients doesn't access internet properly. Some web sites doesn't load at all while others work perfectly.

It seems that there is a conflict in the http protocol traffic. If I delete the later iptables rule (for the WEB container) , container regain the ability for the apt-get update and vpn clients can access all the web sites. In that way the drawback is that I cannot access the web server from the outside world.

Finally some network info:

ifconfig @ HOST

br0       Link encap:Ethernet  HWaddr fe:cd:03:40:b8:ca  
          inet addr:192.168.1.1  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::2024:18ff:febf:2d13/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1939159 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1432506 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:1631320323 (1.5 GiB)  TX bytes:1595149625 (1.4 GiB)

eth0      Link encap:Ethernet  HWaddr de:2b:44:3f:a0:03  
          inet addr:10.8.44.199  Bcast:10.255.255.255  Mask:255.255.255.254
          inet6 addr: 2001:bc8:4700:2300::9:1107/127 Scope:Global
          inet6 addr: fe80::dc2b:44ff:fe3f:a003/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:2694541 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3348824 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:1823342534 (1.6 GiB)  TX bytes:1909806405 (1.7 GiB)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:3 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1 
          RX bytes:204 (204.0 B)  TX bytes:204 (204.0 B)

vethVPN   Link encap:Ethernet  HWaddr fe:cd:03:40:b8:ca  
          inet6 addr: fe80::fccd:3ff:fe40:b8ca/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1932533 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1423250 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:1657633998 (1.5 GiB)  TX bytes:1582663302 (1.4 GiB)

vethWEB   Link encap:Ethernet  HWaddr fe:d2:6e:95:19:46  
          inet6 addr: fe80::fcd2:6eff:fe95:1946/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:937 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1273 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:135281 (132.1 KiB)  TX bytes:103741 (101.3 KiB)

route -n @ HOST

Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.8.44.198     0.0.0.0         UG    0      0        0 eth0
10.8.44.198     0.0.0.0         255.255.255.254 U     0      0        0 eth0
192.168.1.0     0.0.0.0         255.255.255.0   U     0      0        0 br0

brctl show @ HOST

bridge name	bridge id		STP enabled	interfaces
br0		8000.fecd0340b8ca	no		vethVPN
							vethWEB

Any idea/hint on how to fix this routing problem will be very thankful because I cannot think anything else to try and my mind is going to burn out.

Thank you.

P.S Something that I just realized is that the error from the apt-get update command show that it tries to communicate with ipv6 protocol and not ipv4. Is that weird ?