Hello out there. A month ago I started to deal with this problem and until now I couldn't cope with it. The quick story is that I'm trying to setup two different lxc containers inside the same host machine running debian linux. One of the containers is running a VPN server, while the second one is running a WEB server. Both of them are connected to the internet through a bridge (br0) interface on the host. In order to route the traffic between these two containers I used iptables.
Let's now see this procedure more closely and step by step.
SETUP THE FIRST CONTAINER (VPN)
@HOST [ /etc/network/interfaces ]
# interfaces(5) file used by ifup(8) and ifdown(8)
# The loopback network interface
auto lo
iface lo inet loopback
# The primary network interface
auto eth0
iface eth0 inet dhcp
# Bridge interface
auto br0
iface br0 inet static
address 192.168.1.1
netmask 255.255.255.0
broadcast 192.168.1.255
bridge_ports none
bridge_fd 2.0
bridge_maxwait 1
@HOST [ /var/lib/lxc/VPN/config ]
# Template used to create this container: /usr/share/lxc/templates/lxc-debian
# Parameters passed to the template: -r jessie
# For additional config options, please look at lxc.container.conf(5)
#lxc.network.type = empty
lxc.rootfs = /var/lib/lxc/VPN/rootfs
# Common configuration
lxc.include = /usr/share/lxc/config/debian.common.conf
# Container specific configuration
lxc.mount = /var/lib/lxc/VPN/fstab
lxc.utsname = VPN
lxc.arch = amd64
lxc.autodev = 1
lxc.kmsg = 0
lxc.start.auto = 1
lxc.network.type = veth
lxc.network.veth.pair = vethVPN
lxc.network.name = veth0
lxc.network.flags = up
lxc.network.hwaddr = 00:16:3e:42:1d:a7
lxc.network.link = br0
## for openvpn
lxc.mount.entry = /dev/net dev/net none bind,create=dir
lxc.cgroup.devices.allow = c 10:200 rwm
@GUEST VPN [ /etc/network/interfaces ]
auto lo
iface lo inet loopback
#auto eth0
#iface eth0 inet dhcp
auto veth0
iface veth0 inet static
address 192.168.1.2
netmask 255.255.255.0
gateway 192.168.1.1
dns-nameservers 192.168.1.1
After some more configuration inside the container for the vpn server (not interesting for this post) i add the following iptables rules in the HOST machine:
# Generated by iptables-save v1.4.21 on Fri Apr 28 16:07:58 2017
*filter
:INPUT ACCEPT [1189211:150089991]
:FORWARD ACCEPT [902865:826112449]
:OUTPUT ACCEPT [1324099:212970374]
COMMIT
# Completed on Fri Apr 28 16:07:58 2017
# Generated by iptables-save v1.4.21 on Fri Apr 28 16:07:58 2017
*nat
:PREROUTING ACCEPT [36:1998]
:INPUT ACCEPT [17:858]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
-A PREROUTING -p udp -m udp --dport 1194 -j DNAT --to-destination 192.168.1.2:1194
-A POSTROUTING -o eth0 -j MASQUERADE
COMMIT
Until that step everything is working as expected. VPN container can ping the outside world, can apt-get update correctly and VPN clients find their way to the outside world as expected.
The next step was to add another container for the WEB server.
@HOST [ /var/lib/lxc/WEB/config ]
# Template used to create this container: /usr/share/lxc/templates/lxc-debian
# Parameters passed to the template: -r jessie
# For additional config options, please look at lxc.container.conf(5)
#lxc.network.type = empty
lxc.rootfs = /var/lib/lxc/WEB/rootfs
# Common configuration
lxc.include = /usr/share/lxc/config/debian.common.conf
# Container specific configuration
lxc.mount = /var/lib/lxc/WEB/fstab
lxc.utsname = WEB
lxc.arch = amd64
lxc.autodev = 1
lxc.kmsg = 0
lxc.start.auto = 0
# Network config
lxc.network.type = veth
lxc.network.veth.pair = vethWEB
lxc.network.name = veth0
lxc.network.flags = up
lxc.network.hwaddr = 00:14:2e:42:1d:a7
lxc.network.link = br0
## for openvpn
lxc.mount.entry = /dev/net dev/net none bind,create=dir
lxc.cgroup.devices.allow = c 10:200 rwm
@GUEST [ /etc/network/interfaces ]
auto lo
iface lo inet loopback
auto veth0
iface veth0 inet static
address 192.168.1.4
netmask 255.255.255.0
gateway 192.168.1.1
dns-nameservers 192.168.1.1
For http traffic to be routed in the WEB container I add the following iptables rule
iptables -t nat -A PREROUTING -p tcp -m tcp --dport 80 -j DNAT --to-destination 192.168.1.4:80
and thus now the final rules are :
# Generated by iptables-save v1.4.21 on Wed May 17 08:13:33 2017
*nat
:PREROUTING ACCEPT [5132:301425]
:INPUT ACCEPT [5124:300824]
:OUTPUT ACCEPT [95:6546]
:POSTROUTING ACCEPT [55:3052]
-A PREROUTING -p udp -m udp --dport 1194 -j DNAT --to-destination 192.168.1.2:1194
-A PREROUTING -p tcp -m tcp --dport 80 -j DNAT --to-destination 192.168.1.4:80
-A POSTROUTING -o eth0 -j MASQUERADE
COMMIT
# Completed on Wed May 17 08:13:33 2017
# Generated by iptables-save v1.4.21 on Wed May 17 08:13:33 2017
*filter
:INPUT ACCEPT [1227003:153631244]
:FORWARD ACCEPT [3344156:3204894200]
:OUTPUT ACCEPT [1377802:229812203]
COMMIT
This is where problems start.
- containers can ping the outside world
PING www.google.com (216.58.208.228) 56(84) bytes of data.
64 bytes from par10s22-in-f228.1e100.net (216.58.208.228): icmp_seq=1 ttl=52 time=12.4 ms
- containers can not apt-get update
Err http://http.debian.net jessie InRelease
Err http://http.debian.net jessie Release.gpg
Cannot initiate the connection to http.debian.net:80 (2605:bc80:3010:b00:0:deb:166:202). - connect (101: Network is unreachable) [IP: 2605:bc80:3010:b00:0:deb:166:202 80]
Reading package lists... Done
W: Failed to fetch http://http.debian.net/debian/dists/jessie/InRelease
W: Failed to fetch http://http.debian.net/debian/dists/jessie/Release.gpg Cannot initiate the connection to http.debian.net:80 (2605:bc80:3010:b00:0:deb:166:202). - connect (101: Network is unreachable) [IP: 2605:bc80:3010:b00:0:deb:166:202 80]
W: Some index files failed to download. They have been ignored, or old ones used instead.
- vpn clients doesn't access internet properly. Some web sites doesn't load at all while others work perfectly.
It seems that there is a conflict in the http protocol traffic. If I delete the later iptables rule (for the WEB container) , container regain the ability for the apt-get update and vpn clients can access all the web sites. In that way the drawback is that I cannot access the web server from the outside world.
Finally some network info:
ifconfig @ HOST
br0 Link encap:Ethernet HWaddr fe:cd:03:40:b8:ca
inet addr:192.168.1.1 Bcast:192.168.1.255 Mask:255.255.255.0
inet6 addr: fe80::2024:18ff:febf:2d13/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:1939159 errors:0 dropped:0 overruns:0 frame:0
TX packets:1432506 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1631320323 (1.5 GiB) TX bytes:1595149625 (1.4 GiB)
eth0 Link encap:Ethernet HWaddr de:2b:44:3f:a0:03
inet addr:10.8.44.199 Bcast:10.255.255.255 Mask:255.255.255.254
inet6 addr: 2001:bc8:4700:2300::9:1107/127 Scope:Global
inet6 addr: fe80::dc2b:44ff:fe3f:a003/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:2694541 errors:0 dropped:0 overruns:0 frame:0
TX packets:3348824 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1823342534 (1.6 GiB) TX bytes:1909806405 (1.7 GiB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:3 errors:0 dropped:0 overruns:0 frame:0
TX packets:3 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1
RX bytes:204 (204.0 B) TX bytes:204 (204.0 B)
vethVPN Link encap:Ethernet HWaddr fe:cd:03:40:b8:ca
inet6 addr: fe80::fccd:3ff:fe40:b8ca/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:1932533 errors:0 dropped:0 overruns:0 frame:0
TX packets:1423250 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1657633998 (1.5 GiB) TX bytes:1582663302 (1.4 GiB)
vethWEB Link encap:Ethernet HWaddr fe:d2:6e:95:19:46
inet6 addr: fe80::fcd2:6eff:fe95:1946/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:937 errors:0 dropped:0 overruns:0 frame:0
TX packets:1273 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:135281 (132.1 KiB) TX bytes:103741 (101.3 KiB)
route -n @ HOST
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.8.44.198 0.0.0.0 UG 0 0 0 eth0
10.8.44.198 0.0.0.0 255.255.255.254 U 0 0 0 eth0
192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0 br0
brctl show @ HOST
bridge name bridge id STP enabled interfaces
br0 8000.fecd0340b8ca no vethVPN
vethWEB
Any idea/hint on how to fix this routing problem will be very thankful because I cannot think anything else to try and my mind is going to burn out.
Thank you.
P.S Something that I just realized is that the error from the apt-get update command show that it tries to communicate with ipv6 protocol and not ipv4. Is that weird ?