Score:2

How to bridge TAP device to overlay network?

pe flag

I would like to allow OpenVPN users to reach all Docker containers that are part of Overlay network as if they were in the same LAN. The simplest example is 3 containers distributed on 2 hosts, and the hosts are connected through overlay network, and all 3 containers are part of that overlay network.

For that, I have prepared one OpenVPN container (OpenVPN in bridge mode with TAP adapter), and in that container I created a bridge br0 and bridged the eth0 and tap0 interfaces to br0. By this I would say that I bridged my TAP and all OpenVPN clients to the overlay network (I am not sure if this statement is correct).

So far, when my OpenVPN users connects to OpenVPN, and try to ping Dummy Container 1 (which is on the same host with OpenVPN container), the users are able to ping the Dummy Container 1 (and of course are able to ping OpenVPN container also). So far so good!

However, the users are not able to ping the Dummy Container 2, which exists on a different host (but is part of the same overlay with OpenVPN container as mentioned above).

So in short, the VPN users that are connected to OpenVPN container (bridged to Docker overlay network) and try to ping the other containers can only ping the containers on the same host, not on the other host, as if bridging work only on local bridge and not with Overlay network.

It is important also to tell, that from inside OpenVPN container itself I can ping the Dummy Container 2 (and of course Dummy Container 1 as well).

This figure samarize the whole story I told above. Notice the red arrow that represents the connection that I would like to achieve but does not work at the moment (VPN user connects to Host 1 via OpenVPN then try to ping Dummy Container 2).

enter image description here

Some detailed technical information

OpenVPN container
/etc/openvpn # ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: tap0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc fq_codel master br0 state UP group default qlen 1000
    link/ether 06:9b:33:89:85:81 brd ff:ff:ff:ff:ff:ff
3: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
    link/ether 02:42:0a:08:00:04 brd ff:ff:ff:ff:ff:ff
    inet 10.8.0.4/24 scope global br0
       valid_lft forever preferred_lft forever
74: eth0@if75: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master br0 state UP group default
    link/ether 02:42:0a:08:00:04 brd ff:ff:ff:ff:ff:ff link-netnsid 0
76: eth1@if77: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether 02:42:ac:12:00:04 brd ff:ff:ff:ff:ff:ff link-netnsid 1
    inet 172.18.0.4/16 brd 172.18.255.255 scope global eth1
       valid_lft forever preferred_lft forever
/etc/openvpn # ip route
default via 172.18.0.1 dev eth1
10.8.0.0/24 dev br0 proto kernel scope link src 10.8.0.4
172.18.0.0/16 dev eth1 proto kernel scope link src 172.18.0.4

OpenVPN server config:

port 1194
proto udp
dev tap0
ca ca.crt
cert server.crt
key server.key # This file should be kept secret
dh dh.pem
ifconfig-pool-persist ipp.txt
client-to-client
keepalive 10 120
cipher AES-256-GCM
compress lz4-v2
push "compress lz4-v2"
persist-key
persist-tun
status /var/log/openvpn-status.log
verb 3
explicit-exit-notify 1

##### Authentication #####
# A client certificate is not required. The client need to authenticate using username/password only.
verify-client-cert none
# Disables keys renegotiation. The default value is one hour, which disconnects the user every hour and causes the end user to be challenged to reauthorize using a new OTP.
reneg-sec 0
# Loads and uses LDAP plugin config.
plugin /usr/lib/openvpn/plugins/openvpn-auth-ldap.so /etc/openvpn/auth-ldap.conf
# Loads and uses PAM-auth plugin (mainly for 2FA).
plugin /usr/lib/openvpn/plugins/openvpn-plugin-auth-pam.so "openvpn login USERNAME password PASSWORD pin OTP"

Briding is done like this:

openvpn --mktun --dev tap0
ip link set tap0 promisc on up

ip link add name br0 type bridge
ip link set dev br0 up
ip link set dev tap0 master br0
ip link set dev eth0 master br0
ip addr flush dev eth0
ip addr add 10.8.0.4/24 dev br0

OpenVPN is started using the command

VPN_SERVER_HOST_IP_ADDRESS=10.8.0.254
VPN_NETMASK=255.255.255.0
VPN_POOL_START_IP_ADDRESS=10.8.0.50
VPN_POOL_END_IP_ADDRESS=10.8.0.100

openvpn \
    --config server.conf \
    --server-bridge $VPN_SERVER_HOST_IP_ADDRESS $VPN_NETMASK $VPN_POOL_START_IP_ADDRESS $VPN_POOL_END_IP_ADDRESS

My Dummy Container 2 (very similar to Dummy Container 1 but with different IPs):

/ # ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
35: eth0@if36: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1450 qdisc noqueue state UP
    link/ether 02:42:0a:08:00:05 brd ff:ff:ff:ff:ff:ff
    inet 10.8.0.5/24 brd 10.8.0.255 scope global eth0
       valid_lft forever preferred_lft forever
37: eth1@if38: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP
    link/ether 02:42:ac:12:00:03 brd ff:ff:ff:ff:ff:ff
    inet 172.18.0.3/16 brd 172.18.255.255 scope global eth1
       valid_lft forever preferred_lft forever
/ # ip route
default via 172.18.0.1 dev eth1
10.8.0.0/24 dev eth0 scope link  src 10.8.0.5
172.18.0.0/16 dev eth1 scope link  src 172.18.0.3
vidarlo avatar
ar flag
Do you need L2 connectivity? If not, tap and bridging is *always* the wrong answer.
Mohammed Noureldin avatar
pe flag
@vidarlo yes I need it. Mainly I will have a domain controller on the other host, which may serve some L2 needs, like printers for example.
vidarlo avatar
ar flag
Domain controllers and printers do not require L2 connectivity. They work Just Fine over L3. L2 is generally implied if you want to do non-TCP/IP traffic.
Mohammed Noureldin avatar
pe flag
@vidarlo Thank you for that info. I just read somewhere that some printers may have troubles if not used in L2. Anyway, it is interesting if I can figure out why it is not working using this setup.
Mohammed Noureldin avatar
pe flag
@vidarlo could you take a look on the new details and let me know if you can figure out what is wrong and why I cannot ping the containers on host2 by VPN users?
mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.