Join Log in
Score:0
Michael avatar Server

No access to nginx container in Docker via IPv6

re flag
Michael

I am struggling to have my Debian VM (5.10.0-15-amd64) accept incoming requests for a plain nginx webserver hosted on Docker. When binding the IPv6 address directly within the docker-compose.yaml file I get an 'Empty reply from server" error:

curl -v sub.domain.id:8030
*   Trying 2003:a:b:c:d:e:fe40:1611:8030...
* Connected to sub.domain.id (2003:a:b:c:d:e:fe40:1611) port 8030 (#0)
> GET / HTTP/1.1
> Host: sub.domain.id:8030
> User-Agent: curl/7.79.1
> Accept: */*
> 
* Empty reply from server
* Closing connection 0
curl: (52) Empty reply from server

---
version: "2.2"
services:
  website:
    image: nginx
    ports:
      - "2003:a:b:c:d:e:fe40:1611:8030:80"
    restart: always
    networks:
      - nginxnet
    sysctls:
      - net.ipv6.conf.all.disable_ipv6=0

networks:
  nginxnet:
    enable_ipv6: true
    ipam:
      config:
        - subnet: "fe80:aaaa:bbbb:ccc::/64"

When I just map the ports, the connection is being reset by peer:

curl -v sub.domain.id:8030
*   Trying 2003:a:b:c:d:e:fe40:1611:8030...
* Connected to sub.domain.id (2003:a:b:c:d:e:fe40:1611) port 8030 (#0)
> GET / HTTP/1.1
> Host: sub.domain.id:8030
> User-Agent: curl/7.79.1
> Accept: */*
> 
* Recv failure: Connection reset by peer
* Closing connection 0
curl: (56) Recv failure: Connection reset by peer

cat docker-compose.yaml 
---
version: "2.2"
services:
  website:
    image: nginx
    ports:
      - 8030:80
    restart: always
    networks:
      - nginxnet
    sysctls:
      - net.ipv6.conf.all.disable_ipv6=0

networks:
  nginxnet:
    enable_ipv6: true
    ipam:
      config:
        - subnet: "fe80:aaaa:bbbb:ccc::/64"

The socket is properly exposed:

ss -tulpn      
Netid      State       Recv-Q      Send-Q                                      Local Address:Port              Peer Address:Port      Process      
udp        UNCONN      0           0                                                 0.0.0.0:6881                   0.0.0.0:*                      
udp        UNCONN      0           0                                                 0.0.0.0:3478                   0.0.0.0:*                      
udp        UNCONN      0           0                                                 0.0.0.0:111                    0.0.0.0:*                      
udp        UNCONN      0           0                                                 0.0.0.0:10001                  0.0.0.0:*                      
udp        UNCONN      0           0                                                 0.0.0.0:1900                   0.0.0.0:*                      
udp        UNCONN      0           0                                                    [::]:111                       [::]:*                      
tcp        LISTEN      0           64                                                0.0.0.0:45311                  0.0.0.0:*                      
tcp        LISTEN      0           4096                                              0.0.0.0:6881                   0.0.0.0:*                      
tcp        LISTEN      0           4096                                              0.0.0.0:6789                   0.0.0.0:*                      
tcp        LISTEN      0           4096                                              0.0.0.0:5514                   0.0.0.0:*                      
tcp        LISTEN      0           4096                                              0.0.0.0:8843                   0.0.0.0:*                      
tcp        LISTEN      0           4096                                            127.0.0.1:44427                  0.0.0.0:*                      
tcp        LISTEN      0           4096                                              0.0.0.0:111                    0.0.0.0:*                      
tcp        LISTEN      0           4096                                              0.0.0.0:8080                   0.0.0.0:*                      
tcp        LISTEN      0           4096                                              0.0.0.0:8880                   0.0.0.0:*                      
tcp        LISTEN      0           4096                                              0.0.0.0:8081                   0.0.0.0:*                      
tcp        LISTEN      0           128                                               0.0.0.0:22                     0.0.0.0:*                      
tcp        LISTEN      0           4096                                              0.0.0.0:8443                   0.0.0.0:*                      
tcp        LISTEN      0           4096               [2003:a:b:c:d:e:fe40:1611]:8030                      [::]:*                      
tcp        LISTEN      0           4096                                                 [::]:111                       [::]:*                      
tcp        LISTEN      0           128                                                  [::]:22                        [::]:*                      
tcp        LISTEN      0           64                                                   [::]:40733                     [::]:*

For Debian, I have switched back from nftables to iptables/6 with:

sudo update-alternatives --set iptables /usr/sbin/iptables-legacy
sudo update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy

And the following iptables are present:

sudo iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy DROP)
target     prot opt source               destination         
DOCKER-USER  all  --  anywhere             anywhere            
DOCKER-ISOLATION-STAGE-1  all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain DOCKER (9 references)
target     prot opt source               destination         
ACCEPT     udp  --  anywhere             172.18.0.2           udp dpt:10001
ACCEPT     tcp  --  anywhere             172.19.0.2           tcp dpt:tproxy
ACCEPT     tcp  --  anywhere             172.18.0.2           tcp dpt:8880
ACCEPT     tcp  --  anywhere             172.19.0.2           tcp dpt:6881
ACCEPT     tcp  --  anywhere             172.18.0.2           tcp dpt:8843
ACCEPT     udp  --  anywhere             172.19.0.2           udp dpt:6881
ACCEPT     tcp  --  anywhere             172.18.0.2           tcp dpt:8443
ACCEPT     tcp  --  anywhere             172.18.0.2           tcp dpt:http-alt
ACCEPT     tcp  --  anywhere             172.18.0.2           tcp dpt:6789
ACCEPT     tcp  --  anywhere             172.18.0.2           tcp dpt:5514
ACCEPT     udp  --  anywhere             172.18.0.2           udp dpt:3478
ACCEPT     udp  --  anywhere             172.18.0.2           udp dpt:1900

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target     prot opt source               destination         
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere            
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere            
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere            
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere            
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere            
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere            
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere            
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere            
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere            
RETURN     all  --  anywhere             anywhere            

Chain DOCKER-ISOLATION-STAGE-2 (9 references)
target     prot opt source               destination         
DROP       all  --  anywhere             anywhere            
DROP       all  --  anywhere             anywhere            
DROP       all  --  anywhere             anywhere            
DROP       all  --  anywhere             anywhere            
DROP       all  --  anywhere             anywhere            
DROP       all  --  anywhere             anywhere            
DROP       all  --  anywhere             anywhere            
DROP       all  --  anywhere             anywhere            
DROP       all  --  anywhere             anywhere            
RETURN     all  --  anywhere             anywhere            

Chain DOCKER-USER (1 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere            



sudo ip6tables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     tcp      anywhere             anywhere             state NEW tcp dpt:8030

Chain FORWARD (policy DROP)
target     prot opt source               destination         
DOCKER-ISOLATION-STAGE-1  all      anywhere             anywhere            
ACCEPT     all      anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all      anywhere             anywhere            
ACCEPT     all      anywhere             anywhere            
ACCEPT     all      anywhere             anywhere            

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain DOCKER (1 references)
target     prot opt source               destination         

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target     prot opt source               destination         
DOCKER-ISOLATION-STAGE-2  all      anywhere             anywhere            
DOCKER-ISOLATION-STAGE-2  all      anywhere             anywhere            
DOCKER-ISOLATION-STAGE-2  all      anywhere             anywhere            
DOCKER-ISOLATION-STAGE-2  all      anywhere             anywhere            
DOCKER-ISOLATION-STAGE-2  all      anywhere             anywhere            
DOCKER-ISOLATION-STAGE-2  all      anywhere             anywhere            
DOCKER-ISOLATION-STAGE-2  all      anywhere             anywhere            
DOCKER-ISOLATION-STAGE-2  all      anywhere             anywhere            
RETURN     all      anywhere             anywhere            

Chain DOCKER-ISOLATION-STAGE-2 (8 references)
target     prot opt source               destination         
DROP       all      anywhere             anywhere            
DROP       all      anywhere             anywhere            
DROP       all      anywhere             anywhere            
DROP       all      anywhere             anywhere            
DROP       all      anywhere             anywhere            
DROP       all      anywhere             anywhere            
DROP       all      anywhere             anywhere            
DROP       all      anywhere             anywhere            
RETURN     all      anywhere             anywhere

The network interface is configured:

2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:a0:98:40:16:11 brd ff:ff:ff:ff:ff:ff
    inet 192.168.2.10/24 brd 192.168.2.255 scope global enp0s3
       valid_lft forever preferred_lft forever
    inet6 2003:a:b:c:d:e:fe40:1611/64 scope global dynamic mngtmpaddr 
       valid_lft 7003sec preferred_lft 1133sec
    inet6 fe80::2a0:98ff:fe40:1611/64 scope link 
       valid_lft forever preferred_lft forever

Any help is highly appreciated as I am struggling with this for weeks and can't get my head around.

673
1 + 0
Score:0
avatar Server
pt flag
larsks

I think your problem here is the fact that you've assigned an address from the link-local address range to your nginxnet subnet.

I have ipv6 enabled in Docker via /etc/docker/daemon.json:

{
  "ipv6": true,
  "fixed-cidr-v6": "fd4a:5a20:acde:8e7a::/64"
}

My host has the following IPv6 address assigned to interface eth1:

$ ip addr show eth1
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether d0:c2:4e:46:ce:00 brd ff:ff:ff:ff:ff:ff
    altname enp14s0u1u1
    inet 10.20.22.189/24 brd 10.20.22.255 scope global dynamic noprefixroute eth1
       valid_lft 82961sec preferred_lft 82961sec
    inet6 2620:52:0:1416:7cc4:ed2b:33c:5a5/64 scope global dynamic noprefixroute 
       valid_lft 2591894sec preferred_lft 604694sec
    inet6 fe80::ff3d:758a:eebb:b9de/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

If I use this docker-compose.yaml...

version: "3"

services:
  website:
    image: docker.io/nginx:mainline
    ports:
      - "[2620:52:0:1416:7cc4:ed2b:33c:5a5]:8030:80"
    networks:
      - nginxnet

networks:
  nginxnet:
    enable_ipv6: true
    ipam:
      config:
        - subnet: "fd4a:5a20:acde:cafe::/64"

...it works as expected; once nginx is up and running I can successfully run:

$ curl 'http://[2620:52:0:1416:7cc4:ed2b:33c:5a5]:8030'

But I modify the configuration to use the same subnet you're trying to use:

networks:
  nginxnet:
    enable_ipv6: true
    ipam:
      config:
        - subnet: "fe80:aaaa:bbbb:ccc::/64"

I see the same behavior you've described in your question:

$ curl 'http://[2620:52:0:1416:7cc4:ed2b:33c:5a5]:8030'
curl: (56) Recv failure: Connection reset by peer
+ 3
Michael avatar
re flag
Michael
Hi @larsks thank you very much for taking the time and looking into this. Regarding your fixed-cidr-v6 setting for the Docker daemon: which prefix are you using with "fd4a:5a20:acde:8e7a::/64" and why are you using this? Also, should I be specifying a subnet from within the global prefix that my ISP assigns? Why can I not use a link local one? I would like to understand this because I feel like I can't randomly chose any prefix then.
0
Reply
pt flag
larsks
Re: 'which prefix are you using with "fd4a:5a20:acde:8e7a::/64"', I'm not sure how to answer that...the prefix that I'm using is `fd4a:5a20:acde:8e7a::/64`, which is a randomly generated prefix from the [unique local address](https://en.wikipedia.org/wiki/Unique_local_address) range. You can randomly select any prefix from within that range. You don't need to use an address from the range your ISP has assigned for your docker subnets. Using a link local address I suspect results in routing problems (because every interface has a route to the link-local range).
0
Reply
Michael avatar
re flag
Michael
Thank you VERY much larsks! This has solved my problem. The fact that it's the type of local IPv6 address that makes a difference is surprising. But I feel there is much to learn for me.
0
Reply
ธงชาติไทย
Elon Musk
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.