Score:0

I might have broken container port mapping

cn flag
etd

Port mapping is not working. I don’t know if it was something I did or what.

Setup: Arch Linux with kernel 5.16, Docker 20.10.12, using nginx-proxy with its acme companion to get certs for various apps in other containers. Everything was definitely working well.

Then all of a sudden I started getting 502 errors from the nginx proxy. It couldn’t reach the containerized apps on the ports I had specified. After investigating, I found out that the other containers were just opening up whatever ports they wanted to on the host. The host even thinks that the originally specified port is open, but it isn’t. So for example one of my apps is just nginx again serving a website. I have told Docker to map port 8001 on the host to 80 in the container. And then when I use lsof to show what ports are open on the host, I see 8001 in use. But then I can’t access anything on that, I can access it on port 80 (even though 80 is also in use by the nginx proxy). nmap confirms that it’s only port 80 that’s actually open on the container.

One thing I did recently was change the default policy in the FORWARD chain in iptables. I don’t see why changing the default would matter for packets that were already being routed somewhere.

Config:

Config of the nginx-proxy container:

[
    {
        "Id": "e24130ccef2bce43a11ebe5686e9a0ca45a7b0b13e32c4649095d11fd0361123",
        "Created": "2022-03-02T16:16:07.626095681Z",
        "Path": "/app/docker-entrypoint.sh",
        "Args": [
            "forego",
            "start",
            "-r"
        ],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 2767,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2022-03-02T16:16:08.672491906Z",
            "FinishedAt": "0001-01-01T00:00:00Z"
        },
        "Image": "sha256:82ea330a72d6f9d955287dc6e2c4c57a1466d480688574a4d0997c981bc495f3",
        "ResolvConfPath": "/var/lib/docker/containers/e24130ccef2bce43a11ebe5686e9a0ca45a7b0b13e32c4649095d11fd0361123/resolv.conf",
        "HostnamePath": "/var/lib/docker/containers/e24130ccef2bce43a11ebe5686e9a0ca45a7b0b13e32c4649095d11fd0361123/hostname",
        "HostsPath": "/var/lib/docker/containers/e24130ccef2bce43a11ebe5686e9a0ca45a7b0b13e32c4649095d11fd0361123/hosts",
        "LogPath": "/var/lib/docker/containers/e24130ccef2bce43a11ebe5686e9a0ca45a7b0b13e32c4649095d11fd0361123/e24130ccef2bce43a11ebe5686e9a0ca45a7b0b13e32c4649095d11fd0361123-json.log",
        "Name": "/nginx-proxy",
        "RestartCount": 0,
        "Driver": "btrfs",
        "Platform": "linux",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": [
                "/var/run/docker.sock:/tmp/docker.sock:ro"
            ],
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {}
            },
            "NetworkMode": "ivonet",
            "PortBindings": {
                "443/tcp": [
                    {
                        "HostIp": "",
                        "HostPort": "443"
                    }
                ],
                "80/tcp": [
                    {
                        "HostIp": "",
                        "HostPort": "80"
                    }
                ]
            },
            "RestartPolicy": {
                "Name": "always",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "CapAdd": null,
            "CapDrop": null,
            "CgroupnsMode": "private",
            "Dns": null,
            "DnsOptions": null,
            "DnsSearch": null,
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "private",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": null,
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": null,
            "DeviceCgroupRules": null,
            "DeviceRequests": null,
            "KernelMemory": 0,
            "KernelMemoryTCP": 0,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": null,
            "OomKillDisable": null,
            "PidsLimit": null,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0,
            "Mounts": [
                {
                    "Type": "volume",
                    "Source": "nginx-proxy_conf",
                    "Target": "/etc/nginx/conf.d",
                    "VolumeOptions": {}
                },
                {
                    "Type": "volume",
                    "Source": "nginx-proxy_vhost",
                    "Target": "/etc/nginx/vhost.d",
                    "VolumeOptions": {}
                },
                {
                    "Type": "volume",
                    "Source": "nginx-proxy_html",
                    "Target": "/usr/share/nginx/html",
                    "VolumeOptions": {}
                },
                {
                    "Type": "volume",
                    "Source": "nginx-proxy_certs",
                    "Target": "/etc/nginx/certs",
                    "ReadOnly": true,
                    "VolumeOptions": {}
                }
            ],
            "MaskedPaths": [
                "/proc/asound",
                "/proc/acpi",
                "/proc/kcore",
                "/proc/keys",
                "/proc/latency_stats",
                "/proc/timer_list",
                "/proc/timer_stats",
                "/proc/sched_debug",
                "/proc/scsi",
                "/sys/firmware"
            ],
            "ReadonlyPaths": [
                "/proc/bus",
                "/proc/fs",
                "/proc/irq",
                "/proc/sys",
                "/proc/sysrq-trigger"
            ]
        },
        "GraphDriver": {
            "Data": null,
            "Name": "btrfs"
        },
        "Mounts": [
            {
                "Type": "volume",
                "Name": "nginx-proxy_html",
                "Source": "/var/lib/docker/volumes/nginx-proxy_html/_data",
                "Destination": "/usr/share/nginx/html",
                "Driver": "local",
                "Mode": "z",
                "RW": true,
                "Propagation": ""
            },
            {
                "Type": "volume",
                "Name": "nginx-proxy_certs",
                "Source": "/var/lib/docker/volumes/nginx-proxy_certs/_data",
                "Destination": "/etc/nginx/certs",
                "Driver": "local",
                "Mode": "z",
                "RW": false,
                "Propagation": ""
            },
            {
                "Type": "bind",
                "Source": "/var/run/docker.sock",
                "Destination": "/tmp/docker.sock",
                "Mode": "ro",
                "RW": false,
                "Propagation": "rprivate"
            },
            {
                "Type": "volume",
                "Name": "nginx-proxy_conf",
                "Source": "/var/lib/docker/volumes/nginx-proxy_conf/_data",
                "Destination": "/etc/nginx/conf.d",
                "Driver": "local",
                "Mode": "z",
                "RW": true,
                "Propagation": ""
            },
            {
                "Type": "volume",
                "Name": "nginx-proxy_vhost",
                "Source": "/var/lib/docker/volumes/nginx-proxy_vhost/_data",
                "Destination": "/etc/nginx/vhost.d",
                "Driver": "local",
                "Mode": "z",
                "RW": true,
                "Propagation": ""
            }
        ],
        "Config": {
            "Hostname": "nginx-proxy",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": true,
            "AttachStderr": true,
            "ExposedPorts": {
                "443/tcp": {},
                "80/tcp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "NGINX_VERSION=1.21.6",
                "NJS_VERSION=0.7.2",
                "PKG_RELEASE=1",
                "NGINX_PROXY_VERSION=0.10.1-29-gfb8ddfd",
                "DOCKER_GEN_VERSION=0.8.2",
                "DOCKER_HOST=unix:///tmp/docker.sock"
            ],
            "Cmd": [
                "forego",
                "start",
                "-r"
            ],
            "Image": "nginxproxy/nginx-proxy:alpine",
            "Volumes": {
                "/etc/nginx/certs": {},
                "/etc/nginx/conf.d": {},
                "/etc/nginx/vhost.d": {},
                "/tmp/docker.sock": {},
                "/usr/share/nginx/html": {}
            },
            "WorkingDir": "/app",
            "Entrypoint": [
                "/app/docker-entrypoint.sh"
            ],
            "OnBuild": null,
            "Labels": {
                "com.docker.compose.config-hash": "3324b86760e2e436e707f1310aef7724e088d661c1bbeaaf573104e2644a08b0",
                "com.docker.compose.container-number": "1",
                "com.docker.compose.depends_on": "",
                "com.docker.compose.image": "sha256:82ea330a72d6f9d955287dc6e2c4c57a1466d480688574a4d0997c981bc495f3",
                "com.docker.compose.oneoff": "False",
                "com.docker.compose.project": "nginx-proxy",
                "com.docker.compose.project.config_files": "/home/winfield/contain/nginx-proxy/docker-compose.yaml",
                "com.docker.compose.project.working_dir": "/home/winfield/contain/nginx-proxy",
                "com.docker.compose.service": "nginx-proxy",
                "com.docker.compose.version": "2.2.3",
                "maintainer": "NGINX Docker Maintainers <[email protected]>",
                "org.opencontainers.image.authors": "Nicolas Duchon <[email protected]> (@buchdag), Jason Wilder",
                "org.opencontainers.image.created": "2022-02-28T00:02:10.384Z",
                "org.opencontainers.image.description": "Automated nginx proxy for Docker containers using docker-gen",
                "org.opencontainers.image.licenses": "MIT",
                "org.opencontainers.image.revision": "fb8ddfd08c0f6cb53e583ea22ff8be06f69c50dc",
                "org.opencontainers.image.source": "https://github.com/nginx-proxy/nginx-proxy",
                "org.opencontainers.image.title": "nginx-proxy",
                "org.opencontainers.image.url": "https://github.com/nginx-proxy/nginx-proxy",
                "org.opencontainers.image.version": "0.10.1-29-gfb8ddfd"
            },
            "StopSignal": "SIGQUIT"
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "cb385fa1d3dab87cccd2d89e70708e27e8bc1815867ae5fbc7e1c2f75000dc25",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {
                "443/tcp": [
                    {
                        "HostIp": "0.0.0.0",
                        "HostPort": "443"
                    },
                    {
                        "HostIp": "::",
                        "HostPort": "443"
                    }
                ],
                "80/tcp": [
                    {
                        "HostIp": "0.0.0.0",
                        "HostPort": "80"
                    },
                    {
                        "HostIp": "::",
                        "HostPort": "80"
                    }
                ]
            },
            "SandboxKey": "/var/run/docker/netns/cb385fa1d3da",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "ivonet": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": [
                        "nginx-proxy",
                        "nginx-proxy",
                        "e24130ccef2b"
                    ],
                    "NetworkID": "694e4c767b2eeca4c039e518db8294c2cc32a3be38f5dd0ad8779bce4099929c",
                    "EndpointID": "e653f85d0381f1c3bdaca9935eff3e4129d7941cf9c246d80e181e477c0bd79b",
                    "Gateway": "172.18.0.1",
                    "IPAddress": "172.18.0.3",
                    "IPPrefixLen": 24,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:ac:12:00:03",
                    "DriverOpts": null
                }
            }
        }
    }
]

Config of the proxied nginx container:

[
    {
        "Id": "e0b1be5c35ff60f337087f58819be190dcf495796114b6a2054dd78cf0e4679c",
        "Created": "2022-03-02T16:16:23.658997558Z",
        "Path": "/docker-entrypoint.sh",
        "Args": [
            "nginx",
            "-g",
            "daemon off;"
        ],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 3633,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2022-03-02T16:16:24.099412527Z",
            "FinishedAt": "0001-01-01T00:00:00Z"
        },
        "Image": "sha256:bef258acf10dc257d641c47c3a600c92f87be4b4ce4a5e4752b3eade7533dcd9",
        "ResolvConfPath": "/var/lib/docker/containers/e0b1be5c35ff60f337087f58819be190dcf495796114b6a2054dd78cf0e4679c/resolv.conf",
        "HostnamePath": "/var/lib/docker/containers/e0b1be5c35ff60f337087f58819be190dcf495796114b6a2054dd78cf0e4679c/hostname",
        "HostsPath": "/var/lib/docker/containers/e0b1be5c35ff60f337087f58819be190dcf495796114b6a2054dd78cf0e4679c/hosts",
        "LogPath": "/var/lib/docker/containers/e0b1be5c35ff60f337087f58819be190dcf495796114b6a2054dd78cf0e4679c/e0b1be5c35ff60f337087f58819be190dcf495796114b6a2054dd78cf0e4679c-json.log",
        "Name": "/atsuo.tg",
        "RestartCount": 0,
        "Driver": "btrfs",
        "Platform": "linux",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": [
                "/home/winfield/contain/atsuo.tg/site:/usr/share/nginx/html:rw"
            ],
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {}
            },
            "NetworkMode": "ivonet",
            "PortBindings": {
                "80/tcp": [
                    {
                        "HostIp": "",
                        "HostPort": "8001"
                    }
                ]
            },
            "RestartPolicy": {
                "Name": "",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "CapAdd": null,
            "CapDrop": null,
            "CgroupnsMode": "private",
            "Dns": null,
            "DnsOptions": null,
            "DnsSearch": null,
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "private",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": null,
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": null,
            "DeviceCgroupRules": null,
            "DeviceRequests": null,
            "KernelMemory": 0,
            "KernelMemoryTCP": 0,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": null,
            "OomKillDisable": null,
            "PidsLimit": null,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0,
            "MaskedPaths": [
                "/proc/asound",
                "/proc/acpi",
                "/proc/kcore",
                "/proc/keys",
                "/proc/latency_stats",
                "/proc/timer_list",
                "/proc/timer_stats",
                "/proc/sched_debug",
                "/proc/scsi",
                "/sys/firmware"
            ],
            "ReadonlyPaths": [
                "/proc/bus",
                "/proc/fs",
                "/proc/irq",
                "/proc/sys",
                "/proc/sysrq-trigger"
            ]
        },
        "GraphDriver": {
            "Data": null,
            "Name": "btrfs"
        },
        "Mounts": [
            {
                "Type": "bind",
                "Source": "/home/winfield/contain/atsuo.tg/site",
                "Destination": "/usr/share/nginx/html",
                "Mode": "rw",
                "RW": true,
                "Propagation": "rprivate"
            }
        ],
        "Config": {
            "Hostname": "atsuotg",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": true,
            "AttachStderr": true,
            "ExposedPorts": {
                "80/tcp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "VIRTUAL_PORT=8001",
                "LETSENCRYPT_HOST=atsuo.tg",
                "VIRTUAL_HOST=atsuo.tg",
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "NGINX_VERSION=1.21.6",
                "NJS_VERSION=0.7.2",
                "PKG_RELEASE=1"
            ],
            "Cmd": [
                "nginx",
                "-g",
                "daemon off;"
            ],
            "Image": "nginx:alpine",
            "Volumes": {
                "/usr/share/nginx/html": {}
            },
            "WorkingDir": "",
            "Entrypoint": [
                "/docker-entrypoint.sh"
            ],
            "OnBuild": null,
            "Labels": {
                "com.docker.compose.config-hash": "1f56d3b21bfd2b60df186db7d9ee19865f777207a470f0f36fd37a27cd65acef",
                "com.docker.compose.container-number": "1",
                "com.docker.compose.depends_on": "",
                "com.docker.compose.oneoff": "False",
                "com.docker.compose.project": "atsuotg",
                "com.docker.compose.project.config_files": "/home/winfield/contain/atsuo.tg/docker-compose.yaml",
                "com.docker.compose.project.working_dir": "/home/winfield/contain/atsuo.tg",
                "com.docker.compose.service": "atsuo.tg",
                "com.docker.compose.version": "2.2.3",
                "maintainer": "NGINX Docker Maintainers <[email protected]>"
            },
            "StopSignal": "SIGQUIT"
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "9339a100e64c7de59e3c6b9f00761ce9bc90789b8139daed8781c14bc91258e8",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {
                "80/tcp": [
                    {
                        "HostIp": "0.0.0.0",
                        "HostPort": "8001"
                    },
                    {
                        "HostIp": "::",
                        "HostPort": "8001"
                    }
                ]
            },
            "SandboxKey": "/var/run/docker/netns/9339a100e64c",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "ivonet": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": [
                        "atsuo.tg",
                        "atsuo.tg",
                        "e0b1be5c35ff",
                        "atsuotg"
                    ],
                    "NetworkID": "694e4c767b2eeca4c039e518db8294c2cc32a3be38f5dd0ad8779bce4099929c",
                    "EndpointID": "6ccfdba4120787c39c71505403db08c67f32284637e5bbf2c4abbf0cdb8c15b7",
                    "Gateway": "172.18.0.1",
                    "IPAddress": "172.18.0.4",
                    "IPPrefixLen": 24,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:ac:12:00:04",
                    "DriverOpts": null
                }
            }
        }
    }
]

Config of the Docker custom bridge network:

[
    {
        "Name": "ivonet",
        "Id": "694e4c767b2eeca4c039e518db8294c2cc32a3be38f5dd0ad8779bce4099929c",
        "Created": "2022-03-02T11:15:32.631561185-05:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": {},
            "Config": [
                {
                    "Subnet": "172.18.0.0/24",
                    "Gateway": "172.18.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "adac9051c7813cd9bab6747c798e058983c061b56fbcc9d5cee9d6dacd35461b": {
                "Name": "nginx-acme",
                "EndpointID": "15ae704194cf33bad77911800541de67f2ee099229014106af65b09d9bf58fa9",
                "MacAddress": "02:42:ac:12:00:02",
                "IPv4Address": "172.18.0.2/24",
                "IPv6Address": ""
            },
            "e0b1be5c35ff60f337087f58819be190dcf495796114b6a2054dd78cf0e4679c": {
                "Name": "atsuo.tg",
                "EndpointID": "6ccfdba4120787c39c71505403db08c67f32284637e5bbf2c4abbf0cdb8c15b7",
                "MacAddress": "02:42:ac:12:00:04",
                "IPv4Address": "172.18.0.4/24",
                "IPv6Address": ""
            },
            "e24130ccef2bce43a11ebe5686e9a0ca45a7b0b13e32c4649095d11fd0361123": {
                "Name": "nginx-proxy",
                "EndpointID": "e653f85d0381f1c3bdaca9935eff3e4129d7941cf9c246d80e181e477c0bd79b",
                "MacAddress": "02:42:ac:12:00:03",
                "IPv4Address": "172.18.0.3/24",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.bridge.enable_icc": "true"
        },
        "Labels": {}
    }
]

Output of iptables-save on host:

# Generated by iptables-save v1.8.7 on Wed Mar  2 15:59:49 2022
*filter
:INPUT ACCEPT [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [0:0]
:DOCKER - [0:0]
:DOCKER-ISOLATION-STAGE-1 - [0:0]
:DOCKER-ISOLATION-STAGE-2 - [0:0]
:DOCKER-USER - [0:0]
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o br-694e4c767b2e -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-694e4c767b2e -j DOCKER
-A FORWARD -i br-694e4c767b2e ! -o br-694e4c767b2e -j ACCEPT
-A FORWARD -i br-694e4c767b2e -o br-694e4c767b2e -j ACCEPT
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A DOCKER -d 172.18.0.3/32 ! -i br-694e4c767b2e -o br-694e4c767b2e -p tcp -m tcp --dport 443 -j ACCEPT
-A DOCKER -d 172.18.0.3/32 ! -i br-694e4c767b2e -o br-694e4c767b2e -p tcp -m tcp --dport 80 -j ACCEPT
-A DOCKER -d 172.18.0.4/32 ! -i br-694e4c767b2e -o br-694e4c767b2e -p tcp -m tcp --dport 80 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i br-694e4c767b2e ! -o br-694e4c767b2e -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o br-694e4c767b2e -j DROP
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN
COMMIT
# Completed on Wed Mar  2 15:59:49 2022
# Generated by iptables-save v1.8.7 on Wed Mar  2 15:59:49 2022
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:DOCKER - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 172.18.0.0/24 ! -o br-694e4c767b2e -j MASQUERADE
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A POSTROUTING -s 172.18.0.3/32 -d 172.18.0.3/32 -p tcp -m tcp --dport 443 -j MASQUERADE
-A POSTROUTING -s 172.18.0.3/32 -d 172.18.0.3/32 -p tcp -m tcp --dport 80 -j MASQUERADE
-A POSTROUTING -s 172.18.0.4/32 -d 172.18.0.4/32 -p tcp -m tcp --dport 80 -j MASQUERADE
-A DOCKER -i br-694e4c767b2e -j RETURN
-A DOCKER -i docker0 -j RETURN
-A DOCKER ! -i br-694e4c767b2e -p tcp -m tcp --dport 443 -j DNAT --to-destination 172.18.0.3:443
-A DOCKER ! -i br-694e4c767b2e -p tcp -m tcp --dport 80 -j DNAT --to-destination 172.18.0.3:80
-A DOCKER ! -i br-694e4c767b2e -p tcp -m tcp --dport 8001 -j DNAT --to-destination 172.18.0.4:80
COMMIT
# Completed on Wed Mar  2 15:59:49 2022

Output of lsof -i -P -n | grep LISTEN on host:

systemd-r  911 systemd-resolve   12u  IPv4  22667      0t0  TCP *:5355 (LISTEN)
systemd-r  911 systemd-resolve   14u  IPv6  22670      0t0  TCP *:5355 (LISTEN)
systemd-r  911 systemd-resolve   18u  IPv4  22673      0t0  TCP 127.0.0.53:53 (LISTEN)
systemd-r  911 systemd-resolve   20u  IPv4  22675      0t0  TCP 127.0.0.54:53 (LISTEN)
sshd       914            root    3u  IPv4  46226      0t0  TCP *:26506 (LISTEN)
sshd       914            root    4u  IPv6  46228      0t0  TCP *:26506 (LISTEN)
mariadbd   964           mysql   19u  IPv4  57354      0t0  TCP *:3306 (LISTEN)
mariadbd   964           mysql   21u  IPv6  57355      0t0  TCP *:3306 (LISTEN)
docker-pr 2674            root    4u  IPv4  53545      0t0  TCP *:443 (LISTEN)
docker-pr 2681            root    4u  IPv6  62592      0t0  TCP *:443 (LISTEN)
docker-pr 2694            root    4u  IPv4  51280      0t0  TCP *:80 (LISTEN)
docker-pr 2700            root    4u  IPv6  53552      0t0  TCP *:80 (LISTEN)
docker-pr 3591            root    4u  IPv4  50347      0t0  TCP *:8001 (LISTEN)
docker-pr 3597            root    4u  IPv6  46444      0t0  TCP *:8001 (LISTEN)
vidarlo avatar
ar flag
Post your configuration for docker and nginx. Otherwise we're running blind.
etd avatar
cn flag
etd
@vidarlo ok I've added what I figure could help. The config for the inner nginx server is default, from the nginx:alpine image
co flag
This question will be missing details needed to answer it if and when the links go bad. Please ensure all the needed content to answer the question is included here and links are only for supplemental details.
etd avatar
cn flag
etd
@BMitch ok, that makes sense. I've moved everything to the body of the post
Score:0
co flag

The VIRTUAL_PORT setting looks incorrect. You are using VIRTUAL_PORT=8001 while the container is listening on port 80. So instead you should have VIRTUAL_PORT=80.

Connectivity between containers happens on a common user created network (e.g. ivonet), and on container ports. The published port on the host is for external users to access the container by connecting to the host port which gets forwarded to the container port.

etd avatar
cn flag
etd
Oh geez, that does it. I have a lot to learn :|
mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.