Join Log in
Score:0
Major Kuprich avatar Server

Varnish doesn't open backend connection

us flag
Major Kuprich

How can I connect Varnish container with PHP-FPM + NGINX containers? Docker started correctly, site works but backendopen logs are empty.

Backend is healthy (returns HTTP 200 status).

varnishlog -g raw -i Backend_health

0 Backend_health - default Still healthy 4---X-RH 7 5 10 0.048069 0.065633 "HTTP/1.1 200 OK"

From http (nginx) container I see request in docker logs.

docker logs http

[10/Sep/2021:17:24:01 +0000] "GET /health_check.php HTTP/1.1" 200 5 "-" "-"

But backend connection doesn't open connection, logs are empty, verified via: varnishlog -i backendopen command. I've tried solution from this site, but not working: Php+Nginx+Varnish on docker-compose, ubuntu18.

docker-compose.yml

http:
    build:
    context: docker/http
    container_name: http
    ports:
       - 80:80
       - 443:443
    depends_on:
       - php
    volumes:
        - ./project:/var/www/project

php:
    build:
        context: docker/php
    container_name: php
    working_dir: /var/www/project
    volumes:
        - ./project:/var/www/project

    
varnish:
    build:
    context: docker/varnish
    container_name: varnish
    ports:
       - 6081:6081
       - 6082:6082

VCL config:

vcl 4.0;

import std;

backend default {
    .host = "http";
    .port = "80";
    .first_byte_timeout = 600s;
    .probe = {
        #.url = "/health_check.php";
        .request =
                    "GET /health_check.php HTTP/1.1"
                    "Host: http"
                    "Connection: close";
        .timeout = 2s;
        .interval = 5s;
        .window = 10;
        .threshold = 5;
   }
}

acl purge {
    "http";
}

sub vcl_recv {
    if (req.restarts > 0) {
        set req.hash_always_miss = true;
    }

    if (req.method == "PURGE") {
        if (client.ip !~ purge) {
            return (synth(405, "Method not allowed"));
        }
        # To use the X-Pool header for purging varnish during automated deployments, make sure the X-Pool header
        # has been added to the response in your backend server config. This is used, for example, by the
        # capistrano-magento2 gem for purging old content from varnish during it's deploy routine.
        if (!req.http.X-Magento-Tags-Pattern && !req.http.X-Pool) {
            return (synth(400, "X-Magento-Tags-Pattern or X-Pool header required"));
        }
        if (req.http.X-Magento-Tags-Pattern) {
          ban("obj.http.X-Magento-Tags ~ " + req.http.X-Magento-Tags-Pattern);
        }
        if (req.http.X-Pool) {
          ban("obj.http.X-Pool ~ " + req.http.X-Pool);
        }
        return (synth(200, "Purged"));
    }

    if (req.method != "GET" &&
        req.method != "HEAD" &&
        req.method != "PUT" &&
        req.method != "POST" &&
        req.method != "TRACE" &&
        req.method != "OPTIONS" &&
        req.method != "DELETE") {
          /* Non-RFC2616 or CONNECT which is weird. */
          return (pipe);
    }

    # We only deal with GET and HEAD by default
    if (req.method != "GET" && req.method != "HEAD") {
        return (pass);
    }

    # Bypass shopping cart and checkout
    if (req.url ~ "/checkout") {
        return (pass);
    }

    # Bypass health check requests
    if (req.url ~ "/pub/health_check.php") {
        return (pass);
    }

    # Set initial grace period usage status
    set req.http.grace = "none";

    # normalize url in case of leading HTTP scheme and domain
    set req.url = regsub(req.url, "^http[s]?://", "");

    # collect all cookies
    std.collect(req.http.Cookie);

    # Compression filter. See https://www.varnish-cache.org/trac/wiki/FAQ/Compression
    if (req.http.Accept-Encoding) {
        if (req.url ~ "\.(jpg|jpeg|png|gif|gz|tgz|bz2|tbz|mp3|ogg|swf|flv)$") {
            # No point in compressing these
            unset req.http.Accept-Encoding;
        } elsif (req.http.Accept-Encoding ~ "gzip") {
            set req.http.Accept-Encoding = "gzip";
        } elsif (req.http.Accept-Encoding ~ "deflate" && req.http.user-agent !~ "MSIE") {
            set req.http.Accept-Encoding = "deflate";
        } else {
            # unknown algorithm
            unset req.http.Accept-Encoding;
        }
    }

    # Remove all marketing get parameters to minimize the cache objects
    if (req.url ~ "(\?|&)(gclid|cx|ie|cof|siteurl|zanpid|origin|fbclid|mc_[a-z]+|utm_[a-z]+|_bta_[a-z]+)=") {
        set req.url = regsuball(req.url, "(gclid|cx|ie|cof|siteurl|zanpid|origin|fbclid|mc_[a-z]+|utm_[a-z]+|_bta_[a-z]+)=[-_A-z0-9+()%.]+&?", "");
        set req.url = regsub(req.url, "[?|&]+$", "");
    }

    # Static files caching
    if (req.url ~ "^/(pub/)?(media|static)/") {
        # Static files should not be cached by default
        return (pass);

        # But if you use a few locales and don't use CDN you can enable caching static files by commenting previous line (#return (pass);) and uncommenting next 3 lines
        #unset req.http.Https;
        #unset req.http.X-Forwarded-Proto;
        #unset req.http.Cookie;
    }

    # Authenticated GraphQL requests should not be cached by default
    if (req.url ~ "/graphql" && req.http.Authorization ~ "^Bearer") {
        return (pass);
    }

    return (hash);
}

sub vcl_hash {
    if (req.http.cookie ~ "X-Magento-Vary=") {
        hash_data(regsub(req.http.cookie, "^.*?X-Magento-Vary=([^;]+);*.*$", "\1"));
    }

    # To make sure http users don't see ssl warning
    if (req.http.X-Forwarded-Proto) {
        hash_data(req.http.X-Forwarded-Proto);
    }
    

    if (req.url ~ "/graphql") {
        call process_graphql_headers;
    }
}

sub process_graphql_headers {
    if (req.http.Store) {
        hash_data(req.http.Store);
    }
    if (req.http.Content-Currency) {
        hash_data(req.http.Content-Currency);
    }
}

sub vcl_backend_response {

    set beresp.grace = 3d;

    if (beresp.http.content-type ~ "text") {
        set beresp.do_esi = true;
    }

    if (bereq.url ~ "\.js$" || beresp.http.content-type ~ "text") {
        set beresp.do_gzip = true;
    }

    if (beresp.http.X-Magento-Debug) {
        set beresp.http.X-Magento-Cache-Control = beresp.http.Cache-Control;
    }

    # cache only successfully responses and 404s
    if (beresp.status != 200 && beresp.status != 404) {
        set beresp.ttl = 0s;
        set beresp.uncacheable = true;
        return (deliver);
    } elsif (beresp.http.Cache-Control ~ "private") {
        set beresp.uncacheable = true;
        set beresp.ttl = 86400s;
        return (deliver);
    }

    # validate if we need to cache it and prevent from setting cookie
    if (beresp.ttl > 0s && (bereq.method == "GET" || bereq.method == "HEAD")) {
        unset beresp.http.set-cookie;
    }

   # If page is not cacheable then bypass varnish for 2 minutes as Hit-For-Pass
   if (beresp.ttl <= 0s ||
       beresp.http.Surrogate-control ~ "no-store" ||
       (!beresp.http.Surrogate-Control &&
       beresp.http.Cache-Control ~ "no-cache|no-store") ||
       beresp.http.Vary == "*") {
        # Mark as Hit-For-Pass for the next 2 minutes
        set beresp.ttl = 120s;
        set beresp.uncacheable = true;
    }

    return (deliver);
}

sub vcl_deliver {
    if (resp.http.X-Magento-Debug) {
        if (resp.http.x-varnish ~ " ") {
            set resp.http.X-Magento-Cache-Debug = "HIT";
            set resp.http.Grace = req.http.grace;
        } else {
            set resp.http.X-Magento-Cache-Debug = "MISS";
        }
    } else {
        unset resp.http.Age;
    }

    # Not letting browser to cache non-static files.
    if (resp.http.Cache-Control !~ "private" && req.url !~ "^/(pub/)?(media|static)/") {
        set resp.http.Pragma = "no-cache";
        set resp.http.Expires = "-1";
        set resp.http.Cache-Control = "no-store, no-cache, must-revalidate, max-age=0";
    }

    unset resp.http.X-Magento-Debug;
    unset resp.http.X-Magento-Tags;
    unset resp.http.X-Powered-By;
    unset resp.http.Server;
    unset resp.http.X-Varnish;
    unset resp.http.Via;
    unset resp.http.Link;
}

sub vcl_hit {
    if (obj.ttl >= 0s) {
        # Hit within TTL period
        return (deliver);
    }
    if (std.healthy(req.backend_hint)) {
        if (obj.ttl + 300s > 0s) {
            # Hit after TTL expiration, but within grace period
            set req.http.grace = "normal (healthy server)";
            return (deliver);
        } else {
            # Hit after TTL and grace expiration
            return (restart);
        }
    } else {
        # server is not healthy, retrieve from cache
        set req.http.grace = "unlimited (unhealthy server)";
        return (deliver);
    }
}
351
0 + 0
php
Score:0
Thijs Feryn avatar Server
in flag
Thijs Feryn

Try running varnishlog -g request -q "ReqUrl eq '/'" to see what's going on when requesting the homepage.

If you perform this command when the cache is empty, we should see an attempt by Varnish to connect with the backend.

Please share the VSL output here and I'll assist.

UPDATE

I noticed in your docker-compose.yml file that your http container is configured to handle traffic on ports 80 & 443.

The problem

The varnish container is listening on ports 6081 & 6082. Unless you're directly routing traffic to port 6081 it is quite obvious that Varnish is not receiving any requests.

The solution

Please make sure your Varnish container also listens for incoming traffic on port 80. In your VCL you can point to port 80 on the http container.

But as far as the exposure of ports go, you might want to forward your http container's port 80 to 8080. This will avoid a clash with Varnish being on port 80 as well.

You can also directly run the official Varnish image instead of building one yourself.

Here's a tutorial on how to run and configure the official Docker image: https://www.varnish-software.com/developers/tutorials/running-varnish-docker/

TLS termination

Currently port 443 is linked to your http container. If you can handle TLS termination there and proxy HTTPS requests to Varnish, that's fine.

If not, you can run a Hitch container that performs the TLS termination for you.

0 + 0
Major Kuprich avatar
us flag
Major Kuprich
Still empty logs: https://i.imgur.com/GFcDXEo.png I've tried update .probe section to: a) remove .request section and uncomment .url b) update host in .request section to my local mapping from /etc/hosts file (IP, hostname) But no success. :( PS Please be informed that I'm using Magento2 CMS inside these containers.
0
Reply
Major Kuprich avatar
us flag
Major Kuprich
Tried update docker-compose.yml to forward 80 to 8080 port in http container. Change port for Varnish to 80. Backend health is sick: 404 Not Found. Backend open not opening, any logs not working. :( Check new config, please: https://pastebin.com/BYi1cHgM
0
Reply
Thijs Feryn avatar
in flag
Thijs Feryn
@MajorKuprich I'm afraid you forwarded the ports in the wrong order. You did `6081:80` but it has to be `80:6081`. The same applies to `80:8080` which should be `8080:80`. Hope that helps. If you can access the `http` container over port `8080` and you can simulate the 404 error, you might be able to fix it and get the whole thing working.
0
Reply
ธงชาติไทย
Elon Musk
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.