Score:1

How can I curl -IL all the website pages starting with the root pagenot just 1 page at the time?

cy flag

I need to curl -IL all the website and maybe do cron job or something every 24hrs so fastcgi can cache all the pages on website

The cache don't hit unless I curl -IL the webpage so I need something that will curl all the webpage available on the website.

Like you can see bellow:

root@ubuntu-s-1vcpu-1gb-amd-sfo3-01:~# curl -IL https://www.example.com
HTTP/2 200 
date: Thu, 23 Feb 2023 10:31:44 GMT
content-type: text/html; charset=UTF-8
link: <https://www.example.com/wp-json/>; rel="https://api.w.org/"
link: <https://www.example.com/wp-json/wp/v2/pages/16274>; rel="alternate"; type="application/json"
link: <https://www.example.com/>; rel=shortlink
x-fastcgi-cache: HIT
cf-cache-status: DYNAMIC
report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=2K%2BxMLMMrvGpe%2FdpG38gU%2FKaBPBAL5I3rYNL5ATQ65wPM2qMnQmaCe2jLx88N%2B%2BHeUw1RRI5EnrYh9%2F6P5fj7xAipg%2FnOxi4IV8e4IqPWMyANW9JnKndXGfBZw0r3LtF1IEl"}],"group":"cf-nel","max_age":604800}
nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
server: cloudflare
cf-ray: 79df4b241af32f15-LAX
alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400
Score:1
nr flag

That's WordPress what you using there. I'm sure there are better ways to warm up the cache: https://de.wordpress.org/plugins/search/cache+warm/

Create a file like warmup.sh with the following code

#!/bin/bash

# URL of main Sitemap
sitemap_url="https://www.vgopromo.com/wp-sitemap.xml"

# Extract all Sitemap URLs
sitemap_urls=$(curl -s "$sitemap_url" | grep -oP '(?<=<loc>)[^<]+')

# Loop over and retrieve the individual URLs
for sitemap in $sitemap_urls; do
    urls=$(curl -s "$sitemap" | grep -oP '(?<=<loc>)[^<]+')
    for url in $urls; do
        curl -IL "$url"
    done
done

This will do what you requested.

You can also a Cronjob that run this file.

# Example: At minute 15 past every hour.
15 */1 * * * /bin/bash /root/warmup.sh

EDIT

This modified code adds the option to define also subdomains.

#!/bin/bash

# Array of Subdomains, just extend in same princip
subdomains=("www" "subdomain_2")

# Loop over Subdomains and retrieve URLs
for subdomain in "${subdomains[@]}"; do
    sitemap_url="https://$subdomain.vgopromo.com/wp-sitemap.xml"
    sitemap_urls=$(curl -s "$sitemap_url" | grep -oP '(?<=<loc>)[^<]+')
    for sitemap in $sitemap_urls; do
        urls=$(curl -s "$sitemap" | grep -oP '(?<=<loc>)[^<]+')
        for url in $urls; do
            curl -IL "$url"
        done
    done
done

Ill personally suggest this way. Makes it compatible with multiple projects

#!/bin/bash

# Array of Domains, just extend in same princip
domains=("https://www.vgopromo.com/wp-sitemap.xml" "https://example.vgopromo.com/wp-sitemap.xml")

# Loop over Domains and retrieve URLs
for domain in "${domains[@]}"; do
    sitemap_url="$domain"
    sitemap_urls=$(curl -s "$sitemap_url" | grep -oP '(?<=<loc>)[^<]+')
    for sitemap in $sitemap_urls; do
        urls=$(curl -s "$sitemap" | grep -oP '(?<=<loc>)[^<]+')
        for url in $urls; do
            curl -IL "$url"
        done
    done
done
Crypto Coupons avatar
cy flag
if I do this sitemap_url="https://*.vgopromo.com/wp-sitemap.xml" will it do it for the subdomains as well ?
scysys avatar
nr flag
No. I think thats not a good idea. Also then it was not easy like this to receive all available subdomains from your domain. Its better to add a variable, where you can enter all available subdomains. Ill edited with examples.
Crypto Coupons avatar
cy flag
# 0 8 * * * /bin/bash /root/warmup.sh <- every 24 hours correct?
scysys avatar
nr flag
I dont think you clean your cache every 24 hours? I think best would be to start only when cache was cleaned. Maybe a cron once per week is enough. But thats up on you :)
Crypto Coupons avatar
cy flag
no I didnt clean it what is the command to clean the cache ?
scysys avatar
nr flag
I just asked how often you do it. You wanted this script to warmup your cache. So it does not need to run when you wont clean your cache inside your wordpress. For sure if you want and have enough system ressources do that like you said every 24 hours :)
Crypto Coupons avatar
cy flag
how to purge the cache ?
Crypto Coupons avatar
cy flag
is there a way to reverse all this curls ?
mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.