Join Log in
Score:0
Lorenzo S avatar Server

Google HTTP Load Balancer and Google Cloud Storage: Apparently random 403 in PUT requests

in flag
Lorenzo S

I have Google Cloud Storage configured with a CNAME according to the documentation, so basically I have my Google Load Balancer handling the requests. In almost 2 years with this configuration I didn't have a problem (traffic is higher now, so the problem was probably obfuscated...I cannot know this). I am using signed URLs.

I have started having some apparently random 403 responses with the PUT operation on the storage, so basically with the file upload operation. Apparently random because it works most of the time (I would say 99%), but the problems are becoming evident in the log viewer and of course in the support requests from our users.

Here's a log entry (some details obfuscated):

{
  "insertId": "<id_here>",
  "jsonPayload": {
    "@type": "type.googleapis.com/google.cloud.loadbalancing.type.LoadBalancerLogEntry",
    "statusDetails": "response_sent_by_backend"
  },
  "httpRequest": {
    "requestMethod": "PUT",
    "requestUrl": "https://storage.mydomain.it/path/2/2021-06-18/m_808/2_2021-06-18-09-09-24_m808_bdb6b3.png?GoogleAccessId=ocr-account%40mydomain.iam.gserviceaccount.com&Expires=1624000464&Signature=<signature_here>",
    "requestSize": "1188472",
    "status": 403,
    "responseSize": "658",
    "userAgent": "Apache-CXF/3.2.5",
    "remoteIp": "88.86.183.XXX",
    "latency": "0.107676s"
  },
  "resource": {
    "type": "http_load_balancer",
    "labels": {
      "backend_service_name": "",
      "forwarding_rule_name": "mydomain-balancer-1-forwarding-rule-2",
      "url_map_name": "mydomain-balancer-1",
      "target_proxy_name": "mydomain-balancer-1-target-proxy-2",
      "project_id": "my_project_id",
      "zone": "global"
    }
  },
  "timestamp": "2021-06-18T07:09:25.037700Z",
  "severity": "WARNING",
  "logName": "projects/myproject/logs/requests",
  "trace": "projects/myproject/traces/cc8123c2a42e70d8d002c84efe15e5de",
  "receiveTimestamp": "2021-06-18T07:09:25.668260183Z",
  "spanId": "e4b4a200767c9de0"
}

response_sent_by_backend is probably an indication that the storage itself has sent a 403 error. Now, there are a lot of possible reasons for a 403 error, but the log viewer is not giving any other information.

How can I approach this problem in order to understand what's happening?

332
0 + 3
John Hanley avatar
cn flag
John Hanley
1) The expiration time is too short. In your log example, the URL expires within five minutes from the log entry. You should be able to get more details in the Cloud Storage logs. 2) The user agent is Apache. Is this your service? If yes, make sure the date, time and time zone are set correctly.
0
Reply
Lorenzo S avatar
in flag
Lorenzo S
Hi @JohnHanley, thanks for your reply. For the expiration problem, I see that the timestamp of the log is "Fri Jun 18 2021 09:09:25 GMT+0200", and the URL expiration date is "Fri Jun 18 2021 09:14:24 GMT+0200 (Central European Summer Time)" so I think we're good in this sense. For the user agent: this can be an interesting observation. I will ask the partner that is using the service (in fact, it's the only partner having this problem). Thanks!
0
Reply
John Hanley avatar
cn flag
John Hanley
Your comment leads me to believe your partner's system time is incorrect. The date/time in a request must be correct for Signed URLs. What is the expiration time for your Signed URLS (time from issuance)? Google rotates the private keys so limit the total valid time to less than 12 hours. https://cloud.google.com/storage/docs/access-control/signed-urls#signing-iam
0
Reply
ธงชาติไทย
Elon Musk
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.