Score:0

Error 500 Internal Server Error from varnish, but no error from php-fpm or httpd or nginx

cn flag
  1. checked the logs /var/log/website/php_errors.log, nothing was found
  2. checked why it was returning an error and it's because of the backend somehow and varnish hit sub vcl_error as a result
  3. it used to work yesterday and with no code change and no server change, it stopped working
  4. no exception logs from php
  5. no log from httpd that explains the shutdown, latest logs come from 3 days ago when the server was fine
Tasks: 114 total,   1 running, 113 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.0%us,  0.0%sy,  0.0%ni, 99.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.3%st
Mem:   1920224k total,  1490924k used,   429300k free,    19092k buffers
Swap:        0k total,        0k used,        0k free,   416996k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                        
    1 root      20   0 19232 1532 1240 S  0.0  0.1   0:01.77 init                                                                            
    2 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kthreadd                                                                        
    3 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration/0                                                                     
    4 root      20   0     0    0    0 S  0.0  0.0   0:00.00 ksoftirqd/0                                                                     
    5 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 stopper/0                                                                       
    6 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 watchdog/0                                                                      
    7 root      20   0     0    0    0 S  0.0  0.0   0:00.03 events/0                                                                        
    8 root      20   0     0    0    0 S  0.0  0.0   0:00.00 events/0                                                                        
    9 root      20   0     0    0    0 S  0.0  0.0   0:00.00 events_long/0                                                                   
   10 root      20   0     0    0    0 S  0.0  0.0   0:00.00 events_power_ef                                                                 
   11 root      20   0     0    0    0 S  0.0  0.0   0:00.00 cgroup                                                                          
   12 root      20   0     0    0    0 S  0.0  0.0   0:00.00 khelper                                                                         
   13 root      20   0     0    0    0 S  0.0  0.0   0:00.00 netns                                                                           
   14 root      20   0     0    0    0 S  0.0  0.0   0:00.00 async/mgr                                                                       
   15 root      20   0     0    0    0 S  0.0  0.0   0:00.00 pm                                                                              
   16 root      20   0     0    0    0 S  0.0  0.0   0:00.00 xenwatch                                                                        
   17 root      20   0     0    0    0 S  0.0  0.0   0:00.00 xenbus                                                                          
   18 root      20   0     0    0    0 S  0.0  0.0   0:00.00 sync_supers                                                                     
   19 root      20   0     0    0    0 S  0.0  0.0   0:00.00 bdi-default                                                                     
   20 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kintegrityd/0                                                                   
   21 root      20   0     0    0    0 S  0.0  0.0   0:00.02 kblockd/0                                                                       
   22 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kacpid                                                                          
   23 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kacpi_notify                                                                    
   24 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kacpi_hotplug                                                                   
   25 root      20   0     0    0    0 S  0.0  0.0   0:00.00 ata_aux                                                                         
   26 root      20   0     0    0    0 S  0.0  0.0   0:00.00 ata_sff/0                                                                       
   27 root      20   0     0    0    0 S  0.0  0.0   0:00.00 ksuspend_usbd                                                                   
   28 root      20   0     0    0    0 S  0.0  0.0   0:00.00 khubd                                                                           
   29 root      20   0     0    0    0 S  0.0  0.0   0:00.03 kseriod                                                                         
   30 root      20   0     0    0    0 S  0.0  0.0   0:00.00 md/0 

Got this before I fixed it:

Error 500 Internal Server Error
Internal Server Error

Guru Meditation:
XID: 763812797
  1. a reboot solved the issue, but how do I check what's causing apache to crash. Is there a way to log what's happening to the server? I am thinking it's a cronjob or some kind of script that takes a lot of memories and then crash the server? What might be the possible reasons why it crashed?
Michael Hampton avatar
cz flag
That 500 error comes from varnish, not Apache or PHP. You should check its logs to find out what is happening.
mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.