Score:1

Apache httpd worker crash (IUS httpd24u on Centos 7)

in flag

I'm running the httpd24u package from IUS on two Centos 7 servers. The version identifier for the package is "2.4.52-1.el7.ius".

For the last week or so, we've been seeing intermittent broken HTTP responses from the servers. The browser reports "net::ERR_CONNECTION_CLOSED", and the server error log includes output like:

*** Error in `/usr/sbin/httpd': free(): invalid next size (fast): 0x00007fb394015f10 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x81329)[0x7fb3cf231329]
/lib64/libcrypto.so.10(CRYPTO_free+0x1d)[0x7fb3c7b1396d]
/lib64/libcrypto.so.10(EVP_MD_CTX_cleanup+0xe9)[0x7fb3c7bd0a69]
/lib64/libcrypto.so.10(+0x1275ee)[0x7fb3c7bcc5ee]
/lib64/libssl.so.10(ssl23_accept+0x39)[0x7fb3c7f37409]
/etc/httpd/modules/mod_ssl.so(+0x18352)[0x7fb3c0122352]
/etc/httpd/modules/mod_ssl.so(+0x19ed6)[0x7fb3c0123ed6]
/etc/httpd/modules/mod_ssl.so(+0xdad0)[0x7fb3c0117ad0]
/usr/sbin/httpd(ap_run_process_connection+0x40)[0x55a2b89abe10]
/etc/httpd/modules/mod_mpm_event.so(+0x714a)[0x7fb3c25db14a]
/etc/httpd/modules/mod_mpm_event.so(+0x7d9e)[0x7fb3c25dbd9e]
/lib64/libpthread.so.0(+0x7ea5)[0x7fb3cf789ea5]
/lib64/libc.so.6(clone+0x6d)[0x7fb3cf2aeb0d]
======= Memory map: ========
55a2b8955000-55a2b89e1000 r-xp 00000000 08:03 1879095252                 /usr/sbin/httpd
55a2b8be0000-55a2b8be3000 r--p 0008b000 08:03 1879095252                 /usr/sbin/httpd
55a2b8be3000-55a2b8be5000 rw-p 0008e000 08:03 1879095252                 /usr/sbin/httpd
55a2b8be5000-55a2b8be9000 rw-p 00000000 00:00 0 
55a2b9fea000-55a2ba3a0000 rw-p 00000000 00:00 0                          [heap]
55a2ba3a0000-55a2ba46c000 rw-p 00000000 00:00 0                          [heap]
7fb34c000000-7fb34c021000 rw-p 00000000 00:00 0 
7fb34c021000-7fb350000000 ---p 00000000 00:00 0 
7fb354000000-7fb354074000 rw-p 00000000 00:00 0 
7fb354074000-7fb358000000 ---p 00000000 00:00 0 
7fb358000000-7fb358021000 rw-p 00000000 00:00 0 

(The memory map continues for several pages.)

I don't know how to interpret this error message. I've learned that the "free(): invalid next size" indicates a memory error (trying to free() unallocated memory or double-free()ing the same memory).

There are a couple of possibly correlated recent events that I can think of:

  1. The httpd24u package updated fairly recently. All of httpd's files (modules, etc) have a last modified time of Jan 7. However, I don't think that our troubles go back that far.
  2. We turned on HTTP/2 recently (Feb 3). This also involved switching httpd from the "worker" to the "event" MPM, at least on one of the servers.

One obvious next troubleshooting step is to disable HTTP/2 and see if the errors go away. However, even if that helps, we don't want to run without HTTP/2 for very long because of the performance improvements it offers. What else can I try in this situation?

--

Update: Reverting to 2.4.51 (with no other changes) has improved our servers' reliability, I think because this bug was created in 2.4.52: https://bz.apache.org/bugzilla/show_bug.cgi?id=65769

Score:0
mp flag

This httpd random crashing issue has also been affecting our CentOS server. It also seems to relate to the MPM worker module. For no apparent reason - and certainly not under load - the httpd crashes, even though the server reports it as still running and OK. UptimeRobot rapidly reports all the sites on the server going down, which is usually quite reliable. It has happened twice today already and it first happened last Friday, 11 February. It was not happening at all before that.

in flag
Thanks for letting me know! Other people have been having some issues too. I'll post an answer about that.
mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.