Memcached corruption overview:

In a working implementation a form submission would cause the Apache server configured with memcached to hang after the form was successfully submitted and processed. The 'httpd/error_log' file repeatedly shows one of the following:

'child pid 26031 exit signal Segmentation fault (11)', or
'*** glibc detected *** /usr/sbin/httpd: double free or corruption (!prev): 0x0a3c5bf0 ***'

Both the memcached and httpd daemons had to be restarted for the other pages of the site to work properly again.

This problem can be consistently repeated, resulting in the same error. More importantly the problem was the result of programming oversight.

Memcached corruption analysis:

Analysis revealed that the data object being stored in memcached was being corrupted. Data corruption was caused by passing an object property to another function, where the object property relied on the '__get()' method of the object to return the property's value. For example:

The function:

function do_something( $param1, $param2 ) {
…..
}

Was invoked as:

$result = do_something( $obj->some_value, $string );

Where 'some_value' was not a defined property of the object. As a result, when 'some_value' is accessed the object invokes the magic __get() method to handle the request. It appears that functions do not handle other functions in the argument list, and in calling the magic __get() method generates unpredictable, yet consistent results - in this case a corrupted data object.

This in and of itself is not what caused the system to hang. The result was simply to corrupt the data object that was cached in memcached. The form data was processed and a new record created in the database.

The server would hang when the next page was loaded that relied on the data object (now corrupted) that was stored in memcached. The corrupt data would remain in memcached until it expired or until the memcached daemon was respawned.

Memcached corruption solution:

Before invoking a function create a temporary buffer for object properties that are accessed with the magic __get() method (or any magic method more than likely) of an object as follows:

This causes unpredictable results:
$result = do_something( $obj->some_value, $string );

This works as expected:
$my_temp_value = $obj->some_value;
$result = do_something( $my_temp_value, $string );

Passing the object in it's entirety would work also, such as:
$result = do_something( $obj, $string );

and then accessing the desired property within the function itself, even if it relies on the magic __get() method.

Things to keep in mind:

• If memcached and httpd are functioning normally with other code in the site it's probably not an error related to either of these two, but rather something in the flow the current script that manifests itself in data being corrupted in memcached and ultimately a subsequent script failing. Don't waste time recompiling or messing with memcached or apache.

• Go through the script causing this error and the preceding scripts to determine exactly what is causing the corruption of data being stored in memcached. Debug line by line if you have to and verify all the result.

• It is not always readily apparent what is causing the problem. In the case above passing an object property ($obj->some_value) to a function seems correct and would be easy overlook when just scanning the code. In fact, it took a few moments to consider why it would cause a problem even when the debug output clearly showed where the problem occurred.

Contributors:

Ben Nduati