Comment by kijin | Hacker News Reader

Comment by kijin | Hacker News Reader

The restart behavior can be a problem when there's high concurrency, long-running requests, or even worse, both.

First, the "opcache.force_restart_timeout" setting kills scripts upon reaching a timeout, independently of the well-known "max_execution_time" php.ini directive, but only from time to time. It is unpredictable and doesn't leave a comparable message in the error_log.

Also, for a long time in the PHP 7.x days there was (still is?) a race condition in the SHM-to-file-and-back-to-SHM sequence that caused weird errors in programs that were executed at an unfortunate moment during the restart. A single bit would flip in a class name, function name or filename, and the script with the bit-flipped name would be cached instead of the original. Imagine the_post() in WordPress suddenly changing to the_qost(). The entire site would grind to a halt until the corrupted entry was evicted from the cache, and in the meantime, error messages would refer to nonexistent functions in nonexistent files. Moreover, because the bit flip can happen to any name, it's hard to search for a clue -- a real head-scratcher! As far as I'm aware of, none of the related issues in the PHP bugtracker has been conclusively fixed.

It's hard to check whether old opcache bug reports have been resolved, because these issues tend to be non-reproducible. It's quite likely that they are, as opcache is tested much more extensively now than it was a few years ago. Back then, the file cache had essentially no automated tests, while nowadays we check our test suite with all four file cache combinations (written with/without SHM, read with/without SHM).

Restarts are an area that still doesn't have much in terms of automated tests, but at least it's covered by fuzz testing.

You mitigate this by taking boxes out of your response pool, deploying, then putting them back in. I generally do 50% deploys at a time.

Solves this issue elegantly.

Been doing that for a while, but it's a workaround at best insofar as the opcache bug is concerned. I wouldn't call it elegant. A thing that calls itself a cache should be able to handle evictions without throwing up.

It's also the kind of workaround that isn't applicable to the vast majority of PHP sites out there: beginners running WordPress on a single box. One day, you update a few plugins like a good security-minded person should do, and this causes a bunch of cache invalidations. Bam! The site is down and the error messages aren't helpful. It really doesn't help bolster PHP's reputation as a stable, beginner-friendly language.

Shared hosting is kinda bottom barrel these days. Even one off static sites I'd isolate into $5/month digitalocean droplets over shared hosting precisely for this reason. PHP is optimizing more for large sites that use multiple servers because frankly that use case is more important.

The opcache bug is actually more likely to hit VPS and dedicated servers, because most sites on shared hosting don't have enough concurrency to trigger the race condition. The last time I helped a customer fix it, they had a cluster of EPYC servers. Their app needed scratch space for generated code, so even offline deployments wouldn't prevent garbage from accumulating in the local opcache, eventually causing a restart.

Even up here in the clouds, there are many times more people who rely on one-click WordPress-preinstalled droplets than there are people who understand how to correctly juggle multiple droplets to deploy updates without downtime. Besides, a bug in the PHP runtime is a bug in the PHP runtime, regardless of whether the user is following today's best practices.

Yup, it's a really really bad bug that Vimeo ran into frequently in 2019-early 2020.

Migrating to K8s-based deployments was the effective workaround, but before that it was not fun to have that sort of unreliability.

Have you links to the bugs?

Reply on news.ycombinator.com