[13:48:42] How can it be that PHP-FPM executes an old version of LocalSettings.php? I've even reloaded the FPM process via SIGUSR2. I'm seeing nginx error logs that are based on an old state of LocalSettings.php (from more than an hour ago). [13:49:21] thankfully it's only a warning [14:44:16] taylan: try restarting fpm fully [14:44:29] It's likely opcache [14:50:19] RhinosF1: shouldn't the opcache for the file be invalidated when it's edited? [14:50:35] Should [14:51:59] wouldn't it be a pretty big problem if that doesn't work reliably? [14:52:59] what also confuses me is that changes to the file do seem to take effect, and yet I constantly get new nginx error log entries that could only happen if the file was in a much older state... [14:54:28] "2024/01/07 15:53:12 [error] 2454493#2454493: *20389324 FastCGI sent in stderr: "PHP message: PHP Warning: Undefined array key "HTTP_X_FORWARDED_FOR" in /var/www/bg3/w/LocalSettings.php on line 20", ..." There's no mention of HTTP_X_FORWARDED_FOR on line 20 anymore... :\ [14:55:12] It was briefly there 2 hours ago to test something before I changed it again. Very confusing. [15:38:31] wtf, even a full restart of php-fpm didn't resolve it... [15:38:43] I wonder if this has something to do with nginx [15:43:02] A full restart of nginx didn't help either. This is extremely interesting. Does MediaWiki store a cached version of LocalSettings.php somewhere persistently? [15:43:49] MW doesn't [15:43:54] PHP opcache could [15:43:59] Tried touching LocalSettings.php? [15:44:29] well I've made many edits to it since, and I can confirm this updates its mtime (that's what's shown by ls by default, right?) [15:44:45] right, can confirm via 'stat LocalSettings.php' [15:46:40] theory: when Nginx serves a cached version of a past response from PHP (I use fastcgi_cache), perhaps the cached "response" includes the stderr of the PHP process back when the response was cached? [15:46:52] because the fcgi cache is persistent [16:47:57] MediaWiki doesn't seem to set any cache headers for Special:MyContributions, which may cause a CDN to cache it and give you a response meant for another user. [16:48:08] Or did I misconfigure something? [17:34:31] I get cache-control: private, must-revalidate, max-age=0 [17:35:05] (MediaWiki 1.39) [17:36:47] Oh wait, no. I was looking at the headers of the target redirect. The special page itself doesn't seem to set cache headers, and indeed it gets cached [17:37:41] T272431 [17:38:04] wm-bot looks dead... https://phabricator.wikimedia.org/T272431 - Special:MyTalk and Special:MyContributions getting cached in varnish on third party wiki [17:39:18] 3-year old bug. Sigh... [17:40:43] T1 [17:40:46] T123 [17:40:47] T123: Turn on "diffusion.allow-http-auth" - https://phabricator.wikimedia.org/T123 [17:40:48] I can't seem to view it, says access denied [17:41:01] Vulpix: thats a security bug [17:41:18] (the one Vulpix posted I mean; the other one I can view) [17:41:43] Yes [17:41:47] It's a security bug [17:42:41] nice, lol [17:43:08] What's your phab username [17:43:14] TaylanUB [17:43:16] http://wm-bot.wmcloud.org/dump/%23mediawiki.htm [17:43:16] @info [17:43:50] that's the LDAP username, to be clear [17:44:02] otherwise displays as "Taylan (Taylan)" [17:45:27] taylan: I added you [17:45:42] There's nothing private in the task that you don't already know [17:45:55] ty, can see now [17:46:16] huh, I didn't notice it's a security bug... [19:20:50] Is it recommended to enable the MariaDB query cache? [20:19:48] I've gotten several instances of an internal error when trying to view the page difference for a page that's brand new. [20:20:38] First were generated by the Discord Wiki bot, but now there's some being generated by the wiki's own "you made for nth edit" notifications. [20:21:11] Old notifications don't show this error though, is this some configuration error I can fix? [20:33:04] !debug | UFFR [20:33:04] UFFR: For information on debugging (including viewing errors), see https://mediawiki.org/wiki/Manual:How_to_debug . A list of related configuration variables is at https://mediawiki.org/wiki/Manual:Configuration_settings#Debug.2Flogging Also see https://mediawiki.org/wiki/Manual:Errors_and_symptoms [20:34:27] There's no magic "do X to solve errors". Having a *descriptive* error message, however, can help to narrow down the cause and bring a possible solution [20:36:46] The error is a bit hard to read, but times it has appeared are all consistent. Here's an example: [20:36:49] https://paste.gg/p/anonymous/9bae16936d1c45c98d2dd6da7138136e [20:37:02] Fairly certain all of the errors are mostly the same. [20:38:50] that's happening inside a library from the vendor directory. Is it a new install, or was it upgraded recently and errors are appearing since the upgrade? [20:39:40] Upgraded from 1.40.0 to 1.41.0 just yesterday. [20:41:15] How did you upgrade? Downloaded tarball from mediawiki.org? or using git? [20:42:21] Tarball [20:42:40] Partially via the web script and the database was updated from the CLI. [20:43:28] Checksums between the seemingly affected PHP scripts in [vendor/oojs/oojs-ui/php/] match though. [20:45:52] were you extracting the tarball on a clean (empty) directory or over the old install? It's recommended to extract on an empty directory (at least for major releases), since some files from the old install may get there causing problems [20:46:39] It was first extracted to an unrelated directory before being copied over to the previous install. [20:46:41] Since the error seems to be coming from VisualEditor hooks, also be sure you haven't copied over extensions from the old install (or at least, not bundled extensions or extensions that need updating) [20:47:22] I'm also fairly certain I updated that extension in particular, but I'll check again. [20:47:44] VisualEditor comes bundled with the 1.41 tarball, it doesn't need a separate download [20:49:10] The only file that needs copied from the old install is LocalSettngs.php. Everything else should be downloaded and not overwritten. If your uploads are inside the MediaWiki install directory, you'll have to move that dir too [20:53:16] Should I replace the extension directory entirely or perform a recursive checksum to confirm differences first? [20:57:09] You can redownload everything (mediawiki and extensions) and then proceed to extract/copy then to a new empty directory. Copy LocalSettings.php from the old directory, and then swap directories (move the current to _old for example, and the new to the current) [23:29:01] I finally got back inside and did the proper upgrade by simply substituting the whole thing with the new version and only copying what is required, but that error still shows up for diffs on brand new pages. [23:29:14] Good news is that the wiki behaves as it should everywhere else though.