[04:53:59] I've encountered two Wikimedia\Rdbms\DBUnexpectedError fatals in exception handler in the last two days, what's up with that? [04:54:42] https://dpaste.org/yziSX/raw [05:01:51] just got two as well [05:02:10] Original exception: [ee791be7-dd86-4afd-ab67-c2f175620b95] 2025-04-02 05:01:29: Fatal exception of type "Wikimedia\Rdbms\DBUnexpectedError" [05:05:55] Amir1: ^ [05:13:41] burst of errors and open connections on x1 https://grafana.wikimedia.org/d/000000278/mysql-aggregated?orgId=1&from=1743560004389&to=1743570804389&viewPanel=9 [06:08:35] Morning folks, I'd appreciate some help in debugging an issue with the Sitenotice on rowiki: it is not displayed on mobile, even though it seems to follow the rules laid out in https://www.mediawiki.org/wiki/Manual:Interface/Sitenotice (only the image has "nomobile" class). [06:08:45] The code: https://ro.wikipedia.org/w/index.php?title=MediaWiki:Sitenotice&action=edit [06:09:39] I tried logged in, logged out, asked a friend, in incognito mode, with several browsers, still doesn't display [06:10:10] I know I have seen other entries in 2024, so I assume $wgMinervaEnableSiteNotice is set [06:19:39] strainu: doesn't look like it? https://github.com/wikimedia/operations-mediawiki-config/blob/bd40a4b9b272f330f11f6bdd89c4add8347836df/wmf-config/skin-Minerva.php#L158-L185 [06:24:14] they may have been central notice messages targeted to the wiki, as that doesn't rely on that setting from memory [06:37:54] Hmmm, so the part with "which is now the default since MediaWiki 1.36" is not actually true and we have to explicitly ask? [06:56:56] it seems it's set to true by default in the extension (although the README says otherwise). but it's disabled for wmf wikis by default. [06:58:21] Thank you for your help folks, I'll start a discussion and then a site request! [08:53:03] ori: [db2395f6-9a2f-46c1-a6b3-ae82a6e9088f] /wiki/*** Wikimedia\Rdbms\DBUnexpectedError: Database servers in extension1 are overloaded. [08:53:11] ref T390510 [08:53:12] T390510: Fatal DBUnexpectedError: "Database servers in extension1 are overloaded" (30 March 2025) - https://phabricator.wikimedia.org/T390510 [09:45:52] ori: Thanks. I will investigate it's T390811 [09:45:52] T390811: Wikimedia\Rdbms\DBUnexpectedError: Database servers in extension1 are overloaded (2025-04-02) - https://phabricator.wikimedia.org/T390811 [14:31:22] Hey, noticed you're having a bit problem. Are you also seeing things that just look like normal unmodified Google Chrome from basically every ASN on the planet? [14:31:26] bot problem* [14:32:07] [[Tech]]; MediaWiki message delivery; /* Temporary Accounts: how to update your code */ new section; https://meta.wikimedia.org/w/index.php?diff=28484892&oldid=28475674&rcid=34894904 [14:33:05] A problem relating to what? [14:33:35] https://diff.wikimedia.org/2025/04/01/how-crawlers-impact-the-operations-of-the-wikimedia-projects/ abusive crawler bots [14:34:05] I think the use of fake UA is prevelant all over the place [14:37:21] <_joe_> Xe: the problem isn't of easy solution or we would've :) [14:45:17] _joe_: yeah i've been fighting my own battles lol [14:45:51] <_joe_> Xe: but more importantly, we've always tried to just adopt "live and let live" as a policy, sadly it doesn't work anymore [14:46:03] <_joe_> (I'm one of the authors of the blog post, FWIW) [15:04:46] yeah, i ended up making a really hacky proof of work thing that works surprisingly well, but from what i've seen the clients just look like legit browsers at every layer of the stack i've checked (TLS fingerprints, HTTP/2 fingerprints, ordering of headers in the requests, etc.) [15:06:13] i'm half-jokingly considering using encrypted media extensions as a way to turn copyright holders against the AI bros when they try to break DRM to bypass it lol [15:13:26] lol [17:30:26] [[Tech]]; TagUser; /* Temporary Accounts: how to update your code */ Reply; https://meta.wikimedia.org/w/index.php?diff=28485650&oldid=28484892&rcid=34896380 [17:31:30] [[Tech]]; TagUser; /* Temporary Accounts: how to update your code */; https://meta.wikimedia.org/w/index.php?diff=28485651&oldid=28485650&rcid=34896385 [17:55:02] [[Tech]]; SGrabarczuk (WMF); /* Temporary Accounts: how to update your code */ Reply; https://meta.wikimedia.org/w/index.php?diff=28485763&oldid=28485651&rcid=34896659 [19:48:04] [[Tech]]; TagUser; /* Temporary Accounts: how to update your code */ Reply; https://meta.wikimedia.org/w/index.php?diff=28486110&oldid=28485763&rcid=34897809