[07:07:08] https://cdn.discordapp.com/attachments/1006789349498699827/1420667925596999680/image.png?ex=68d63b9c&is=68d4ea1c&hm=ad64fb185e190e231b0f34751c6f8e1e287809b6e1b9fd100712f0ab981d872b& [07:07:10] oh? [07:11:39] [1/2] Error 503 Backend fetch failed via cp201 at Thu, 25 Sep 2025 07:11:26 GMT [07:11:39] [2/2] Varnish XID 598770006, serving [REDACTED] [07:12:43] oh -ops is getting noisy with icinga, good(?) [07:13:21] its down right now apparently so just give her a bit [07:13:25] occasional 502 from Cloudflare side and ^, Ray id `Cloudflare Ray ID: 9848b777d93d84e7 ` [07:13:39] Oh yeah I have been fixing side of it for a while :P [07:13:52] (That is, I stopped being sysadmin in 2018 lol) [07:24:00] https://cdn.discordapp.com/attachments/1006789349498699827/1420672171717562408/image0.jpg?ex=68d63f90&is=68d4ee10&hm=8f7b70dceb99785d9c1d302de841a2a83580797de907c71de2002e1fb9b22fac& [07:24:20] skyrocketed [07:24:52] now givin me DBConnection error, new one :P [07:29:04] I'm getting 'MediaWiki internal error' and 'Cannot access the database' alternately [07:29:49] Is this on the radar? [07:30:14] Iโ€™m not seeing any such errors [07:31:19] Weird [07:31:40] I'm experiencing slowness on some pages but not others [07:31:44] I got this before the edit restored [07:32:15] [1/3] - 502 from Cloudflare [07:32:16] [2/3] - 503 from Varnish [07:32:16] [3/3] - DBConnectionError and even ConnectionRefused (Database) [07:33:53] Yeah, I was seeing the same earlier [07:43:11] [1/2] I wonder if it has anything to do with increased bot activity. I'm seeing bots cause slowdowns on moegirl.uk recently: unique visitors shot up from 50k-100k to more than 700k. [07:43:11] [2/2] https://cdn.discordapp.com/attachments/1006789349498699827/1420676996945678366/image.png?ex=68d6440e&is=68d4f28e&hm=26f76333a88fe7dfd446aa74ef1672de79dcb96ec9358b2e94712396dd228c19& [07:44:02] Itโ€™s a bug with MirahezeFunctions. We randomly get flooded with thousands of queries for its most expensive function [07:44:18] CosmicAlpha said heโ€™d look into it when he returns [07:45:09] Ah that makes sense. I guess bot traffic would result in slowness for everyone instead of database downage. [09:17:02] Well, Miraheze has been slow for days now ๐Ÿ™ Really frustrating to get anything done. [09:20:48] so its not just me who feels its slow [09:41:31] the servers do be under quite a bit of load, yes [09:44:53] I just noticed mw173 has swap? Probably shouldn't since most other mw servers don't have since we don't want them swappin' [09:45:57] actually it's not just mw173, it's more like mw 1*3 [09:57:37] https://cdn.discordapp.com/attachments/1006789349498699827/1420710827312283709/IMG_1240.png?ex=68d66390&is=68d51210&hm=5b030b25e4ca6b24489b741aaf1f3fd75dda9b1e317e980222530c0224b4768d& [10:01:34] Yeah me too [10:02:07] Yeah there are no any notice for any problems [10:10:26] looks to be a 'load' issue. [14:06:35] I've submitted three patches to CommentStreams and the first two points will be fixed once somebody from BlueSpice merges them [14:47:29] CA will be greeted by lots of MH work once he gets back home. [14:47:54] Lol [14:56:08] thanks! [18:52:59] [1/32] https://lists.wikimedia.org/hyperkitty/list/wikitech-l@lists.wikimedia.org/thread/OXB3J6GLNIU425VMGTCU26FFU4AMHPNT/ [18:52:59] [2/32] Andre Klapper @ 25 Sep 2025 10:48 a.m. [18:52:59] [3/32] I am excited to announce that this is an exciting announcement. [18:53:00] [4/32] Today the Wikimedia Release Engineering Team deployed a larger software upgrade of https://phabricator.wikimedia.org [18:53:00] [5/32] [Check the hyperkitty link for proper details on this section. Details were removed due to discord limits.] [18:53:00] [6/32] You now better enjoy [18:53:00] [7/32] * Performance improvements: [18:53:01] [8/32] * Search user interface: [18:53:01] [9/32] * User Profile pages: [18:53:01] [10/32] * Project Workboards: [18:53:02] [11/32] * Herald automated actions: [18:53:02] [12/32] * General text input: [18:53:02] [13/32] * Conduit API: [18:53:03] [14/32] * User Preferences: [18:53:03] [15/32] * Projects: [18:53:04] [16/32] * Project Report Charts: [18:53:04] [17/32] * Files: [18:53:05] [18/32] * Mobile: Support zooming on pages (e.g. when looking at image files) [18:53:05] [19/32] * Mobile: Detection of Firefox on Android (to adjust content width) [18:53:06] [20/32] * Tokens: Allow filtering given tokens by token types [18:53:06] [21/32] * Wikibugs IRC bot: Report the color of milestones correctly [18:53:07] [22/32] * (admins only) Allow changing email of bot and mailing list accounts [18:53:07] [23/32] * Numerous crasher fixes [18:53:08] [24/32] * Some accessibility improvements (ARIA labels, titles, etc) [18:53:08] [25/32] * Cleaner CSS (less noise in your browser's Developer Tools' console) [18:53:09] [26/32] Our Downstream dependency trees of tasks are at [18:53:09] [27/32] https://phabricator.wikimedia.org/T386558 and https://phabricator.wikimedia.org/T393840 [18:53:10] [28/32] I'd like to thank the Wikimedians BlankEclair, mainframe98, MatmaRex, pppery, taavi, valerio.bozzolan, xtex, who contributed patches to the upstream project at https://we.phorge.it. (Hope I forgot nobody?) [18:53:10] [29/32] As usual, if you have comments or questions about Phab, please bring them up on https://www.mediawiki.org/wiki/Talk:Phabricator/Help ! [18:53:11] [30/32] Cheers, [18:53:11] [31/32] andre (on behalf of the Wikimedia Release Engineering Team) [18:53:12] [32/32] [Truncated due to discord limits. hyperkitty link has full details] [18:53:32] `I am excited to announce that this is an exciting announcement.` [19:08:21] does he ever not? ๐Ÿ˜‚ [19:24:57] Nein [19:58:11] nooit [19:58:46] aldrig [21:47:44] [1/2] I don't know if the Miraheze admins added any blocks, but on the wiki I have admin on (electowiki), the bot traffic quieted down for a few days but is now back. [21:47:45] [2/2] It's all Chinese IPs claiming to be Mac OS 10.15.