[00:02:03] Project beta-scap-sync-world build #35427: 15ABORTED in 7 min 50 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/35427/ [00:05:38] 10Continuous-Integration-Infrastructure, 10DC-Ops, 10SRE, 10netops, 10ops-codfw: DRAC firmware upgrades codfw (was: Flapping codfw management alarm ( contint2001.mgmt/SSH is CRITICAL ))) - https://phabricator.wikimedia.org/T283582 (10Papaul) @hashar since Monday is a Holiday, let is do this on the 18th a... [01:03:54] Project beta-scap-sync-world build #35428: 15ABORTED in 1 hr 0 min: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/35428/ [01:05:35] 10Release-Engineering-Team (Next), 10Patch-For-Review, 10Release, 10Train Deployments: 1.38.0-wmf.17 deployment blockers - https://phabricator.wikimedia.org/T293958 (10dduvall) [01:47:29] !log revert to scap 4.1.1-1+0~20220113154148.133~1.gbp6e3a17 in beta [01:47:30] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [01:50:30] (03PS1) 10Ahmon Dancy: Revert "beta-code-update-eqiad: Use scap prep auto" [integration/config] - 10https://gerrit.wikimedia.org/r/753832 [01:50:44] (03PS1) 10Ahmon Dancy: Revert "Remove beta-mediawiki-config-update-eqiad job" [integration/config] - 10https://gerrit.wikimedia.org/r/753833 [01:52:46] (03CR) 10Ahmon Dancy: [C: 03+2] Revert "beta-code-update-eqiad: Use scap prep auto" [integration/config] - 10https://gerrit.wikimedia.org/r/753832 (owner: 10Ahmon Dancy) [01:53:15] (03CR) 10Ahmon Dancy: [C: 03+2] Revert "Remove beta-mediawiki-config-update-eqiad job" [integration/config] - 10https://gerrit.wikimedia.org/r/753833 (owner: 10Ahmon Dancy) [01:55:22] (03Merged) 10jenkins-bot: Revert "beta-code-update-eqiad: Use scap prep auto" [integration/config] - 10https://gerrit.wikimedia.org/r/753832 (owner: 10Ahmon Dancy) [01:56:31] (03CR) 10jerkins-bot: [V: 04-1] Revert "Remove beta-mediawiki-config-update-eqiad job" [integration/config] - 10https://gerrit.wikimedia.org/r/753833 (owner: 10Ahmon Dancy) [01:58:03] (03CR) 10Ahmon Dancy: [C: 03+2] "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/753833 (owner: 10Ahmon Dancy) [02:00:04] (03CR) 10jerkins-bot: [V: 04-1] Revert "Remove beta-mediawiki-config-update-eqiad job" [integration/config] - 10https://gerrit.wikimedia.org/r/753833 (owner: 10Ahmon Dancy) [02:02:18] (03CR) 10Ahmon Dancy: [C: 03+2] "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/753833 (owner: 10Ahmon Dancy) [02:04:55] (03CR) 10jerkins-bot: [V: 04-1] Revert "Remove beta-mediawiki-config-update-eqiad job" [integration/config] - 10https://gerrit.wikimedia.org/r/753833 (owner: 10Ahmon Dancy) [02:06:32] (03PS2) 10Ahmon Dancy: Revert "Remove beta-mediawiki-config-update-eqiad job" (phase 1) [integration/config] - 10https://gerrit.wikimedia.org/r/753833 [02:09:13] (03CR) 10Ahmon Dancy: [C: 03+2] Revert "Remove beta-mediawiki-config-update-eqiad job" (phase 1) [integration/config] - 10https://gerrit.wikimedia.org/r/753833 (owner: 10Ahmon Dancy) [02:09:17] dancy: You need to `./jjb-update beta-mediawiki-config-update-eqiad` before it'll land. [02:09:32] (Unless you now have?) [02:09:55] https://integration.wikimedia.org/ci/job/beta-mediawiki-config-update-eqiad thinks no. [02:09:56] I did try that but it wasn't enough. I ended up splitting the commit.. once to define the job and one to reference it in layout.yaml (not pushed yet) [02:10:00] hmm. [02:10:07] lemme make sure I typed the right command. [02:10:12] Right, but before it will C+2 you need to push it. [02:10:16] Want me to try? [02:11:26] (03Merged) 10jenkins-bot: Revert "Remove beta-mediawiki-config-update-eqiad job" (phase 1) [integration/config] - 10https://gerrit.wikimedia.org/r/753833 (owner: 10Ahmon Dancy) [02:11:49] Not yet. :-) [02:11:53] Ack. [02:14:17] https://www.irccloud.com/pastebin/3ZUOmiWx/ [02:14:32] ok, I'm ready for you to try and see if you're more successful. [02:15:12] Hmm. [02:15:41] 1 generated, 0 updated.. sus [02:15:45] Yeah, same issue, even with '*beta*'. [02:15:49] (Where I get 21/0) [02:16:00] I have clearly broken things badly [02:16:17] Or it was never properly being created and we just didn't notice? [02:16:35] It existed earlier today before I deleted it. [02:17:38] * James_F Yes, but its repo-definition might have been broken for the past year and we never noticed. [02:17:46] We don't push changes to the beta jobs very often… [02:19:07] I'm going to try some hacking. [02:19:21] GLHF. [02:20:03] Project beta-update-databases-eqiad build #56001: 04FAILURE in 3 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/56001/ [02:20:24] Initial hack attempt failed. :-) [02:21:34] I should say :-( because of tired and want to stop working [02:24:24] +I'm [02:26:42] Oh, totally. [02:26:51] But it's a Friday, so new config things won't slip out overnight. [02:27:01] And if they do, oh well, Beta Cluster is merely Best Efforts™. [02:29:38] Alright. I'm going to leave it in this state then. Thanks for the encouragement. [02:59:09] 10Release-Engineering-Team (Next), 10Patch-For-Review, 10Release, 10Train Deployments: 1.38.0-wmf.17 deployment blockers - https://phabricator.wikimedia.org/T293958 (10AlexisJazz) T299191 Can't remember where I'm supposed to report possible blockers, but it seems serious. [03:30:07] Yippee, build fixed! [03:30:08] Project beta-update-databases-eqiad build #56002: 09FIXED in 10 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/56002/ [05:51:58] 10Release-Engineering-Team (Next), 10Patch-For-Review, 10Release, 10Train Deployments: 1.38.0-wmf.17 deployment blockers - https://phabricator.wikimedia.org/T293958 (10Legoktm) There is a report on enwp's VPT that category counts are wrong and not updating properly, affecting deletion processes. See 10Release-Engineering-Team (Next), 10Release, 10Train Deployments: 1.38.0-wmf.18 deployment blockers - https://phabricator.wikimedia.org/T293959 (10Majavah) [06:01:22] 10Release-Engineering-Team (Next), 10Patch-For-Review, 10Release, 10Train Deployments: 1.38.0-wmf.17 deployment blockers - https://phabricator.wikimedia.org/T293958 (10AntiCompositeNumber) I first noticed category update problems on Commons on 5 January, but didn't pay much mind to it as I've become desens... [06:18:21] 10Release-Engineering-Team (Next), 10Release, 10Train Deployments: 1.38.0-wmf.18 deployment blockers - https://phabricator.wikimedia.org/T293959 (10matmarex) ##### Risky Patch! 🚂🔥 * **Change**: https://gerrit.wikimedia.org/r/753557 Update OOUI to v0.43.0 * **Summary**: OOUI v0.43.0 is an unusually big rel... [09:26:54] 10Release-Engineering-Team (Next), 10Release, 10Train Deployments: 1.38.0-wmf.18 deployment blockers - https://phabricator.wikimedia.org/T293959 (10Zabe) [10:04:24] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Cloud-VPS, 10Patch-For-Review, 10cloud-services-team (Kanban): integration instances suffer from high IO latency due to Ceph - https://phabricator.wikimedia.org/T266777 (10hashar) Yes this task should have been... [10:08:21] 10Continuous-Integration-Infrastructure, 10Quibble, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)), and 2 others: Terminating MySQL takes several minutes in (Wikibase?) CI jobs - https://phabricator.wikimedia.org/T265615 (10hashar) I have... [10:23:12] 10Continuous-Integration-Infrastructure, 10Quibble, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)), and 2 others: Terminating MySQL takes several minutes in (Wikibase?) CI jobs - https://phabricator.wikimedia.org/T265615 (10dcaro) I've ad... [10:23:39] 10Continuous-Integration-Infrastructure, 10Quibble, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-10-01 to 2020-12-31 (Q2)), and 2 others: Terminating MySQL takes several minutes in (Wikibase?) CI jobs - https://phabricator.wikimedia.org/T265615 (10hashar) [11:23:49] Project mediawiki-core-doxygen-docker build #31090: 04FAILURE in 19 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/31090/ [12:17:58] Yippee, build fixed! [12:17:59] Project mediawiki-core-doxygen-docker build #31091: 09FIXED in 13 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/31091/ [12:51:05] 10Project-Admins: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706 (10Scann) Hi! I'm part of the #web2cit project (I'm doing some of the community work) and I'd like to make some edits to the project. Can I be given permission to edit? Thanks! [13:30:24] 10Project-Admins: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706 (10Aklapper) @Lena.Milenko: Hi, I've added you. //Usual disclaimer: Please follow [guidelines](https://www.mediawiki.org/wiki/Phabricator/Creating_and_renaming_projects#Creating_new_pro... [13:31:19] 10Project-Admins: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706 (10Aklapper) @Scann: Hi, editing a project itself (e.g. its description) should not require membership in #acl*Project-Admins; this is about creating projects. [13:49:34] !log Restarting all CI Docker agents via Horizon to apply new flavor settings T265615 T299211 [13:49:37] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:49:37] T299211: Request increased quota for integration Cloud VPS project - https://phabricator.wikimedia.org/T299211 [13:49:37] T265615: Terminating MySQL takes several minutes in (Wikibase?) CI jobs - https://phabricator.wikimedia.org/T265615 [13:58:25] 10Release-Engineering-Team (Next), 10Release, 10Train Deployments: 1.38.0-wmf.18 deployment blockers - https://phabricator.wikimedia.org/T293959 (10AlexisJazz) >>! In T293959#7621695, @matmarex wrote: > ##### Risky Patch! 🚂🔥 > > * **Change**: https://gerrit.wikimedia.org/r/753557 Update OOUI to v0.43.0 >... [14:55:26] 10Continuous-Integration-Infrastructure: Create first CI agent with the new disk system - https://phabricator.wikimedia.org/T290783 (10hashar) If I get it right, the bulk of the work has been done via T277078. It was to create a Bullseye image based image in order to benefit from a newer Qemu version, that ended... [14:59:12] !log Starting VM integration-agent-docker-1022 which was in shutdown state since December and is Bullseye based # T290783 [14:59:14] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:59:15] T290783: Create first CI agent with the new disk system - https://phabricator.wikimedia.org/T290783 [15:12:09] 10Release-Engineering-Team (Radar), 10MediaWiki-Vagrant, 10Parsoid, 10WMDE-Technical-Wishes-Maintenance, and 2 others: Support Parsoid/PHP in MediaWiki-Vagrant - https://phabricator.wikimedia.org/T258940 (10thiemowmde) [15:24:39] 10Continuous-Integration-Infrastructure: Create first CI agent with the new disk system - https://phabricator.wikimedia.org/T290783 (10hashar) When bringing back the instance, it has Docker shipped from Debian: docker.io 20.10.5+dfsg1-1+deb11u1 which sounds good. For the disks: ` name=lsblk NAME MAJ:MIN RM... [15:26:12] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Doing), 10ci-test-error (WMF-deployed Build Failure): TAR_ENTRY_ERROR ENOSPC: no space left on device - https://phabricator.wikimedia.org/T292729 (10hashar) [15:26:14] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen): Move all Wikimedia CI (WMCS integration project) instances from stretch to buster - https://phabricator.wikimedia.org/T252071 (10hashar) [15:28:23] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen): Move all Wikimedia CI (WMCS integration project) instances from stretch to buster/bullseye - https://phabricator.wikimedia.org/T252071 (10Majavah) [15:30:14] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Doing), 10ci-test-error (WMF-deployed Build Failure): TAR_ENTRY_ERROR ENOSPC: no space left on device - https://phabricator.wikimedia.org/T292729 (10hashar) So in short a build of `wmf-quibble-selenium-php72-docker` uses ~ 7.2 GBytes and si... [15:39:23] I’ve seen several Node out-of-memory crashes in termbox-pipeline-test builds today, e.g. https://integration.wikimedia.org/ci/job/termbox-pipeline-test/230/console [15:39:36] (“FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory”) [15:39:55] has anyone else seen these? usually they’re pretty rare as far as I’m aware [15:40:09] I think it might be related to several termbox-pipeline-test jobs running in parallel but I’m not sure [15:42:06] 10Continuous-Integration-Infrastructure: Create first CI agent with the new disk system - https://phabricator.wikimedia.org/T290783 (10hashar) Comparison of partitions: | Partition | Old disk.80 | New disk20.ephemeral40 |--|--|-- | / | 20G | 20 G | /var/lib/docker | 42.7 G | 28 G | /srv | 18.3 G | 12 G What... [15:51:32] (03PS1) 10Ahmon Dancy: Revert "Remove beta-mediawiki-config-update-eqiad job" (phase 2) [integration/config] - 10https://gerrit.wikimedia.org/r/753981 [15:53:57] (03CR) 10Ahmon Dancy: [C: 03+2] Revert "Remove beta-mediawiki-config-update-eqiad job" (phase 2) [integration/config] - 10https://gerrit.wikimedia.org/r/753981 (owner: 10Ahmon Dancy) [15:56:09] (03Merged) 10jenkins-bot: Revert "Remove beta-mediawiki-config-update-eqiad job" (phase 2) [integration/config] - 10https://gerrit.wikimedia.org/r/753981 (owner: 10Ahmon Dancy) [15:56:50] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/753981 [15:56:51] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:40:51] 10Release-Engineering-Team (Radar), 10Scap, 10Patch-For-Review, 10User-jijiki: Update Scap to perform rolling restart for all MW deploy - https://phabricator.wikimedia.org/T266055 (10dancy) Retest after the 3 second grace period commit was merged: ` dancy@deploy1002:/srv/mediawiki-staging$ scap sync-file... [17:38:45] 10Release-Engineering-Team (Next), 10Release, 10Train Deployments: 1.38.0-wmf.18 deployment blockers - https://phabricator.wikimedia.org/T293959 (10AlexisJazz) @Nardog I did a quick search and found https://en.wikipedia.org/wiki/User:Nardog/IPAInput-core.js and https://en.wikipedia.org/wiki/User:Ykhwong/Gadg... [18:02:13] 10Release-Engineering-Team (Next), 10Release, 10Train Deployments: 1.38.0-wmf.18 deployment blockers - https://phabricator.wikimedia.org/T293959 (10Volker_E) >>! In T293959#7622769, @AlexisJazz wrote: > @Nardog I did a quick search and found https://en.wikipedia.org/wiki/User:Nardog/IPAInput-core.js and http... [18:56:12] (03PS1) 10Ahmon Dancy: finalized git merge-base code [tools/scap] - 10https://gerrit.wikimedia.org/r/754017 [18:58:15] (03PS1) 10Ahmon Dancy: mirror all available extensions and skins [tools/train-dev] - 10https://gerrit.wikimedia.org/r/754018 [18:58:23] (03CR) 10jerkins-bot: [V: 04-1] mirror all available extensions and skins [tools/train-dev] - 10https://gerrit.wikimedia.org/r/754018 (owner: 10Ahmon Dancy) [18:58:57] (03CR) 10jerkins-bot: [V: 04-1] finalized git merge-base code [tools/scap] - 10https://gerrit.wikimedia.org/r/754017 (owner: 10Ahmon Dancy) [18:59:18] (03PS2) 10Ahmon Dancy: mirror all available extensions and skins [tools/train-dev] - 10https://gerrit.wikimedia.org/r/754018 [18:59:42] (03CR) 10jerkins-bot: [V: 04-1] mirror all available extensions and skins [tools/train-dev] - 10https://gerrit.wikimedia.org/r/754018 (owner: 10Ahmon Dancy) [19:01:39] (03PS2) 10Ahmon Dancy: finalized git merge-base code [tools/scap] - 10https://gerrit.wikimedia.org/r/754017 [19:04:53] (03PS3) 10Ahmon Dancy: mirror all available extensions and skins [tools/train-dev] - 10https://gerrit.wikimedia.org/r/754018 [19:05:09] (03CR) 10Ahmon Dancy: [C: 03+2] finalized git merge-base code [tools/scap] - 10https://gerrit.wikimedia.org/r/754017 (owner: 10Ahmon Dancy) [19:05:32] (03CR) 10Ahmon Dancy: [C: 03+2] mirror all available extensions and skins [tools/train-dev] - 10https://gerrit.wikimedia.org/r/754018 (owner: 10Ahmon Dancy) [19:05:51] (03Merged) 10jenkins-bot: finalized git merge-base code [tools/scap] - 10https://gerrit.wikimedia.org/r/754017 (owner: 10Ahmon Dancy) [19:06:27] (03Merged) 10jenkins-bot: mirror all available extensions and skins [tools/train-dev] - 10https://gerrit.wikimedia.org/r/754018 (owner: 10Ahmon Dancy) [20:13:48] (03PS3) 10Jforrester: Zuul: [mediawiki/extensions/Math] Add Math to the main gate [integration/config] - 10https://gerrit.wikimedia.org/r/715144 (https://phabricator.wikimedia.org/T232948) [20:14:32] (03PS3) 10Jforrester: parameter_functions: Math extension is now tarballed [integration/config] - 10https://gerrit.wikimedia.org/r/715085 (https://phabricator.wikimedia.org/T232948) [20:17:33] (03CR) 10Jforrester: "PS3: Rebased, ahead of MW release process meeting next week where hopefully this will get decided upon." [integration/config] - 10https://gerrit.wikimedia.org/r/715144 (https://phabricator.wikimedia.org/T232948) (owner: 10Jforrester) [20:17:44] (03PS2) 10Jforrester: Bundle Math extension with MediaWIki [tools/release] - 10https://gerrit.wikimedia.org/r/715082 (https://phabricator.wikimedia.org/T232948) [20:17:51] (03PS3) 10Jforrester: Bundle Math extension with MediaWiki [tools/release] - 10https://gerrit.wikimedia.org/r/715082 (https://phabricator.wikimedia.org/T232948) [20:18:29] (03PS4) 10Jforrester: Bundle Math extension with MediaWiki [tools/release] - 10https://gerrit.wikimedia.org/r/715082 (https://phabricator.wikimedia.org/T232948) [20:42:03] PROBLEM - SSH on contint1001.mgmt is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [21:53:47] dduvall: hi, did you see https://en.wikipedia.org/wiki/Wikipedia:Village_pump_(technical)#Problems_with_speedy_deletion_category_counts ? I commented on the train blockers task [21:53:51] it's starting to become a huge problem [21:58:39] dancy, jeena, twentyafterfour: ^^ [21:58:56] o/ [21:58:57] Hey [21:59:17] hi :) [21:59:33] taking a look at the train blocker task... [21:59:38] tl;dr: something changed (time lines up with the train) that category counts are no longer being updated/calculated correctly [22:00:06] and this is impacting speedy deletion workflows because admins rely on categories to find which pages are marked for deletion [22:00:43] ok. Sounds like a request to roll back the train. All on wikis? I'll see if I can raise Dan [22:01:02] I'm here [22:01:10] *rollback all wikis? [22:01:23] Hey Mukunda [22:01:28] that is my feeling too, however I have no insight into what was deployed this week and what the consequences of a rollback are [22:01:42] I understand that feeling. :-) [22:02:03] I'm not sure about a rollback either. The train was a shitshow this week [22:02:08] multiple risky patches rolled out [22:02:17] one data corruption bug hit group1 [22:02:31] one of the risky patches (https://phabricator.wikimedia.org/T293958#7612230) seems potentially related [22:02:40] but I'm not intimately familiar with the actual changes [22:03:05] taavi: that's the one that went all wrong the first time through [22:03:36] corrupted a lot of db records (fortunately could be re-generated) [22:03:52] can you ping the people it says to ping on slack? [22:04:54] based on the description, that linksupdate change seems suspect too [22:05:21] T299244 seems to be this, so I'll mark it as UBN + blocker for this week [22:05:22] T299244: {{PAGESINCATEGORY:Wikipedia:Nuweg}} not decreased when page is deleted - https://phabricator.wikimedia.org/T299244 [22:05:52] 10Release-Engineering-Team (Next), 10Patch-For-Review, 10Release, 10Train Deployments: 1.38.0-wmf.17 deployment blockers - https://phabricator.wikimedia.org/T293958 (10Majavah) [22:08:36] nice find [22:10:44] I left a comment on-wiki saying it's being actively looked into [22:10:54] I managed to reproduce locally, btw [22:12:18] taavi: nice catch [22:15:04] are you bisecting or trying for a fix? [22:15:22] so do we have a suspect for the culprit? fallout from the linksupdate changes? [22:15:33] I pinged on slack [22:15:39] but not seeing any response yet [22:15:50] legoktm: I'm bisecting [22:17:18] ok :) [22:20:38] it's the linksupdate refactor [22:23:12] makes sense [22:23:20] does it cleanly revert? [22:23:32] that's the thing, I'm not so sure it does [22:24:02] no [22:24:03] at least I'd like to have someone with more confidence around before trying it [22:31:43] Given that it's late on a friday and I have to pick up my girlfriend in an hour or so before a blizzard hits later tonight, I'm not itching to sign up for a potentially messy deployment [22:32:51] yeah, and it's already saturday here and I don't particularly want to spend my night debugging a totally non-familiar area of mediawiki [22:34:10] should we call/page other people then? [22:34:15] I don't think the status quo is acceptable [22:34:18] yes please [22:34:36] (and by "we", I mean someone else who has access to the contact list, etc. :)) [22:42:12] taavi: do you want to update the task at least with your findings? [22:42:38] sure [22:46:06] dumped my findings in there [22:48:02] this incident probably deserves a post-mortem for our response, I don't think the time it took us to respond is acceptable nor is the fact that the patch authors aren't available and I don't know who to escalate this to [22:51:31] taavi: Your findings ended at "seems to have no" – did you drop some of your comment? [22:52:03] James_F: yeah, missing a word (effect), fixed :/ [22:53:00] Ack. And yeah, reverting the LinksUpdate change is going to be rather hard given how messy it is. But rollback won't be great either. [22:54:47] it seems like tim is the one to escalate to? [22:55:48] as I said I don't know [22:56:37] someone needs to figure out how to a) fix the code to work correctly again and b) fix the invalid entries that have been created in the meanwhile [22:57:16] even if we can get just a) that would be a big step forward [22:58:29] I imagine that both of the people familiar are asleep or at least offline by now and not coming back until monday [22:59:26] its 9am saturday for tim [22:59:37] ah that's not too bad then I guess [22:59:56] * twentyafterfour looks up contact info [23:00:20] actually maybe dls so it would be 10am [23:07:26] hmm, text isn't going through not sure if I even have the ability to call or text australia [23:07:44] or if the number is just out of date [23:07:47] lemme try [23:10:10] twentyafterfour: my phone says it's sent, we'll see [23:10:41] (and we'll see how much it cost me later ;) ) [23:10:51] lol yeah [23:11:16] I still remember getting charged $20 for 1 small png file back in the day when I got my first "smart" phone [23:11:28] $0.50 per kilobyte [23:13:20] phab says Tyler is not available :/ is there anyone else to escalate this to? I have a fear this is getting lost until Monday then [23:15:05] Yeah Tyler definitely isn't available and wouldn't have any more clue than the rest of us. [23:15:11] taavi: his eyes were just lasered, so yeah, out of commission. I haven't read the full diagnosis above so I'm not sure if this is klaxon/whatever worthy [23:15:46] category counts aren't getting updated which is disrupting some on wiki workflows [23:15:48] greg-g: Category counts are not being updated correctly. Apparently some community processes rely on the category counts being 'right'. [23:16:07] (I've never experienced accurate category counts in my life, but I've only been using MW since 2002.) [23:16:14] oh [23:16:20] less than ideal but I don't know if it's worth risking a rollback which could have other fallout [23:16:26] * greg-g nods [23:16:45] I think it's not great but not as bad as a rollback might be. [23:16:50] and I can't stick around much longer to babysit any deployments [23:16:55] Ack. [23:17:12] Worst-case I can, but it's not my job (much like greg-g) any more. [23:17:14] twentyafterfour: can you update the task with a summary of that ^ [23:17:22] sure [23:17:30] sorry, not yer boss ;) [23:18:08] more specifically, category counts don't go down when something's deleted. [23:18:36] Yeah, which means speedy-deletion categories look "full" when they're empty. [23:19:24] alright, I'm going back to things-that-aren't-this now. [23:20:12] For example, https://en.wikipedia.org/wiki/Category:Candidates_for_speedy_deletion_as_abandoned_drafts_or_AfC_submissions has nothing in it, but on https://en.wikipedia.org/wiki/Category:Speedy_deletion it's listed as having five pages in it. [23:20:57] Are increments happening, just not decrements on deletion? If so, that strongly suggests the bug is that the newly-deleted page's ID isn't transmitted properly. [23:21:05] * James_F looks at some code. [23:22:57] 10Release-Engineering-Team (Next), 10Patch-For-Review, 10Release, 10Train Deployments: 1.38.0-wmf.17 deployment blockers - https://phabricator.wikimedia.org/T293958 (10mmodell) So we don't currently have a fix for T299244 but we also aren't comfortable reverting the branch at this point due to several comp... [23:25:55] task updated [23:26:01] Thanks twentyafterfour [23:26:35] James_F: are you working out a revert or a patch? [23:29:39] * legoktm will start working on a revert once his repo finishes updating [23:33:46] legoktm: Neither, just trying to get my head around the code and what might be going wrong. [23:35:50] ok, I think I have a revert [23:36:14] PageIdentity::getId() -> Title::getArticleID() which shouldn't return 0 except if canExist() returns false? [23:36:45] legoktm: How stable is that revert likely to be? [23:37:26] Oh wait. [23:37:51] If you call getFieldFromPageStore() and the page doesn't exist (any more) it'll return false which is cast to an int. [23:37:53] * James_F sighs. [23:38:40] OK, so we need to pass that through? But PageIdentity doesn't allow for passing a non READ_NORMAL request in. [23:38:45] Because why would we need that? [23:38:47] Meh. [23:38:54] 15:38:39 <+wikibugs> (PS1) Legoktm: Revert "LinksUpdate refactor" and follow-ups [core] (wmf/1.38.0-wmf.17) - https://gerrit.wikimedia.org/r/754046 (https://phabricator.wikimedia.org/T299244) [23:38:56] I'll thought-dump on the task. [23:39:16] which is the squashed version of `git revert 682aad7557ebb09c2aefa84d2c0c1f6c87ea5b76 87d8ccbd3e5280582a1bd60771b821ee5bbc95a7 1aecb692f64b3166cbaf1a7de9d85790ebc8759f d3b2b800678e91fd1a6177d80fde790c9006d423` [23:55:42] (discussion has moved to -operations about deploying the revert)