[00:00:33] (Queue (Jenkins jobs + Zuul functions) alert) firing: - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert [00:11:55] PROBLEM - Work requests waiting in Zuul Gearman server on contint2001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [400.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/d/000000322/zuul-gearman?orgId=1&viewPanel=10 [00:14:26] 10GitLab (Integrations), 10Software-Licensing: https://gitlab.wikimedia.org/repos/releng/gitlab-phabricator needs a LICENSE file - https://phabricator.wikimedia.org/T331943 (10Legoktm) [00:20:33] (Queue (Jenkins jobs + Zuul functions) alert) resolved: - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert [00:21:25] RECOVERY - Work requests waiting in Zuul Gearman server on contint2001 is OK: OK: Less than 100.00% above the threshold [200.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/d/000000322/zuul-gearman?orgId=1&viewPanel=10 [02:09:17] 10MediaWiki-Releasing, 10MediaWiki-Release-Tools: When branching a new major version of MW, the release tool should also bump the extension.json requirement of tarballed and WMF prod extensions to the new version - https://phabricator.wikimedia.org/T324110 (10Samwilson) I'm not quite sure I'm understanding: so... [02:12:49] I see lib-up is removing code to make an intentinally disabled rule pass, https://gerrit.wikimedia.org/r/c/mediawiki/libs/less.php/+/897780/1/.phpcs.xml [02:12:50] ref T278474 [02:12:50] T278474: Investigate conflict between RedundantVarName sniff and Doxygen due to how the variable name is wanted on @var - https://phabricator.wikimedia.org/T278474 [02:13:17] I've not noticed it do that before, I hope it's not doing that to all libs and breaking Doxygen [02:13:41] Krinkle: It is. [02:14:01] Krinkle: The idea that we intentionally want to disable a rule rather than fix it is not compatible with how mw-codesniffer is developed. [02:14:28] 10MediaWiki-Releasing, 10MediaWiki-Release-Tools: When branching a new major version of MW, the release tool should also bump the extension.json requirement of tarballed and WMF prod extensions to the new version - https://phabricator.wikimedia.org/T324110 (10Jdforrester-WMF) >>! In T324110#8690388, @Samwilson... [02:15:57] right, I'm guessing that has changed is that RedundantVarName now has an auto-fix? [02:16:01] what* [02:16:07] I presume so, yes. [02:16:10] * James_F looks. [02:16:19] all the more reason to add it to the preset so that we can stop wasting time on this one. [02:16:42] and enable RedundantVarName locally to mw-core as that's the only repo where the custom doxygen filter is set up to support omitting it. [02:16:48] it's not redundant anywhere else [02:17:07] * James_F nods. [02:19:37] Krinkle: Nothing obvious in https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/tools/codesniffer/+/HEAD/HISTORY.md but perhaps? [02:26:18] (03CR) 10Krinkle: zuul: Remove some Renameuser dependencies (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/896311 (https://phabricator.wikimedia.org/T27482) (owner: 10Majavah) [02:34:16] Krinkle, James_F: if a rule really should be disabled, use 0 instead and libup won't try to auto-fix it, I think. [02:34:51] Oh, yes, good point. [02:34:52] 10MediaWiki-Releasing, 10MediaWiki-Release-Tools: When branching a new major version of MW, the release tool should also bump the extension.json requirement of tarballed and WMF prod extensions to the new version - https://phabricator.wikimedia.org/T324110 (10Samwilson) Ah right, yep, that sounds good. It does... [02:34:57] legoktm: Also hello. :-) [02:35:23] hi :)) [02:35:38] * James_F is about to pull the REL1_40 branch hammer. [02:36:28] 10MediaWiki-Releasing, 10MediaWiki-Release-Tools: When branching a new major version of MW, the release tool should also bump the extension.json requirement of tarballed and WMF prod extensions to the new version - https://phabricator.wikimedia.org/T324110 (10Jdforrester-WMF) >>! In T324110#8690430, @Samwilson... [02:37:21] we're in the middle of doing a release at work, and I was like "ugh, why is the SecureDrop release process so complicated?" and then I thought for a moment and remembered that I've just internalized exactly how complex the MW release process is [02:41:44] :-D [02:42:12] TBF it's now (ish) down to three bash scripts and then a bunch of human futzing: https://phabricator.wikimedia.org/T329079 [02:44:29] log MediaWiki being branched to REL1_40 from fd3ce7633984d0120288d4a6fe48343cf9ba86a2 for T329079 [02:44:30] T329079: Branch REL1_40 for MediaWiki and all extensions and skins - https://phabricator.wikimedia.org/T329079 [02:44:45] !log MediaWiki being branched to REL1_40 from fd3ce7633984d0120288d4a6fe48343cf9ba86a2 for T329079 [02:44:48] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [02:50:18] James_F: branching is just the start! it only ends when we announcepublished tarballs, and that whole process is a lot more complex :) [02:50:40] legoktm: The tarball generation is also a single sh command now. But yes. [02:50:52] (Also I want to add docker image generation into the release script at some point, etc.) [02:52:21] hmmm, the unofficial docker hub image or a different image? [02:58:43] Actual official ones. [02:58:50] Hence "I want". [02:58:55] Not the Docker Hub ones. [02:59:09] Pre-built with wiki-farm support etc. [03:01:31] Gah. [03:01:38] Why won't gerrit let me push? [03:10:29] OK, cherry-pick from master will have to do. Oh well. [03:14:33] 10MediaWiki-Releasing, 10MW-1.40-release, 10Patch-For-Review: Branch REL1_40 for MediaWiki and all extensions and skins - https://phabricator.wikimedia.org/T329079 (10Jdforrester-WMF) 05Open→03Resolved [03:14:37] 10MediaWiki-Releasing, 10MW-1.40-release, 10Release: Release MW 1.40.0 - https://phabricator.wikimedia.org/T329075 (10Jdforrester-WMF) [03:19:55] James_F: hmm, shouldn't this weeks deployment branch be 1.41.0-wmf.1? [03:20:15] or am I off by one? [03:27:25] legoktm: This week is 1.40.0-wmf.27. [03:27:37] legoktm: That way REL1_40 === the last alpha of 1.40. [03:27:47] legoktm: So yes, off by one. [03:27:52] gotcha [03:32:50] Anyway, it's late enough that I'm probably making more mistakes than I'm fixing, so I'm calling it for now. [05:34:25] 10GitLab (Integrations), 10Release-Engineering-Team, 10Software-Licensing, 10User-brennen: https://gitlab.wikimedia.org/repos/releng/gitlab-phabricator needs a LICENSE file - https://phabricator.wikimedia.org/T331943 (10brennen) a:03brennen Good catch and thanks for the task - I remember thinking I neede... [06:33:09] (03CR) 10Hashar: [C: 03+2] Zuul: Add Msz2001 to the CI allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/897963 (owner: 10Urbanecm) [06:34:18] (03Merged) 10jenkins-bot: Zuul: Add Msz2001 to the CI allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/897963 (owner: 10Urbanecm) [06:35:25] (03CR) 10Hashar: [C: 03+2] "Deployed" [integration/config] - 10https://gerrit.wikimedia.org/r/897963 (owner: 10Urbanecm) [06:54:30] (03CR) 10Hashar: [C: 03+2] Zuul: [mediawiki/extensions/Renameuser] Move to non-production block [integration/config] - 10https://gerrit.wikimedia.org/r/895797 (owner: 10Jforrester) [06:55:38] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/Renameuser] Move to non-production block [integration/config] - 10https://gerrit.wikimedia.org/r/895797 (owner: 10Jforrester) [07:17:26] !log `dwalden@deployment-mwmaint02:~$ mwscript maintenance/refreshLinks.php --wiki enwiki --category='Pages that use Phonos'` for T326163 [07:17:29] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [07:17:29] T326163: Add page properties for Phonos usage data - https://phabricator.wikimedia.org/T326163 [07:24:47] !log `dwalden@deployment-mwmaint02:~$ mwscript maintenance/refreshLinks.php --wiki enwiktionary --category='Pages that use Phonos'` for T326163 [07:24:50] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [07:24:51] T326163: Add page properties for Phonos usage data - https://phabricator.wikimedia.org/T326163 [07:25:14] !log `dwalden@deployment-mwmaint02:~$ mwscript maintenance/refreshLinks.php --wiki en_rtlwiki --category='Pages that use Phonos'` for T326163 [07:25:17] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:42:44] (03PS1) 10Zoranzoki21: Zuul: Archive the ChangeUserPasswords extension [integration/config] - 10https://gerrit.wikimedia.org/r/898688 (https://phabricator.wikimedia.org/T316122) [11:55:44] (03CR) 10Hashar: [C: 03+2] Zuul: Archive the ChangeUserPasswords extension [integration/config] - 10https://gerrit.wikimedia.org/r/898688 (https://phabricator.wikimedia.org/T316122) (owner: 10Zoranzoki21) [11:56:57] (03Merged) 10jenkins-bot: Zuul: Archive the ChangeUserPasswords extension [integration/config] - 10https://gerrit.wikimedia.org/r/898688 (https://phabricator.wikimedia.org/T316122) (owner: 10Zoranzoki21) [11:58:08] 10Scap, 10serviceops-collab: scap deploy-local fails on aphlict servers - https://phabricator.wikimedia.org/T331278 (10eoghan) 05Open→03Resolved a:03eoghan This is fixed now, aphlict2001 is happy. See the update here for details: T322369#8690007 [12:00:38] 10Phabricator, 10Release-Engineering-Team, 10serviceops-collab, 10Patch-For-Review: Deploy phabricator to aphlict2001.codfw.wmnet - https://phabricator.wikimedia.org/T329908 (10eoghan) 05Open→03Resolved a:03eoghan scap deployments on the aphlict host now works, some updates are here: T322369#8690007 [12:08:12] 10Phabricator, 10Release-Engineering-Team, 10serviceops-collab, 10Patch-For-Review: Deploy phabricator to aphlict2001.codfw.wmnet - https://phabricator.wikimedia.org/T329908 (10eoghan) [12:08:24] 10Phabricator, 10serviceops-collab: create aphlict2001 (Phabricator realtime notifications codfw) - https://phabricator.wikimedia.org/T322369 (10eoghan) [12:08:30] 10Release-Engineering-Team (GitLab V: Event Horizon 🌄), 10Scap, 10SRE, 10serviceops-collab: Scap fails on debian bullseye targets - https://phabricator.wikimedia.org/T326668 (10eoghan) [12:08:36] 10Phabricator, 10Release-Engineering-Team (GitLab V: Event Horizon 🌄), 10Scap, 10serviceops-collab: puppet/scap: failing on aphlict2001 - https://phabricator.wikimedia.org/T330393 (10eoghan) 05Open→03Resolved a:03eoghan This is resolved, some updates here to explain what changed: T322369#8690007 [12:28:04] (03Abandoned) 10Hashar: Use our packages for node16, bringing in npm v8 [integration/config] - 10https://gerrit.wikimedia.org/r/840336 (owner: 10Legoktm) [12:28:26] 10Scap, 10MW-on-K8s, 10serviceops, 10Upstream: Kubernetes configuration file is group-readable - https://phabricator.wikimedia.org/T329899 (10TheresNoTime) If I'm reading the output of a scap failure I just had (P45862) correctly, this warning is now triggering an "error", and causing a stage to rollback?... [12:29:34] hello, I have two more small-ish things I'd like to deploy, but don't feel comfortable doing so until I'm certain that ^ (https://phabricator.wikimedia.org/T329899#8691439) is safe to ignore etc. [12:30:05] yikes [12:32:53] (fwiw the changes still deployed successfully, which feels wrong but *shrug*) [12:32:58] o_O [12:33:41] probably worth mentioning that in the task? wasn’t obvious to me from reading the comments + paste [12:36:00] (done) [12:38:43] (thx) [12:49:11] TheresNoTime: we're still treating K8s errors as "soft" in scap, my understanding is that's still the desired behavior [12:49:18] but https://phabricator.wikimedia.org/T329899#8691439 doesn't look great [12:49:50] claime: would you know more about those K8s errors? ^^ [12:51:07] I'm kind of busy with the DNS weirdness we're having regarding A/A repool rn [12:51:29] Can you check your deployments with https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting ? [12:52:47] 10Continuous-Integration-Infrastructure, 10Quibble, 10Developer Productivity: Quibble: Include the failed command in data passed to external endpoint - https://phabricator.wikimedia.org/T330750 (10kostajh) 05Open→03Resolved [12:52:53] 10Continuous-Integration-Infrastructure, 10Quibble, 10Developer Productivity, 10MW-1.40-notes (1.40.0-wmf.26; 2023-03-06): Provide early feedback when a patch has job failures - https://phabricator.wikimedia.org/T323750 (10kostajh) [12:53:04] 10Continuous-Integration-Infrastructure, 10Quibble, 10Developer Productivity: Exclude certain jobs from early feedback - https://phabricator.wikimedia.org/T331236 (10kostajh) 05Open→03Resolved [12:53:07] 10Continuous-Integration-Infrastructure, 10Quibble, 10Developer Productivity, 10MW-1.40-notes (1.40.0-wmf.26; 2023-03-06): Provide early feedback when a patch has job failures - https://phabricator.wikimedia.org/T323750 (10kostajh) [12:54:56] I don't think I'll have the time to do any troubleshooting (until much later). There is a deployment coming up soon, I'll try it and see if it fails again [12:55:33] Lucas_WMDE: unless you'll be around for the window in 5m? [12:55:41] 10Continuous-Integration-Infrastructure, 10Quibble, 10Developer Productivity: Quibble: Generate structured output for command success/failures - https://phabricator.wikimedia.org/T331996 (10kostajh) [12:55:56] I should be making lunch then [12:56:05] I can peek at IRC but not be available for that much troubleshooting [12:56:45] (heh, lunch at a reasonable time) [12:58:37] what a concept [13:01:32] (03PS1) 10Zoranzoki21: Zuul: Archive the BaseHooks extension [integration/config] - 10https://gerrit.wikimedia.org/r/898734 (https://phabricator.wikimedia.org/T299257) [13:06:32] (03PS2) 10Hashar: jjb: aggregate some builders/publishers [integration/config] - 10https://gerrit.wikimedia.org/r/896447 [13:06:34] (03PS1) 10Hashar: jjb: use same order when creating directories [integration/config] - 10https://gerrit.wikimedia.org/r/898735 [13:06:44] (03CR) 10Hashar: [C: 03+2] Zuul: Archive the BaseHooks extension [integration/config] - 10https://gerrit.wikimedia.org/r/898734 (https://phabricator.wikimedia.org/T299257) (owner: 10Zoranzoki21) [13:07:56] (03Merged) 10jenkins-bot: Zuul: Archive the BaseHooks extension [integration/config] - 10https://gerrit.wikimedia.org/r/898734 (https://phabricator.wikimedia.org/T299257) (owner: 10Zoranzoki21) [13:17:06] (and of course it didn't do the error this time) [13:29:29] 10Scap, 10MW-on-K8s, 10serviceops, 10Upstream: Kubernetes configuration file is group-readable - https://phabricator.wikimedia.org/T329899 (10JMeybohm) >>! In T329899#8691439, @TheresNoTime wrote: > If I'm reading the output of a scap failure I just had (P45862) correctly, this warning is now triggering an... [13:36:19] 10Continuous-Integration-Config, 10Wikidata, 10Wikidata Dev Team, 10Wikidata Query Builder, and 4 others: [QB] [WDQS-GUI] Move build scripts from CI to the repository - https://phabricator.wikimedia.org/T328543 (10ItamarWMDE) [13:40:47] (03PS1) 10Subramanya Sastry: Changes after translatewiki fr message change was reverted [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/898742 [13:43:11] (03CR) 10Jforrester: [C: 03+1] jjb: use same order when creating directories [integration/config] - 10https://gerrit.wikimedia.org/r/898735 (owner: 10Hashar) [13:44:24] (03CR) 10Jforrester: [C: 03+1] jjb: aggregate some builders/publishers [integration/config] - 10https://gerrit.wikimedia.org/r/896447 (owner: 10Hashar) [13:46:02] (03CR) 10Hashar: [C: 03+2] jjb: use same order when creating directories [integration/config] - 10https://gerrit.wikimedia.org/r/898735 (owner: 10Hashar) [13:46:08] (03CR) 10Hashar: [C: 03+2] jjb: aggregate some builders/publishers [integration/config] - 10https://gerrit.wikimedia.org/r/896447 (owner: 10Hashar) [13:47:53] (03Merged) 10jenkins-bot: jjb: use same order when creating directories [integration/config] - 10https://gerrit.wikimedia.org/r/898735 (owner: 10Hashar) [13:48:08] (03Merged) 10jenkins-bot: jjb: aggregate some builders/publishers [integration/config] - 10https://gerrit.wikimedia.org/r/896447 (owner: 10Hashar) [13:58:47] (03PS1) 10Hashar: Archive analytics/wikistats [integration/config] - 10https://gerrit.wikimedia.org/r/898745 (https://phabricator.wikimedia.org/T332004) [13:59:29] hashar: Nice! [13:59:43] James_F: I thought we phased out all stretch based images :-\ [13:59:54] Yeah, I forgot about that one too. [14:00:15] ah no https://phabricator.wikimedia.org/T278203 :D [14:00:34] Yeah. :-( [14:00:39] 10Continuous-Integration-Infrastructure, 10Patch-For-Review: Migrate all CI jobs from stretch to buster or later and drop stretch testing support - https://phabricator.wikimedia.org/T278203 (10hashar) [14:00:43] A bunch of those are done now. [14:00:51] well eventually it will be done [14:00:53] Tox etc. [14:01:08] Did we migrate the zuul-cloner image in the end? [14:01:13] I know it was a bit worrying. [14:01:24] nop [14:01:35] I guess I might give it a try [14:01:37] one day :] [14:01:58] Ha. [14:01:58] Bon chance! [14:02:09] merci mon ami [14:15:24] (03PS1) 10Zoranzoki21: Zuul: Archive the RT extension [integration/config] - 10https://gerrit.wikimedia.org/r/898757 (https://phabricator.wikimedia.org/T289340) [15:45:24] 10Phabricator, 10Release-Engineering-Team, 10Security-Team, 10SecTeam-Processed: Phabricator Admin Access Request for Scott Bassett - https://phabricator.wikimedia.org/T331928 (10sbassett) [15:48:54] (03PS1) 10Hashar: dockerfiles: update gradle stretch > bullseye [integration/config] - 10https://gerrit.wikimedia.org/r/898787 (https://phabricator.wikimedia.org/T278203) [15:48:56] (03PS1) 10Hashar: jjb: update for Grable 4.4.1 [integration/config] - 10https://gerrit.wikimedia.org/r/898788 (https://phabricator.wikimedia.org/T278203) [15:50:50] (03PS1) 10Hashar: blubber: update base image to Gradle 4.4.1 [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/898789 (https://phabricator.wikimedia.org/T278203) [15:51:28] (03CR) 10CI reject: [V: 04-1] blubber: update base image to Gradle 4.4.1 [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/898789 (https://phabricator.wikimedia.org/T278203) (owner: 10Hashar) [15:52:10] 10Phabricator, 10Release-Engineering-Team, 10Security-Team, 10SecTeam-Processed, 10user-sbassett: Phabricator Admin Access Request for Scott Bassett - https://phabricator.wikimedia.org/T331928 (10sbassett) [15:52:53] (03PS2) 10Hashar: blubber: update base image to Gradle 4.4.1 [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/898789 (https://phabricator.wikimedia.org/T278203) [15:53:00] (03PS2) 10Hashar: jjb: update for Grable 4.4.1 [integration/config] - 10https://gerrit.wikimedia.org/r/898788 (https://phabricator.wikimedia.org/T278203) [15:53:07] (03CR) 10CI reject: [V: 04-1] blubber: update base image to Gradle 4.4.1 [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/898789 (https://phabricator.wikimedia.org/T278203) (owner: 10Hashar) [15:53:43] 10Continuous-Integration-Infrastructure, 10Patch-For-Review: Migrate all CI jobs from stretch to buster or later and drop stretch testing support - https://phabricator.wikimedia.org/T278203 (10hashar) [15:55:07] 10Phabricator, 10Release-Engineering-Team, 10Security-Team, 10SecTeam-Processed, 10user-sbassett: Phabricator Admin Access Request for Scott Bassett - https://phabricator.wikimedia.org/T331928 (10sbassett) Ping @thcipriani @brennen et al to see if there are any objections to this request. Thanks. [15:55:37] (03CR) 10Hashar: [C: 03+2] Zuul: Archive the RT extension [integration/config] - 10https://gerrit.wikimedia.org/r/898757 (https://phabricator.wikimedia.org/T289340) (owner: 10Zoranzoki21) [15:56:29] (03CR) 10Hashar: [C: 03+2] dockerfiles: update gradle stretch > bullseye [integration/config] - 10https://gerrit.wikimedia.org/r/898787 (https://phabricator.wikimedia.org/T278203) (owner: 10Hashar) [15:56:43] (03Merged) 10jenkins-bot: Zuul: Archive the RT extension [integration/config] - 10https://gerrit.wikimedia.org/r/898757 (https://phabricator.wikimedia.org/T289340) (owner: 10Zoranzoki21) [15:57:41] 10Release-Engineering-Team (Watching / External), 10Machine-Learning-Team (Active Tasks), 10Sustainability (Incident Followup): [Spike] Write reports about why Ext:ORES is helping cause server 500s and write tasks to fix - https://phabricator.wikimedia.org/T181010 (10akosiaris) [15:58:02] (03Merged) 10jenkins-bot: dockerfiles: update gradle stretch > bullseye [integration/config] - 10https://gerrit.wikimedia.org/r/898787 (https://phabricator.wikimedia.org/T278203) (owner: 10Hashar) [16:00:30] !log updating Docker images for https://gerrit.wikimedia.org/r/c/integration/config/+/898787 [16:00:32] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:04:09] 10Phabricator, 10Release-Engineering-Team, 10Security-Team, 10SecTeam-Processed, 10user-sbassett: Phabricator Admin Access Request for Scott Bassett - https://phabricator.wikimedia.org/T331928 (10brennen) No objections here. [16:13:42] (03PS1) 10Hashar: dockerfiles: add example for Gradle image [integration/config] - 10https://gerrit.wikimedia.org/r/898797 (https://phabricator.wikimedia.org/T278203) [16:14:24] (03CR) 10Hashar: [C: 03+2] "I ran it locally with 4.4.1 and it passed!" [integration/config] - 10https://gerrit.wikimedia.org/r/898797 (https://phabricator.wikimedia.org/T278203) (owner: 10Hashar) [16:15:32] (03Merged) 10jenkins-bot: dockerfiles: add example for Gradle image [integration/config] - 10https://gerrit.wikimedia.org/r/898797 (https://phabricator.wikimedia.org/T278203) (owner: 10Hashar) [16:15:54] (03CR) 10Hashar: "recheck after I have build the image https://gerrit.wikimedia.org/r/c/integration/config/+/898787/ and tested it with https://gerrit.wikim" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/898789 (https://phabricator.wikimedia.org/T278203) (owner: 10Hashar) [16:16:38] (03PS1) 10Zoranzoki21: Zuul: Archive the Blackout extension [integration/config] - 10https://gerrit.wikimedia.org/r/898798 (https://phabricator.wikimedia.org/T299259) [16:18:01] (03CR) 10Hashar: [C: 03+2] "The tests passed on https://gerrit.wikimedia.org/r/c/integration/pipelinelib/+/898789" [integration/config] - 10https://gerrit.wikimedia.org/r/898788 (https://phabricator.wikimedia.org/T278203) (owner: 10Hashar) [16:19:17] (03Merged) 10jenkins-bot: jjb: update for Grable 4.4.1 [integration/config] - 10https://gerrit.wikimedia.org/r/898788 (https://phabricator.wikimedia.org/T278203) (owner: 10Hashar) [16:20:57] (03PS2) 10Hashar: Zuul: Archive the Blackout extension [integration/config] - 10https://gerrit.wikimedia.org/r/898798 (https://phabricator.wikimedia.org/T299259) (owner: 10Zoranzoki21) [16:22:55] (03CR) 10Hashar: [C: 03+2] "CI config has been applied. We will see whether `gradle groovydoc` passes properly and updates the doc at https://doc.wikimedia.org/pipeli" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/898789 (https://phabricator.wikimedia.org/T278203) (owner: 10Hashar) [16:23:46] (03Merged) 10jenkins-bot: blubber: update base image to Gradle 4.4.1 [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/898789 (https://phabricator.wikimedia.org/T278203) (owner: 10Hashar) [16:24:54] (03PS1) 10Hashar: Link Groovy doc of PipelineLib [integration/docroot] - 10https://gerrit.wikimedia.org/r/898802 (https://phabricator.wikimedia.org/T222199) [16:26:28] (03CR) 10Hashar: [C: 03+2] "Success!!!" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/898789 (https://phabricator.wikimedia.org/T278203) (owner: 10Hashar) [16:27:50] 10Continuous-Integration-Infrastructure, 10Patch-For-Review: Migrate all CI jobs from stretch to buster or later and drop stretch testing support - https://phabricator.wikimedia.org/T278203 (10hashar) [16:29:14] 10Continuous-Integration-Infrastructure, 10Patch-For-Review: Migrate all CI jobs from stretch to buster or later and drop stretch testing support - https://phabricator.wikimedia.org/T278203 (10hashar) [16:29:35] 10Continuous-Integration-Infrastructure, 10Patch-For-Review: Migrate all CI jobs from stretch to buster or later and drop stretch testing support - https://phabricator.wikimedia.org/T278203 (10hashar) I have fixed `gradle` and updated it to Bullseye. `helm-linter` got updated ages ago. For `typos` I think we... [16:30:15] (03CR) 10Hashar: [C: 03+2] Zuul: Archive the Blackout extension [integration/config] - 10https://gerrit.wikimedia.org/r/898798 (https://phabricator.wikimedia.org/T299259) (owner: 10Zoranzoki21) [16:31:18] (03Merged) 10jenkins-bot: Zuul: Archive the Blackout extension [integration/config] - 10https://gerrit.wikimedia.org/r/898798 (https://phabricator.wikimedia.org/T299259) (owner: 10Zoranzoki21) [16:41:29] !log Tag Quibble 1.5.3 @ df49e8a04f5 # T331621 T331061 [16:41:33] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:41:33] T331061: Capture output from failed command and transmit to earlywarningbot - https://phabricator.wikimedia.org/T331061 [16:41:34] T331621: CI: Don't show PHPUnit progress - https://phabricator.wikimedia.org/T331621 [16:45:39] (03PS1) 10Hashar: dockerfiles: update to Quibble 1.5.3 [integration/config] - 10https://gerrit.wikimedia.org/r/898805 (https://phabricator.wikimedia.org/T331061) [16:47:38] jjb/job-templates.yaml:695 ERROR: docker-registry.wikimedia.org/releng/node14-test-browser not found! [16:47:39] bah [16:48:25] (03PS1) 10Hashar: jjb: update Quibble jobs to 1.5.3 [integration/config] - 10https://gerrit.wikimedia.org/r/898807 (https://phabricator.wikimedia.org/T331061) [16:48:41] (03CR) 10Hashar: [C: 03+2] dockerfiles: update to Quibble 1.5.3 [integration/config] - 10https://gerrit.wikimedia.org/r/898805 (https://phabricator.wikimedia.org/T331061) (owner: 10Hashar) [16:50:07] (03Merged) 10jenkins-bot: dockerfiles: update to Quibble 1.5.3 [integration/config] - 10https://gerrit.wikimedia.org/r/898805 (https://phabricator.wikimedia.org/T331061) (owner: 10Hashar) [16:54:36] !log Building Docker images for Quibble 1.5.3 [16:54:37] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:57:11] waiting for that build to have completed and I will call it a day [17:02:42] have the deployment calendar entries _always_ had little icons [17:05:15] can’t speak for “always” but the choo choo train has been there for more than, like, a week at least ^^ [17:05:26] I remember seeing that and the wrench/screwdriver before [17:05:29] (03CR) 10Hashar: [C: 03+2] "Successfully published image docker-registry.discovery.wmnet/releng/quibble-buster-php74-bundle:1.5.3" [integration/config] - 10https://gerrit.wikimedia.org/r/898805 (https://phabricator.wikimedia.org/T331061) (owner: 10Hashar) [17:06:21] * hashar blames https://en.wikipedia.org/wiki/The_Emoji_Movie 🚄 [17:06:45] I am off & :] [17:08:02] looks like they were added at some point between jan 2022 https://web.archive.org/web/20220106023457/https://wikitech.wikimedia.org/wiki/Deployments [17:08:05] and dec 2022 https://web.archive.org/web/20221226112250/https://wikitech.wikimedia.org/wiki/Deployments [17:08:10] I can’t be bothered to narrow it down more than that ;) [17:08:33] (but that’s already a bit more recent than I would’ve guessed, tbh) [17:11:15] brennen: that thcipriani kid will stick an emoji on anything ;) [17:12:42] weee [17:21:08] (03CR) 10Kosta Harlan: [C: 03+1] jjb: update Quibble jobs to 1.5.3 [integration/config] - 10https://gerrit.wikimedia.org/r/898807 (https://phabricator.wikimedia.org/T331061) (owner: 10Hashar) [17:21:30] haha [17:21:32] they're charming [17:28:46] 10Deployments, 10Phabricator, 10Release-Engineering-Team, 10User-brennen: Phabricator deployment 2023-03-15 - https://phabricator.wikimedia.org/T331915 (10brennen) [17:30:30] 10Continuous-Integration-Config, 10serviceops, 10Regression, 10Sustainability (Incident Followup): operations-apache-config-lint replacement doesn't check syntax - https://phabricator.wikimedia.org/T114801 (10akosiaris) Removing #SRE, has already been triaged to a more specific SRE subteam [17:34:05] 10Release-Engineering-Team (Radar), 10SRE, 10observability, 10Epic, 10Sustainability (Incident Followup): Tracking: Monitoring and alerts for "business" metrics - https://phabricator.wikimedia.org/T140942 (10akosiaris) 05Open→03Resolved a:03akosiaris No comments in 6 years, I am gonna tentatively r... [17:37:33] 10Release-Engineering-Team (Seen), 10SRE, 10Sustainability (Incident Followup): Review new service 'pre-deployment to production' checklist - https://phabricator.wikimedia.org/T141897 (10akosiaris) 05Open→03Resolved a:03akosiaris No comments, updates or anything for that matter since the opening of thi... [17:52:53] 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10Sustainability (Incident Followup): docker-registry: some layers has been corrupted due to deleting other swift containers - https://phabricator.wikimedia.org/T228196 (10akosiaris) [17:53:09] 10Release-Engineering-Team (Radar), 10SRE, 10serviceops, 10Sustainability (Incident Followup): create swift container-to-container synchronization metrics - https://phabricator.wikimedia.org/T229117 (10akosiaris) 05Open→03Declined I am gonna close this as Declined. Many years later I don't remember any... [17:53:18] 10Release-Engineering-Team-TODO, 10SRE, 10serviceops, 10Sustainability (Incident Followup): docker-registry: some layers has been corrupted due to deleting other swift containers - https://phabricator.wikimedia.org/T228196 (10akosiaris) [17:53:24] 10Phabricator, 10Release-Engineering-Team (Seen), 10Sustainability (Incident Followup): Analyze and amend (if necessary) workflow of user reporting and detecting large regressions/outages - https://phabricator.wikimedia.org/T219589 (10akosiaris) I am gonna tentatively remove #SRE as there doesn't appear to b... [17:53:30] 10Release-Engineering-Team (Radar), 10SRE, 10serviceops, 10Sustainability (Incident Followup): create a docker_registry_codfw swift container backup - https://phabricator.wikimedia.org/T229118 (10akosiaris) 05Open→03Declined I am gonna close this as Declined. Many years later I don't remember any more... [17:54:58] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10serviceops-collab, 10Datacenter-Switchover: switch releases.wikimedia.org from eqiad to codfw - https://phabricator.wikimedia.org/T330960 (10LSobanski) >>! In T330960#8687674, @hashar wrote: >>>! In T330960#8679878, @LSobanski wrote: >>... [18:03:09] 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.40.0-wmf.27 deployment blockers - https://phabricator.wikimedia.org/T330205 (10brennen) 05Open→03In progress [18:23:25] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10serviceops-collab, 10Datacenter-Switchover: switch releases.wikimedia.org from eqiad to codfw - https://phabricator.wikimedia.org/T330960 (10Dzahn) 05Stalled→03Declined closing as declined based on comments by both Hashar and LSoban... [20:22:46] (03CR) 10Arlolra: [C: 03+2] "From https://translatewiki.net/wiki/MediaWiki_talk:Cite_references_link_many_format_backlink_labels/fr#Impact_of_this_change_on_size_of_CS" [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/898742 (owner: 10Subramanya Sastry) [20:23:42] (03Merged) 10jenkins-bot: Changes after translatewiki fr message change was reverted [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/898742 (owner: 10Subramanya Sastry) [22:25:34] maintenance-disconnect-full-disks build 473605 integration-agent-docker-1034 (/: 29%, /srv: 99%, /var/lib/docker: 46%): OFFLINE due to disk space [22:30:33] maintenance-disconnect-full-disks build 473606 integration-agent-docker-1034 (/: 29%, /srv: 6%, /var/lib/docker: 43%): RECOVERY disk space OK