[00:01:47] <jinxer-wm>	 (DatasourceError) firing: Queue (Jenkins jobs + Zuul functions) alert - https://grafana.wikimedia.org/alerting/grafana/iS0FSjJ4z/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError   - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError
[00:11:47] <jinxer-wm>	 (DatasourceError) resolved: Queue (Jenkins jobs + Zuul functions) alert - https://grafana.wikimedia.org/alerting/grafana/iS0FSjJ4z/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError   - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError
[02:54:01] <TimStarling>	 !log on deployment-deploy03 truncate tables block and block_target on all beta wikis T357366
[02:54:04] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[02:54:04] <stashbot>	 T357366: ipblocks/block divergence in write-both mode - https://phabricator.wikimedia.org/T357366
[02:57:45] <TimStarling>	 !log on deployment-deploy03 running foreachwiki maintenance/migrateBlocks.php again (T355034, T357366)
[02:57:51] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[02:57:51] <stashbot>	 T355034: Deploy new block_target schema - https://phabricator.wikimedia.org/T355034
[03:39:18] <wikibugs>	 10Beta-Cluster-Infrastructure: foreachwiki on beta does not include en_rtlwiki - https://phabricator.wikimedia.org/T357877#9553996 (10tstarling)
[05:31:29] <wikibugs>	 10Release-Engineering-Team, 10Gerrit (Gerrit 3.8), 10Patch-For-Review: Upgrade to Gerrit 3.8 - https://phabricator.wikimedia.org/T354886#9554021 (10hashar)
[08:24:33] <wikibugs>	 (03PS1) 10WMDE-Fisch: Add Popups to Cite jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1004099 (https://phabricator.wikimedia.org/T355194)
[09:31:45] <wmf-insecte>	 Project beta-update-databases-eqiad build #74023: 04FAILURE in 11 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/74023/
[10:04:34] <hashar>	 jnuche: good morning, about the CI jobs finding that .whl files are not Zipfiles, they must be LFS placeholders ;)
[10:04:55] <hashar>	 git-lfs is not installed in the ci-src-setup-simple docker image which used to fetch the repo / patch
[10:04:57] <hashar>	 https://gerrit.wikimedia.org/r/c/integration/config/+/1004245 would do it ;)
[10:05:10] <jnuche>	 hashar: hi, good morning
[10:05:16] <jnuche>	 yeah, I had already seen your change, thx
[10:05:22] <wikibugs>	 (03Merged) 10jenkins-bot: Add Popups to Cite jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1004099 (https://phabricator.wikimedia.org/T355194) (owner: 10WMDE-Fisch)
[10:05:37] <hashar>	 wanna +2 / deploy it to have the image built or should I ? ;)
[10:06:19] <hashar>	 !log Reloaded Zuul for https://gerrit.wikimedia.org/r/1004099 # T355194
[10:06:22] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[10:06:22] <stashbot>	 T355194: Move the reference previews code to Cite extension - https://phabricator.wikimedia.org/T355194
[10:12:22] <wikibugs>	 (03CR) 10Jaime Nuche: [C: 03+2] dockerfiles: add git-lfs to base image [integration/config] - 10https://gerrit.wikimedia.org/r/1004245 (owner: 10Hashar)
[10:12:57] <jnuche>	 hashar: I'm fine with you deploying it, otherwise I can do it later
[10:14:09] <wikibugs>	 (03Merged) 10jenkins-bot: dockerfiles: add git-lfs to base image [integration/config] - 10https://gerrit.wikimedia.org/r/1004245 (owner: 10Hashar)
[10:18:18] <hashar>	 it is quite straight forward )
[10:18:31] <hashar>	 then the Jenkins jobs gotta be updated
[10:24:22] <wikibugs>	 (03CR) 10Hashar: "The `mediawiki-i18n-check-docker` Jenkins job is also triggering for `labs/tools/*` repositories.  They are the repositories mentioned in " [integration/config] - 10https://gerrit.wikimedia.org/r/1003764 (https://phabricator.wikimedia.org/T357670) (owner: 10Amire80)
[10:25:50] <hashar>	 !log Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/1004245
[10:25:52] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[10:32:29] <wmf-insecte>	 Yippee, build fixed!
[10:32:30] <wmf-insecte>	 Project beta-update-databases-eqiad build #74024: 09FIXED in 12 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/74024/
[10:34:11] <wikibugs>	 (03PS1) 10Hashar: jjb: update jobs for ci-src-setup-simple image [integration/config] - 10https://gerrit.wikimedia.org/r/1004622
[10:40:11] <hashar>	 !log Updated Jenkins jobs for https://gerrit.wikimedia.org/r/c/integration/config/+/1004622
[10:40:12] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[10:40:23] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] jjb: update jobs for ci-src-setup-simple image [integration/config] - 10https://gerrit.wikimedia.org/r/1004622 (owner: 10Hashar)
[10:42:11] <wikibugs>	 (03Merged) 10jenkins-bot: jjb: update jobs for ci-src-setup-simple image [integration/config] - 10https://gerrit.wikimedia.org/r/1004622 (owner: 10Hashar)
[10:48:00] <hashar>	 that should have fixed it
[11:27:17] <wikibugs>	 10Project-Admins: Remove "cookbooks" tag from SRE-tools herald rule. - https://phabricator.wikimedia.org/T357895#9554590 (10joanna_borun)
[11:28:09] <wikibugs>	 10Project-Admins: Remove "cookbooks" alias from SRE-tools tag - https://phabricator.wikimedia.org/T357895#9554600 (10taavi)
[11:28:56] <wikibugs>	 10Project-Admins: Remove "cookbooks" alias from SRE-tools tag - https://phabricator.wikimedia.org/T357895#9554590 (10taavi) 05Open→03Resolved a:03taavi Done.
[11:34:46] <wikibugs>	 10Project-Admins: Expansion of cookbooks tag usage across teams - https://phabricator.wikimedia.org/T357895#9554640 (10joanna_borun) 05Resolved→03Open
[11:58:55] <wikibugs>	 (03CR) 10Jaime Nuche: "recheck" [integration/zuul/deploy] - 10https://gerrit.wikimedia.org/r/998933 (https://phabricator.wikimedia.org/T259611) (owner: 10Hashar)
[12:12:42] <wikibugs>	 10Project-Admins: Expansion of cookbooks tag usage across teams - https://phabricator.wikimedia.org/T357895#9554757 (10taavi) a:05taavi→03None
[12:37:38] <wikibugs>	 (03CR) 10Jaime Nuche: "recheck" [integration/zuul/deploy] - 10https://gerrit.wikimedia.org/r/998933 (https://phabricator.wikimedia.org/T259611) (owner: 10Hashar)
[12:44:38] <wikibugs>	 (03CR) 10Jaime Nuche: "recheck" [integration/zuul/deploy] - 10https://gerrit.wikimedia.org/r/998933 (https://phabricator.wikimedia.org/T259611) (owner: 10Hashar)
[12:47:26] <wikibugs>	 (03CR) 10Jaime Nuche: [C: 03+2] Track wheels individually [integration/zuul/deploy] - 10https://gerrit.wikimedia.org/r/998933 (https://phabricator.wikimedia.org/T259611) (owner: 10Hashar)
[12:47:57] <wikibugs>	 (03Merged) 10jenkins-bot: Track wheels individually [integration/zuul/deploy] - 10https://gerrit.wikimedia.org/r/998933 (https://phabricator.wikimedia.org/T259611) (owner: 10Hashar)
[13:02:25] <wikibugs>	 (03PS8) 10Jaime Nuche: jjb: overhaul Zuul deploy jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1003437 (https://phabricator.wikimedia.org/T342346)
[13:21:15] <wikibugs>	 10Release-Engineering-Team (Now this 🫠), 10Security-Team, 10SecTeam-Processed: Notify MediaWiki security tasks as soon as an uploaded patch fails to apply - https://phabricator.wikimedia.org/T350065#9554948 (10jnuche) 05Stalled→03In progress
[13:32:18] <wikibugs>	 (03CR) 10Hashar: jjb: overhaul Zuul deploy jobs (035 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/1003437 (https://phabricator.wikimedia.org/T342346) (owner: 10Jaime Nuche)
[13:37:33] <wikibugs>	 (03PS9) 10Jaime Nuche: jjb: overhaul Zuul deploy jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1003437 (https://phabricator.wikimedia.org/T342346)
[13:37:42] <wikibugs>	 (03CR) 10Jaime Nuche: jjb: overhaul Zuul deploy jobs (035 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/1003437 (https://phabricator.wikimedia.org/T342346) (owner: 10Jaime Nuche)
[13:39:00] <wikibugs>	 (03PS10) 10Jaime Nuche: jjb: overhaul Zuul deploy jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1003437 (https://phabricator.wikimedia.org/T342346)
[13:51:21] <Gerges>	 Is it now UTC afternoon backport window?
[13:56:18] <wikibugs>	 (03CR) 10Hashar: [C: 03+1] "Feel free to merge any time :)" [integration/config] - 10https://gerrit.wikimedia.org/r/1003437 (https://phabricator.wikimedia.org/T342346) (owner: 10Jaime Nuche)
[14:04:09] <Lucas_WMDE>	 Gerges: yes, but it’s in #wikimedia-operations, not here
[14:14:01] <wikibugs>	 (03CR) 10Jaime Nuche: "Thanks, could you +2 this since you're the only reviewer though? It doesn't feel right to Code-Review +2 my own patch when someone else wa" [integration/config] - 10https://gerrit.wikimedia.org/r/1003437 (https://phabricator.wikimedia.org/T342346) (owner: 10Jaime Nuche)
[14:16:18] <Lucas_WMDE>	 Gerges: please join the #wikimedia-operations channel and ping me there
[14:20:01] <Gerges>	 Lucas_WMDE He appears Cannot join channel (+r) - you need to be logged into your NickServ account
[14:20:09] <Lucas_WMDE>	 ah, I didn’t realize that
[14:20:45] <Lucas_WMDE>	 then I think you need to follow the instructions at https://meta.wikimedia.org/wiki/IRC/Instructions#Register_your_nickname,_identify,_and_enforce
[14:20:47] <wikibugs>	 10Release-Engineering-Team (Seen), 10MW-on-K8s, 10SRE, 10Traffic, and 2 others: Migrate internal traffic to k8s - https://phabricator.wikimedia.org/T333120#9555136 (10Clement_Goubert)
[14:21:06] <Lucas_WMDE>	 to register the account and make sure you can join the channel
[14:21:08] <Gerges>	 Ok
[14:24:49] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] jjb: overhaul Zuul deploy jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1003437 (https://phabricator.wikimedia.org/T342346) (owner: 10Jaime Nuche)
[14:25:51] <wikibugs>	 (03CR) 10Jaime Nuche: "Done" [integration/config] - 10https://gerrit.wikimedia.org/r/1003437 (https://phabricator.wikimedia.org/T342346) (owner: 10Jaime Nuche)
[14:27:14] <wikibugs>	 (03Merged) 10jenkins-bot: jjb: overhaul Zuul deploy jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1003437 (https://phabricator.wikimedia.org/T342346) (owner: 10Jaime Nuche)
[14:53:37] <wikibugs>	 10Project-Admins: Expansion of cookbooks tag usage across teams - https://phabricator.wikimedia.org/T357895#9555210 (10joanna_borun)
[15:22:00] <wikibugs>	 (03CR) 10Amire80: "Oh, OK. Thanks." [integration/config] - 10https://gerrit.wikimedia.org/r/1003764 (https://phabricator.wikimedia.org/T357670) (owner: 10Amire80)
[15:22:07] <wikibugs>	 (03PS2) 10Amire80: Change <page> to <pages> in message XML tag checks [integration/config] - 10https://gerrit.wikimedia.org/r/1003764 (https://phabricator.wikimedia.org/T357670)
[15:26:35] <wikibugs>	 (03PS3) 10Amire80: Add <pages> to message XML tag checks [integration/config] - 10https://gerrit.wikimedia.org/r/1003764 (https://phabricator.wikimedia.org/T357670)
[15:28:26] <wikibugs>	 10Continuous-Integration-Config, 10I18n, 10Language-Team (Language-2024-January-March), 10Language-Technical Support (Language-Technical Support (Current) ), and 2 others: Automatically allow <pages> tag in message translations - https://phabricator.wikimedia.org/T357670#9555319 (10Amire80)
[15:28:32] <jnuche>	 !log reload zuul for https://gerrit.wikimedia.org/r/1003437
[15:28:34] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[15:32:45] <wikibugs>	 (03CR) 10Jaime Nuche: "recheck" [integration/zuul/deploy] - 10https://gerrit.wikimedia.org/r/1002466 (https://phabricator.wikimedia.org/T342346) (owner: 10Jaime Nuche)
[16:21:59] <wikibugs>	 10Release-Engineering-Team (Now this 🫠), 10Scap, 10collaboration-services, 10User-brennen: scap install fails on new Phabricator/Phorge host due to missing user - https://phabricator.wikimedia.org/T357572#9555600 (10LSobanski) p:05Triage→03Medium
[17:20:08] <wmf-insecte>	 Project beta-update-databases-eqiad build #74031: 04FAILURE in 7.7 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/74031/
[17:21:19] <wikibugs>	 10Release-Engineering-Team (Seen), 10MW-on-K8s, 10SRE, 10Traffic, and 2 others: Move MediaWiki jobs to mw-on-k8s - https://phabricator.wikimedia.org/T349796#9555793 (10hnowlan)
[17:21:27] <wikibugs>	 10Release-Engineering-Team (Seen), 10MW-on-K8s, 10SRE, 10Traffic, and 2 others: Move MediaWiki jobs to mw-on-k8s - https://phabricator.wikimedia.org/T349796#9352065 (10hnowlan)
[18:20:10] <wmf-insecte>	 Project beta-update-databases-eqiad build #74032: 04STILL FAILING in 9.7 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/74032/
[19:32:03] <wmf-insecte>	 Yippee, build fixed!
[19:32:03] <wmf-insecte>	 Project beta-update-databases-eqiad build #74033: 09FIXED in 12 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/74033/
[20:21:32] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] "I have updated the job!" [integration/config] - 10https://gerrit.wikimedia.org/r/1003764 (https://phabricator.wikimedia.org/T357670) (owner: 10Amire80)
[20:23:04] <wikibugs>	 (03Merged) 10jenkins-bot: Add <pages> to message XML tag checks [integration/config] - 10https://gerrit.wikimedia.org/r/1003764 (https://phabricator.wikimedia.org/T357670) (owner: 10Amire80)
[20:27:14] <wikibugs>	 10Continuous-Integration-Config, 10I18n, 10Language-Team (Language-2024-January-March), 10Language-Technical Support (Language-Technical Support (Current) ), and 2 others: Automatically allow <pages> tag in message translations - https://phabricator.wikimedia.org/T357670#9556200 (10hashar) 05Open→03Reso...