[00:01:46] 10Continuous-Integration-Config, 10MediaWiki-General, 10translatewiki.net, 10Documentation, 10MW-1.36-release: Formally EOL REL1_36 - https://phabricator.wikimedia.org/T307531 (10Reedy) [00:02:57] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-CentralAuth: Beta cluster internal error on changing user's global rights (CentralAuthGroupMembershipProxy::getWikiId() undefined) - https://phabricator.wikimedia.org/T307518 (10Reedy) [00:03:57] 10Beta-Cluster-Infrastructure, 10Growth-Team, 10MediaWiki-extensions-CentralAuth, 10Notifications: Beta cluster internal error on changing user's global rights (CentralAuthGroupMembershipProxy::getWikiId() undefined) - https://phabricator.wikimedia.org/T307518 (10Reedy) [00:05:41] 10Beta-Cluster-Infrastructure, 10Growth-Team, 10MediaWiki-extensions-CentralAuth, 10Notifications, 10Regression: Beta cluster internal error on changing user's global rights (CentralAuthGroupMembershipProxy::getWikiId() undefined) - https://phabricator.wikimedia.org/T307518 (10Reedy) Caused by {de4a85b7f... [00:09:42] 10Beta-Cluster-Infrastructure, 10Growth-Team, 10MediaWiki-extensions-CentralAuth, 10Notifications, 10Regression: Beta cluster internal error on changing user's global rights (CentralAuthGroupMembershipProxy::getWikiId() undefined) - https://phabricator.wikimedia.org/T307518 (10Reedy) [00:09:53] 10Release-Engineering-Team (Doing), 10Patch-For-Review, 10Release, 10Train Deployments: 1.39.0-wmf.10 deployment blockers - https://phabricator.wikimedia.org/T305216 (10Reedy) [04:04:46] 10GitLab (Project Migration), 10Release-Engineering-Team (GitLab-a-thon 🦊), 10User-brennen: Mirror a subset of active Gerrit repositories read-only to GitLab instance - https://phabricator.wikimedia.org/T307246 (10brennen) 05Openβ†’03Declined After discussion within the team, we've concluded this probably... [04:04:52] 10GitLab (Project Migration), 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10User-brennen: Mirror all active Gerrit repositories to GitLab instance - https://phabricator.wikimedia.org/T305984 (10brennen) [04:05:11] 10GitLab (Project Migration), 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10User-brennen: Mirror all active Gerrit repositories to GitLab instance - https://phabricator.wikimedia.org/T305984 (10brennen) 05Openβ†’03Stalled [04:07:57] 10GitLab (Project Migration), 10Release-Engineering-Team (GitLab-a-thon 🦊), 10User-dduvall: Create Blubber repo on GitLab, archive Gerrit repo - https://phabricator.wikimedia.org/T307533 (10brennen) [04:08:18] 10GitLab (Project Migration), 10Release-Engineering-Team (GitLab-a-thon 🦊), 10User-dduvall: Create Blubber repo on GitLab, archive Gerrit repo - https://phabricator.wikimedia.org/T307533 (10brennen) [04:10:25] 10GitLab (Project Migration), 10Release-Engineering-Team (GitLab-a-thon 🦊), 10User-dduvall: Implement linting and unit tests for Blubber on GitLab CI - https://phabricator.wikimedia.org/T307534 (10brennen) [04:10:37] 10GitLab (Project Migration), 10Release-Engineering-Team (GitLab-a-thon 🦊), 10User-dduvall: Implement linting and unit tests for Blubber on GitLab CI - https://phabricator.wikimedia.org/T307534 (10brennen) [04:17:25] 10GitLab (Project Migration), 10Release-Engineering-Team (GitLab-a-thon 🦊), 10User-dduvall: Update Blubber documentation, codesearch, and other references for new GitLab location - https://phabricator.wikimedia.org/T307535 (10brennen) [04:17:43] 10GitLab (Project Migration), 10Release-Engineering-Team (GitLab-a-thon 🦊), 10User-dduvall: Update Blubber documentation, codesearch, and other references for new GitLab location - https://phabricator.wikimedia.org/T307535 (10brennen) [04:24:01] 10GitLab (Project Migration), 10Release-Engineering-Team (GitLab-a-thon 🦊): Build Blubber images on GitLab - https://phabricator.wikimedia.org/T307536 (10brennen) [04:29:40] 10GitLab (Administration, Settings & Policy), 10Release-Engineering-Team (GitLab-a-thon 🦊), 10cloud-services-team (Kanban): Assess GitLab-provided docker container registry as a default for docker-in-docker build processes - https://phabricator.wikimedia.org/T307537 (10brennen) p:05Triageβ†’03Medium [04:29:51] 10GitLab (Project Migration), 10Release-Engineering-Team (GitLab-a-thon 🦊): Build Blubber images on GitLab - https://phabricator.wikimedia.org/T307536 (10brennen) [04:29:59] 10GitLab (Administration, Settings & Policy), 10Release-Engineering-Team (GitLab-a-thon 🦊), 10cloud-services-team (Kanban): Assess GitLab-provided docker container registry as a default for docker-in-docker build processes - https://phabricator.wikimedia.org/T307537 (10brennen) [04:30:49] 10GitLab (Project Migration), 10Release-Engineering-Team (GitLab-a-thon 🦊): Build Blubber images on GitLab - https://phabricator.wikimedia.org/T307536 (10brennen) [04:30:55] 10GitLab (Project Migration), 10Release-Engineering-Team (GitLab-a-thon 🦊), 10User-dduvall: Migrate Blubber project to GitLab - https://phabricator.wikimedia.org/T301168 (10brennen) [04:35:48] 10GitLab (Project Migration), 10Release-Engineering-Team (GitLab-a-thon 🦊), 10User-dduvall: Write a GitLab "Migrating a Project" runbook / manual based on Blubber migration - https://phabricator.wikimedia.org/T307538 (10brennen) [04:35:56] 10GitLab (Project Migration), 10Release-Engineering-Team (GitLab-a-thon 🦊), 10User-dduvall: Write a GitLab "Migrating a Project" runbook / manual based on Blubber migration - https://phabricator.wikimedia.org/T307538 (10brennen) [05:04:13] 10Beta-Cluster-Infrastructure, 10Growth-Team, 10MediaWiki-extensions-CentralAuth, 10Notifications, 10Regression: Beta cluster internal error on changing user's global rights (CentralAuthGroupMembershipProxy::getWikiId() undefined) - https://phabricator.wikimedia.org/T307518 (10Majavah) [05:04:23] 10Release-Engineering-Team (Doing), 10Patch-For-Review, 10Release, 10Train Deployments: 1.39.0-wmf.10 deployment blockers - https://phabricator.wikimedia.org/T305216 (10Majavah) [05:05:35] 10Beta-Cluster-Infrastructure, 10Growth-Team, 10MediaWiki-extensions-CentralAuth, 10Notifications, 10Regression: Beta cluster internal error on changing user's global rights (CentralAuthGroupMembershipProxy::getWikiId() undefined) - https://phabricator.wikimedia.org/T307518 (10Majavah) [05:05:45] 10Release-Engineering-Team (Doing), 10Patch-For-Review, 10Release, 10Train Deployments: 1.39.0-wmf.10 deployment blockers - https://phabricator.wikimedia.org/T305216 (10Majavah) [05:30:50] 10GitLab (Project Migration), 10Release-Engineering-Team (GitLab-a-thon 🦊): Establish image trust system for GitLab/Blubber - https://phabricator.wikimedia.org/T307541 (10brennen) [06:08:54] 10Release-Engineering-Team (Radar), 10Infrastructure-Foundations, 10SRE, 10SRE-Access-Requests, 10serviceops: Need a service account on deploy servers for automated train pre-sync operations - https://phabricator.wikimedia.org/T303857 (10Joe) >>! In T303857#7896843, @dancy wrote: > @Joe Pinging on this t... [06:54:23] 10Release-Engineering-Team (Doing), 10Patch-For-Review, 10Release, 10Train Deployments: 1.39.0-wmf.10 deployment blockers - https://phabricator.wikimedia.org/T305216 (10AlexisJazz) >>! In T305216#7899173, @hashar wrote: > I have looked at the new errors in the log and it is quiet. There are a very few show... [07:02:24] 10Release-Engineering-Team (Doing), 10Patch-For-Review, 10Release, 10Train Deployments: 1.39.0-wmf.10 deployment blockers - https://phabricator.wikimedia.org/T305216 (10RhinosF1) >>! In T305216#7902012, @AlexisJazz wrote: >>>! In T305216#7899173, @hashar wrote: >> I have looked at the new errors in the log... [07:44:38] (03CR) 10Hashar: [C: 04-1] "Nice, there are a few bash nitpick and some hints about handling phpcs exit code." [tools/code-utils] - 10https://gerrit.wikimedia.org/r/788384 (owner: 10Daniel Kinzler) [07:47:01] (03CR) 10Jaime Nuche: [C: 03+2] Report rsync averages and totals. [tools/scap] - 10https://gerrit.wikimedia.org/r/788765 (owner: 10Ahmon Dancy) [07:47:56] (03Merged) 10jenkins-bot: Report rsync averages and totals. [tools/scap] - 10https://gerrit.wikimedia.org/r/788765 (owner: 10Ahmon Dancy) [07:49:01] (03CR) 10Hashar: [C: 04-1] "That is pending documentation at https://doc.wikimedia.org/VisualEditor-standalone/master/ :)" [integration/docroot] - 10https://gerrit.wikimedia.org/r/131323 (owner: 10Jforrester) [08:19:42] 10Release-Engineering-Team (Doing), 10Patch-For-Review, 10Release, 10Train Deployments: 1.39.0-wmf.10 deployment blockers - https://phabricator.wikimedia.org/T305216 (10Majavah) [08:29:36] [[phab:T307287]] any idea why this is happening? [08:29:36] T307287: Gerrit produces"Authentication failed" error when attempting to log in - https://phabricator.wikimedia.org/T307287 [08:41:26] hashar: ^ [08:58:25] RhinosF1: will check after the mw train [09:00:25] 10Gerrit: Gerrit produces"Authentication failed" error when attempting to log in - https://phabricator.wikimedia.org/T307287 (10hashar) a:03hashar I am currently updating MediaWiki on the Wikimedia cluster, will check the issue you are encountering once I am done. [09:04:44] 10Release-Engineering-Team (Doing), 10Scap: Stage new scap release - https://phabricator.wikimedia.org/T307086 (10jnuche) [09:20:20] 10Release-Engineering-Team (Doing), 10Patch-For-Review, 10Release, 10Train Deployments: 1.39.0-wmf.10 deployment blockers - https://phabricator.wikimedia.org/T305216 (10hashar) [09:34:41] 10Gerrit: Gerrit produces"Authentication failed" error when attempting to log in - https://phabricator.wikimedia.org/T307287 (10hashar) The account is deactivated in Gerrit since April 4th 2022: ` commit 6223addd4f84f67216a4546c3026f8a9c7f7fb13 (HEAD -> users/98/9298, origin/users/98/9298) Author: [BOT] Gerrit C... [10:06:47] 10Gerrit: Gerrit produces"Authentication failed" error when attempting to log in - https://phabricator.wikimedia.org/T307287 (10hashar) The reason is we issued a block for Wikitech user `epic` at https://wikitech.wikimedia.org/wiki/Special:BlockList/epic . We have a hook in https://gerrit.wikimedia.org/g/operati... [10:09:31] 10Gerrit: Gerrit produces"Authentication failed" error when attempting to log in - https://phabricator.wikimedia.org/T307287 (10hashar) I have manually enabled the account using the account id: `gerrit set-account 9298 --active` Should be good now. [10:11:35] 10Gerrit, 10wikitech.wikimedia.org: wikitech hook to disable Gerrit user uses partial matches to identify account - https://phabricator.wikimedia.org/T307558 (10hashar) [10:11:49] 10Gerrit: Gerrit produces"Authentication failed" error when attempting to log in - https://phabricator.wikimedia.org/T307287 (10hashar) 05Openβ†’03Resolved Follow up action for us is T307558 [10:12:45] RhinosF1: I have fixed it. The reason is we blocked the account 'epic' on wikitech then the way we issue the block to Gerrit cause it to look up for any account that match 'epic', that has yield 'EpicPupper' which ended up blocked [10:12:53] I have activated it again [10:12:56] solved :] [10:13:01] out for lunch [10:22:00] hashar: ah, that seems like a bug [10:22:11] You filed one! [10:28:31] Yes πŸ˜„ [11:00:51] 10Beta-Cluster-Infrastructure: Create "Global File Deletion Reviewer" group on Beta Cluster - https://phabricator.wikimedia.org/T307432 (10TheresNoTime) 05Stalledβ†’03Resolved T307518 has been resolved, and I have added you to the global group. Closing 😸 [11:16:18] 10Release-Engineering-Team (Doing), 10Scap: Puppet config for scap-over-scap - https://phabricator.wikimedia.org/T306991 (10jnuche) [11:40:49] 10Release-Engineering-Team (Doing), 10Scap: Puppet config for scap-over-scap - https://phabricator.wikimedia.org/T306991 (10jnuche) [12:03:52] James_F: for the GoogleNewsSitemap https://phabricator.wikimedia.org/T307552#7902627 shouldn't we revert https://gerrit.wikimedia.org/r/c/mediawiki/extensions/FlaggedRevs/+/773242 ? [12:04:20] I mean by using the revert button :D [12:12:34] anyway I have rebased [12:17:38] 10Beta-Cluster-Infrastructure, 10Commons: Story namespace on betaenwiki uses production Commons - https://phabricator.wikimedia.org/T307564 (10AlexisJazz) [12:19:55] hashar: I wrote my patch before I found that I wasn't just fixing things, I was reverting things. [12:19:56] 10Gerrit: Gerrit produces"Authentication failed" error when attempting to log in - https://phabricator.wikimedia.org/T307287 (10EpicPupper) Haha, thanks so much! [12:20:13] James_F: ah that makes sense [12:20:23] hashar: wmf.10 cherry-pick: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/FlaggedRevs/+/788861 [12:20:30] you might want to give a bit more details to thiemo as to why stubbing cause troubles [12:22:03] I did. [12:30:35] what I am wondering is when GoogleNewsSitemap had the classes namespaces, shouldn't we have Phan running against FlaggedRevs since it is using it ? [12:31:02] seems like we miss checking the reverse dependencies [12:34:31] 10Beta-Cluster-Infrastructure, 10Abstract Wikipedia team, 10Patch-For-Review: Create a Beta Cluster version of Wikifunctions.org - https://phabricator.wikimedia.org/T284162 (10Jdforrester-WMF) >>! In T284162#7901490, @ori wrote: > Automatic tagging of 'latest' should be fixed with https://gerrit.wikimedia.or... [12:42:01] (03PS1) 10Jforrester: jjb: [mediawiki/services/function-evaluator] Provide bespoke pipeline jobs [integration/config] - 10https://gerrit.wikimedia.org/r/789140 (https://phabricator.wikimedia.org/T307507) [12:42:03] (03PS1) 10Jforrester: Zuul: [mediawiki/services/function-evaluator] Use bespoke pipeline jobs only [integration/config] - 10https://gerrit.wikimedia.org/r/789141 (https://phabricator.wikimedia.org/T307507) [12:42:05] (03PS1) 10Jforrester: jjb: [mediawiki/services/function-evaluator] Drop duplicate test-only pipeline job [integration/config] - 10https://gerrit.wikimedia.org/r/789142 (https://phabricator.wikimedia.org/T307507) [12:44:14] (03CR) 10jerkins-bot: [V: 04-1] jjb: [mediawiki/services/function-evaluator] Drop duplicate test-only pipeline job [integration/config] - 10https://gerrit.wikimedia.org/r/789142 (https://phabricator.wikimedia.org/T307507) (owner: 10Jforrester) [12:51:22] 10Release-Engineering-Team (Doing), 10Patch-For-Review, 10Release, 10Train Deployments: 1.39.0-wmf.10 deployment blockers - https://phabricator.wikimedia.org/T305216 (10hashar) [13:16:35] 10Release-Engineering-Team, 10MediaWiki-Debug-Logger, 10MediaWiki-Maintenance-system: Maintenance scripts should consistently log errors - https://phabricator.wikimedia.org/T307402 (10Krinkle) Thanks for raising this. We ran into a similar issue last year as part of the ParserCache epic (T282761). My take o... [13:26:41] (Queue (Jenkins jobs + Zuul functions) alert) firing: Queue (Jenkins jobs + Zuul functions) alert - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert [13:27:22] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Next), 10User-brennen: Provision untrusted instance-wide GitLab job runners to handle user-level projects and merge requests from forks - https://phabricator.wikimedia.org/T297426 (10Jelto) https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner... [13:31:55] 10Release-Engineering-Team (Doing), 10Patch-For-Review, 10Release, 10Train Deployments: 1.39.0-wmf.10 deployment blockers - https://phabricator.wikimedia.org/T305216 (10hashar) [13:32:26] 10GitLab (Project Migration), 10Release-Engineering-Team (GitLab-a-thon 🦊): Establish image trust system for GitLab/Blubber - https://phabricator.wikimedia.org/T307541 (10Jelto) That's related to T295481. Trust is most probably established by using dedicated and more secure runners. So this builds will need t... [13:32:32] 10Continuous-Integration-Infrastructure, 10MinervaNeue, 10Vector, 10Accessibility, and 3 others: Add automated accessibility tests in CI to generate accessibility benchmarks for Skins - https://phabricator.wikimedia.org/T301184 (10zeljkofilipin) From the job terminal output ([[ https://integration.wikimedi... [13:32:54] 10Release-Engineering-Team (Doing), 10Patch-For-Review, 10Release, 10Train Deployments: 1.39.0-wmf.10 deployment blockers - https://phabricator.wikimedia.org/T305216 (10hashar) group 1 wikis have been promoted to 1.39.0-wmf.10 confirming all blockers have indeed been fully resolved. πŸš‹ 🚭 [13:38:05] (03CR) 10Jforrester: [C: 03+2] "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/789140 (https://phabricator.wikimedia.org/T307507) (owner: 10Jforrester) [13:38:16] (03CR) 10Jforrester: [C: 03+2] Zuul: [mediawiki/services/function-evaluator] Use bespoke pipeline jobs only [integration/config] - 10https://gerrit.wikimedia.org/r/789141 (https://phabricator.wikimedia.org/T307507) (owner: 10Jforrester) [13:38:28] 10Continuous-Integration-Config, 10Toolhub, 10Documentation, 10User-Slst2020: Publish docs on doc.wikimedia.org - https://phabricator.wikimedia.org/T305914 (10hashar) @dduvall @jeena do we have past experience with a PipelineLib enabled repository publishing artifacts to doc.wikimedia.org ? I couldn't find... [13:38:50] (03PS2) 10Jforrester: jjb: [mediawiki/services/function-evaluator] Drop duplicate pipeline jobs [integration/config] - 10https://gerrit.wikimedia.org/r/789142 (https://phabricator.wikimedia.org/T307507) [13:39:15] (03CR) 10Jforrester: [C: 03+2] jjb: [mediawiki/services/function-evaluator] Drop duplicate pipeline jobs [integration/config] - 10https://gerrit.wikimedia.org/r/789142 (https://phabricator.wikimedia.org/T307507) (owner: 10Jforrester) [13:40:03] (03Merged) 10jenkins-bot: jjb: [mediawiki/services/function-evaluator] Provide bespoke pipeline jobs [integration/config] - 10https://gerrit.wikimedia.org/r/789140 (https://phabricator.wikimedia.org/T307507) (owner: 10Jforrester) [13:40:07] (03Merged) 10jenkins-bot: Zuul: [mediawiki/services/function-evaluator] Use bespoke pipeline jobs only [integration/config] - 10https://gerrit.wikimedia.org/r/789141 (https://phabricator.wikimedia.org/T307507) (owner: 10Jforrester) [13:41:13] (03Merged) 10jenkins-bot: jjb: [mediawiki/services/function-evaluator] Drop duplicate pipeline jobs [integration/config] - 10https://gerrit.wikimedia.org/r/789142 (https://phabricator.wikimedia.org/T307507) (owner: 10Jforrester) [13:41:41] (Queue (Jenkins jobs + Zuul functions) alert) firing: (2) Queue (Jenkins jobs + Zuul functions) alert - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert [13:43:59] 10Continuous-Integration-Config, 10Release-Engineering-Team (Doing), 10Release Pipeline, 10Patch-For-Review, 10User-dduvall: Fully deprecate service-pipeline-test and service-pipeline-test-and-publish jobs - https://phabricator.wikimedia.org/T307507 (10Jdforrester-WMF) [13:44:25] !log Zuul: [mediawiki/services/function-evaluator] Use bespoke pipeline jobs only T307507 [13:44:27] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:44:27] T307507: Fully deprecate service-pipeline-test and service-pipeline-test-and-publish jobs - https://phabricator.wikimedia.org/T307507 [14:01:41] (Queue (Jenkins jobs + Zuul functions) alert) resolved: Queue (Jenkins jobs + Zuul functions) alert - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert [14:19:27] I have possibly done something a little dumb - created an account to push through a global rename on the beta cluster, and it feels a little stuck ( https://meta.wikimedia.beta.wmflabs.org/wiki/Special:GlobalRenameProgress/TheresNoTime ), though I'm not sure how long a rename there would normally take (been ~2 hours so far)... can't see any errors on beta-logs.wmcloud.org [14:21:00] * TheresNoTime would work through https://wikitech.wikimedia.org/wiki/Stuck_global_renames to try to fix her muck up if she could :(( [14:22:51] 10Continuous-Integration-Infrastructure, 10MinervaNeue, 10Vector, 10Accessibility, and 3 others: Add automated accessibility tests in CI to generate accessibility benchmarks for Skins - https://phabricator.wikimedia.org/T301184 (10zeljkofilipin) Looks like `archive-log-allow-empty` publisher is actually ar... [14:29:41] 10Beta-Cluster-Infrastructure: Stuck rename on Beta Cluster - https://phabricator.wikimedia.org/T307590 (10TheresNoTime) [14:30:27] 10Beta-Cluster-Infrastructure, 10GlobalRename, 10MediaWiki-extensions-CentralAuth: Stuck rename on Beta Cluster - https://phabricator.wikimedia.org/T307590 (10TheresNoTime) [14:32:32] 10Beta-Cluster-Infrastructure: Stuck rename on Beta Cluster - https://phabricator.wikimedia.org/T307590 (10Majavah) [14:33:00] TheresNoTime: sounds like an issue with the deployment-prep job queue in general :/ [14:33:16] ah okay, thanks for removing them taavi [14:34:07] I know ottomata was working on the kafka nodes in T304433, so maybe that's related? [14:34:08] T304433: Upgrade event platform related VMs in deployment-prep to Debian bullsye (or buster) - https://phabricator.wikimedia.org/T304433 [14:42:53] taavi: i did change the kafka brokers, its possible that I missed some other configuration setting for job queue consumers [14:43:53] huh i htink i did [14:44:09] https://gerrit.wikimedia.org/g/operations/deployment-charts/+/8149dcadf8de4aba6fbf8eed9b1e44e4862dbc74/charts/changeprop/values-beta-changeprop.yaml https://gerrit.wikimedia.org/g/operations/deployment-charts/+/8149dcadf8de4aba6fbf8eed9b1e44e4862dbc74/charts/changeprop/values-beta-jobqueue.yaml [14:44:20] cp/jobqueue is running in beta in k8s?!?!?!? [14:44:26] didn't even know that was possible [14:44:48] maybe not... [14:44:48] https://gerrit.wikimedia.org/r/plugins/gitiles/operations/deployment-charts/+/8149dcadf8de4aba6fbf8eed9b1e44e4862dbc74/charts/changeprop#generating-beta-configurations [14:46:12] ottomata: was it perhaps your changes then? :) [14:46:46] most likely [14:47:02] the kafka brokers cp and jobqueue are configured to use no longer exist [14:49:47] investigating [14:52:13] :D [15:10:21] 10Release-Engineering-Team (Doing), 10Patch-For-Review, 10Release, 10Train Deployments: 1.39.0-wmf.10 deployment blockers - https://phabricator.wikimedia.org/T305216 (10hashar) I have pocked Slack `#engineering-all` , wikitech-l and ops-l about the T307586 blocker. [15:12:17] Lucas_WMDE: hi again ;) I am guessing the issue with wbsearchentities is going to be a bit challenging to figure out. At least I am around tomorrow so I can do the backports / promote etc [15:12:47] else for this evening we have plenty of meeting but brennen is my back up and on west coast of us. In case there is a fix this evening [15:13:17] yeah, i can roll forward if we get a fix during the US day. [15:13:17] taavi: there's been a few reports about jobs on train task [15:15:49] ottomata: are you tracking beta jobs anywhere? They are 3 tasks I can see open with the cause being jobs not running. [15:15:57] They probably should be merged [15:23:43] James_F: thanks for handling function-evaluator! [15:24:52] i'm going to steal your use of "bespoke" as a descriptor since i have been struggling to find a word that differentiates .pipeline/config.yaml based pipelines from the general pipelines. perfect [15:35:11] Happy to. Go for it. :-) [15:36:22] (03PS1) 10Dduvall: jjb: [mediawiki/tools/cli] Provide bespoke pipeline jobs [integration/config] - 10https://gerrit.wikimedia.org/r/789194 (https://phabricator.wikimedia.org/T307507) [15:36:24] (03PS1) 10Dduvall: zuul: [mediawiki/tools/cli] Use bespoke pipeline jobs only [integration/config] - 10https://gerrit.wikimedia.org/r/789195 (https://phabricator.wikimedia.org/T307507) [15:40:18] (03PS2) 10Dduvall: jjb: [mediawiki/tools/cli] Provide bespoke pipeline jobs [integration/config] - 10https://gerrit.wikimedia.org/r/789194 (https://phabricator.wikimedia.org/T307507) [15:40:20] (03PS2) 10Dduvall: zuul: [mediawiki/tools/cli] Use bespoke pipeline jobs only [integration/config] - 10https://gerrit.wikimedia.org/r/789195 (https://phabricator.wikimedia.org/T307507) [15:42:16] !log created 2 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789194 [15:42:17] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:42:35] (03CR) 10Dduvall: [C: 03+2] jjb: [mediawiki/tools/cli] Provide bespoke pipeline jobs [integration/config] - 10https://gerrit.wikimedia.org/r/789194 (https://phabricator.wikimedia.org/T307507) (owner: 10Dduvall) [15:42:51] (03CR) 10Dduvall: [C: 03+2] zuul: [mediawiki/tools/cli] Use bespoke pipeline jobs only [integration/config] - 10https://gerrit.wikimedia.org/r/789195 (https://phabricator.wikimedia.org/T307507) (owner: 10Dduvall) [15:44:33] (03Merged) 10jenkins-bot: jjb: [mediawiki/tools/cli] Provide bespoke pipeline jobs [integration/config] - 10https://gerrit.wikimedia.org/r/789194 (https://phabricator.wikimedia.org/T307507) (owner: 10Dduvall) [15:44:45] (03Merged) 10jenkins-bot: zuul: [mediawiki/tools/cli] Use bespoke pipeline jobs only [integration/config] - 10https://gerrit.wikimedia.org/r/789195 (https://phabricator.wikimedia.org/T307507) (owner: 10Dduvall) [15:47:53] 10Continuous-Integration-Config, 10Release-Engineering-Team (Doing), 10Release Pipeline, 10Patch-For-Review, 10User-dduvall: Fully deprecate service-pipeline-test and service-pipeline-test-and-publish jobs - https://phabricator.wikimedia.org/T307507 (10dduvall) [15:56:41] (03PS1) 10Dduvall: jjb: [mediawiki/tools/scap] Provide bespoke pipeline jobs [integration/config] - 10https://gerrit.wikimedia.org/r/789203 (https://phabricator.wikimedia.org/T307507) [15:56:43] (03PS1) 10Dduvall: zuul: [mediawiki/tools/scap] Use bespoke pipeline jobs only [integration/config] - 10https://gerrit.wikimedia.org/r/789204 (https://phabricator.wikimedia.org/T307507) [16:01:18] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789195 [16:01:19] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:01:54] !log created 2 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789203 [16:01:55] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:02:23] (03CR) 10Dduvall: [C: 03+2] jjb: [mediawiki/tools/scap] Provide bespoke pipeline jobs [integration/config] - 10https://gerrit.wikimedia.org/r/789203 (https://phabricator.wikimedia.org/T307507) (owner: 10Dduvall) [16:04:28] (03Merged) 10jenkins-bot: jjb: [mediawiki/tools/scap] Provide bespoke pipeline jobs [integration/config] - 10https://gerrit.wikimedia.org/r/789203 (https://phabricator.wikimedia.org/T307507) (owner: 10Dduvall) [16:04:42] (03CR) 10Dduvall: [C: 03+2] zuul: [mediawiki/tools/scap] Use bespoke pipeline jobs only [integration/config] - 10https://gerrit.wikimedia.org/r/789204 (https://phabricator.wikimedia.org/T307507) (owner: 10Dduvall) [16:07:02] (03Merged) 10jenkins-bot: zuul: [mediawiki/tools/scap] Use bespoke pipeline jobs only [integration/config] - 10https://gerrit.wikimedia.org/r/789204 (https://phabricator.wikimedia.org/T307507) (owner: 10Dduvall) [16:11:46] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789204 [16:11:47] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:14:09] taavi: RhinosF1 TheresNoTime, hnowlan merged the config change fix [16:14:18] i know very litttle about job queue, especially in beta [16:14:21] ottomata: cool [16:14:33] if the change it applied, the consumers should be using the correct configs now [16:14:33] 10Continuous-Integration-Config, 10Release-Engineering-Team (Doing), 10Release Pipeline, 10Patch-For-Review, 10User-dduvall: Fully deprecate service-pipeline-test and service-pipeline-test-and-publish jobs - https://phabricator.wikimedia.org/T307507 (10dduvall) [16:14:50] my job queue knowledge is something that's very good at exploding at miraheze in spectacular fashion [16:14:56] so I assume job queueu will start worrking again? dunno how it manages catching up tho. hopefully it uses kafka offsets and will just start from backlog [16:15:33] should do [16:30:22] 10Beta-Cluster-Infrastructure: Stuck rename on Beta Cluster - https://phabricator.wikimedia.org/T307590 (10TheresNoTime) Apparently possibly related to T304433, which has recently had a patch merged to resolve 😸 [16:36:32] 10GitLab (Project Migration), 10Release-Engineering-Team (GitLab-a-thon 🦊): Establish image trust system for GitLab/Blubber - https://phabricator.wikimedia.org/T307541 (10thcipriani) Things to look investigate: - ContentTrustCLI - Concept of using signing to establish trust - CNCF paper about image signing [16:40:53] 10GitLab (Project Migration), 10Release-Engineering-Team (GitLab-a-thon 🦊): Establish image trust system for GitLab/Blubber - https://phabricator.wikimedia.org/T307541 (10thcipriani) [16:43:02] 10GitLab (Administration, Settings & Policy), 10Release-Engineering-Team (GitLab-a-thon 🦊), 10cloud-services-team (Kanban): Assess GitLab-provided docker container registry as a default for docker-in-docker build processes - https://phabricator.wikimedia.org/T307537 (10thcipriani) Related to {T307541} [16:45:01] 10GitLab (Project Migration), 10Release-Engineering-Team (GitLab-a-thon 🦊), 10User-dduvall: Update Blubber documentation, codesearch, and other references for new GitLab location - https://phabricator.wikimedia.org/T307535 (10thcipriani) Possible other references in various subpages of https://wikitech.wikim... [16:50:58] 10Release-Engineering-Team (GitLab-a-thon 🦊): Investigate alternatives to docker-in-docker for container image creation in GitLab - https://phabricator.wikimedia.org/T307599 (10thcipriani) [16:51:32] 10Release-Engineering-Team (GitLab-a-thon 🦊): Investigate alternatives to docker-in-docker for container image creation in GitLab - https://phabricator.wikimedia.org/T307599 (10thcipriani) [16:52:18] 10Release-Engineering-Team (GitLab-a-thon 🦊): Investigate alternatives to docker-in-docker for container image creation in GitLab - https://phabricator.wikimedia.org/T307599 (10thcipriani) [16:52:21] 10GitLab (Project Migration), 10Release-Engineering-Team (GitLab-a-thon 🦊): Build Blubber images on GitLab - https://phabricator.wikimedia.org/T307536 (10thcipriani) [16:52:28] 10GitLab, 10Release-Engineering-Team (GitLab-a-thon 🦊): Investigate alternatives to docker-in-docker for container image creation in GitLab - https://phabricator.wikimedia.org/T307599 (10thcipriani) [16:53:12] 10GitLab (Project Migration), 10Release-Engineering-Team (GitLab-a-thon 🦊): Build Blubber images on GitLab - https://phabricator.wikimedia.org/T307536 (10thcipriani) [16:54:15] 10GitLab (Project Migration), 10Release-Engineering-Team (GitLab-a-thon 🦊): Build Blubber images on GitLab - https://phabricator.wikimedia.org/T307536 (10thcipriani) [16:55:39] 10GitLab (Project Migration), 10Release-Engineering-Team (GitLab-a-thon 🦊), 10User-dduvall: Implement linting and unit tests for Blubber on GitLab CI - https://phabricator.wikimedia.org/T307534 (10thcipriani) Changing estimate per discussion in team meeting [16:58:30] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (GitLab-a-thon 🦊), 10Security-Team, 10Patch-For-Review, and 2 others: Limit GitLab shared runners to images from Wikimedia Docker registry - https://phabricator.wikimedia.org/T291978 (10thcipriani) [17:00:13] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (GitLab-a-thon 🦊), 10Security-Team, 10Patch-For-Review, and 2 others: Limit GitLab shared runners to images from Wikimedia Docker registry - https://phabricator.wikimedia.org/T291978 (10thcipriani) Talked about doing this as part of #gitlabsperiment n... [17:11:57] (03PS1) 10Dduvall: jjb: [operations/software/tegola] Provide bespoke pipeline jobs [integration/config] - 10https://gerrit.wikimedia.org/r/789217 (https://phabricator.wikimedia.org/T307507) [17:11:59] (03PS1) 10Dduvall: zuul: [operations/software/tegola] Use bespoke pipeline jobs only [integration/config] - 10https://gerrit.wikimedia.org/r/789218 (https://phabricator.wikimedia.org/T307507) [17:12:12] !log created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789217 [17:12:14] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:14:37] ebernhardson: hello, for the wbsearchentities giving empty results ( T307586 ), if you need deployment assistance brennen is the back up I guess you can ping here :) [17:14:37] T307586: wbsearchentities produces no results on 1.39.0-wmf.10 - https://phabricator.wikimedia.org/T307586 [17:14:50] ebernhardson: and I can follow up with Lucas and David tomorrow morning :] [17:14:55] thanks for stepping in! [17:15:22] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Cloud-VPS (Debian Stretch Deprecation): Cloud VPS "integration" project Stretch deprecation - https://phabricator.wikimedia.org/T306078 (10komla) @hashar has requested for an exemption. [17:15:49] 10GitLab, 10Release-Engineering-Team (GitLab-a-thon 🦊): Investigate alternatives to docker-in-docker for container image creation in GitLab - https://phabricator.wikimedia.org/T307599 (10dduvall) [17:16:05] (03CR) 10Dduvall: [C: 03+2] jjb: [operations/software/tegola] Provide bespoke pipeline jobs [integration/config] - 10https://gerrit.wikimedia.org/r/789217 (https://phabricator.wikimedia.org/T307507) (owner: 10Dduvall) [17:16:26] (03CR) 10Dduvall: [C: 03+2] zuul: [operations/software/tegola] Use bespoke pipeline jobs only [integration/config] - 10https://gerrit.wikimedia.org/r/789218 (https://phabricator.wikimedia.org/T307507) (owner: 10Dduvall) [17:18:47] (03Merged) 10jenkins-bot: jjb: [operations/software/tegola] Provide bespoke pipeline jobs [integration/config] - 10https://gerrit.wikimedia.org/r/789217 (https://phabricator.wikimedia.org/T307507) (owner: 10Dduvall) [17:18:49] (03Merged) 10jenkins-bot: zuul: [operations/software/tegola] Use bespoke pipeline jobs only [integration/config] - 10https://gerrit.wikimedia.org/r/789218 (https://phabricator.wikimedia.org/T307507) (owner: 10Dduvall) [17:22:30] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789218 [17:22:31] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:23:59] 10GitLab, 10Release-Engineering-Team (GitLab-a-thon 🦊): Investigate alternatives to docker-in-docker for container image creation in GitLab - https://phabricator.wikimedia.org/T307599 (10hashar) [17:26:46] * hashar diner time [17:30:18] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen), 10Cloud-VPS (Debian Stretch Deprecation), 10Patch-For-Review: Move all Wikimedia CI (WMCS integration project) instances from stretch to buster/bullseye - https://phabricator.wikimedia.org/T252071 (10komla) @hashar has requested fo... [17:30:46] (03PS1) 10Dduvall: ci: Provide basic `.pipeline/config.yaml` [tools/scap] - 10https://gerrit.wikimedia.org/r/789223 (https://phabricator.wikimedia.org/T307507) [17:36:22] (03CR) 10Dduvall: [C: 03+2] ci: Provide basic `.pipeline/config.yaml` [tools/scap] - 10https://gerrit.wikimedia.org/r/789223 (https://phabricator.wikimedia.org/T307507) (owner: 10Dduvall) [17:37:11] 10Continuous-Integration-Config, 10Release-Engineering-Team (Doing), 10Release Pipeline, 10Patch-For-Review, 10User-dduvall: Fully deprecate service-pipeline-test and service-pipeline-test-and-publish jobs - https://phabricator.wikimedia.org/T307507 (10dduvall) [17:39:34] (03Merged) 10jenkins-bot: ci: Provide basic `.pipeline/config.yaml` [tools/scap] - 10https://gerrit.wikimedia.org/r/789223 (https://phabricator.wikimedia.org/T307507) (owner: 10Dduvall) [17:46:55] (03PS1) 10Dduvall: jjb: [mediawiki/services/apertium] Provide bespoke pipeline jobs [integration/config] - 10https://gerrit.wikimedia.org/r/789225 (https://phabricator.wikimedia.org/T307507) [17:46:57] (03PS1) 10Dduvall: zuul: [mediawiki/services/apertium] Use bespoke pipeline jobs only [integration/config] - 10https://gerrit.wikimedia.org/r/789226 (https://phabricator.wikimedia.org/T307507) [17:48:44] (03PS2) 10Dduvall: jjb: [mediawiki/services/apertium] Provide bespoke pipeline jobs [integration/config] - 10https://gerrit.wikimedia.org/r/789225 (https://phabricator.wikimedia.org/T307507) [17:48:46] (03PS2) 10Dduvall: zuul: [mediawiki/services/apertium] Use bespoke pipeline jobs only [integration/config] - 10https://gerrit.wikimedia.org/r/789226 (https://phabricator.wikimedia.org/T307507) [17:48:50] (03CR) 10jerkins-bot: [V: 04-1] jjb: [mediawiki/services/apertium] Provide bespoke pipeline jobs [integration/config] - 10https://gerrit.wikimedia.org/r/789225 (https://phabricator.wikimedia.org/T307507) (owner: 10Dduvall) [17:51:04] !log created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789225 [17:51:05] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:51:36] (03CR) 10Dduvall: [C: 03+2] jjb: [mediawiki/services/apertium] Provide bespoke pipeline jobs [integration/config] - 10https://gerrit.wikimedia.org/r/789225 (https://phabricator.wikimedia.org/T307507) (owner: 10Dduvall) [17:52:40] (03CR) 10Dduvall: [C: 03+2] zuul: [mediawiki/services/apertium] Use bespoke pipeline jobs only [integration/config] - 10https://gerrit.wikimedia.org/r/789226 (https://phabricator.wikimedia.org/T307507) (owner: 10Dduvall) [17:53:24] (03Merged) 10jenkins-bot: jjb: [mediawiki/services/apertium] Provide bespoke pipeline jobs [integration/config] - 10https://gerrit.wikimedia.org/r/789225 (https://phabricator.wikimedia.org/T307507) (owner: 10Dduvall) [17:54:33] (03Merged) 10jenkins-bot: zuul: [mediawiki/services/apertium] Use bespoke pipeline jobs only [integration/config] - 10https://gerrit.wikimedia.org/r/789226 (https://phabricator.wikimedia.org/T307507) (owner: 10Dduvall) [17:55:33] !Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789226 [18:04:03] ref T307590 ( https://meta.wikimedia.beta.wmflabs.org/wiki/Special:GlobalRenameProgress/TheresNoTime being stuck due to maybe T304433 ), can someone give the job queue on the beta cluster a poke? [18:04:04] T304433: Upgrade event platform related VMs in deployment-prep to Debian bullsye (or buster) - https://phabricator.wikimedia.org/T304433 [18:04:04] T307590: Stuck rename on Beta Cluster - https://phabricator.wikimedia.org/T307590 [18:05:55] (03PS1) 10Jeena Huneidi: scap backport: add options stop-before-sync, yes [tools/scap] - 10https://gerrit.wikimedia.org/r/789230 [18:07:51] (03PS1) 10Stang: zuul: Add new email address of Stang [integration/config] - 10https://gerrit.wikimedia.org/r/789232 [18:11:51] (03PS1) 10Jeena Huneidi: scap backport: some ssh changes [tools/scap] - 10https://gerrit.wikimedia.org/r/789233 (https://phabricator.wikimedia.org/T295495) [18:13:43] (03PS2) 10Jeena Huneidi: scap backport: some ssh changes [tools/scap] - 10https://gerrit.wikimedia.org/r/789233 (https://phabricator.wikimedia.org/T295495) [18:18:21] (03CR) 10Jeena Huneidi: [C: 03+2] scap backport: some ssh changes [tools/scap] - 10https://gerrit.wikimedia.org/r/789233 (https://phabricator.wikimedia.org/T295495) (owner: 10Jeena Huneidi) [18:19:34] (03Merged) 10jenkins-bot: scap backport: some ssh changes [tools/scap] - 10https://gerrit.wikimedia.org/r/789233 (https://phabricator.wikimedia.org/T295495) (owner: 10Jeena Huneidi) [18:19:43] 10Continuous-Integration-Config, 10Release-Engineering-Team (Doing), 10Release Pipeline, 10Patch-For-Review, 10User-dduvall: Fully deprecate service-pipeline-test and service-pipeline-test-and-publish jobs - https://phabricator.wikimedia.org/T307507 (10dduvall) [18:23:48] (03PS1) 10Dduvall: jjb: [eventgate-wikimedia] Provide bespoke pipeline jobs [integration/config] - 10https://gerrit.wikimedia.org/r/789237 (https://phabricator.wikimedia.org/T307507) [18:23:50] (03PS1) 10Dduvall: zuul: [eventgate-wikimedia] Use bespoke pipeline jobs only [integration/config] - 10https://gerrit.wikimedia.org/r/789238 (https://phabricator.wikimedia.org/T307507) [18:24:45] !log created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789237 [18:24:46] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:28:42] (03CR) 10Dduvall: [C: 03+2] jjb: [eventgate-wikimedia] Provide bespoke pipeline jobs [integration/config] - 10https://gerrit.wikimedia.org/r/789237 (https://phabricator.wikimedia.org/T307507) (owner: 10Dduvall) [18:28:47] (03CR) 10Dduvall: [C: 03+2] zuul: [eventgate-wikimedia] Use bespoke pipeline jobs only [integration/config] - 10https://gerrit.wikimedia.org/r/789238 (https://phabricator.wikimedia.org/T307507) (owner: 10Dduvall) [18:30:34] (03Merged) 10jenkins-bot: jjb: [eventgate-wikimedia] Provide bespoke pipeline jobs [integration/config] - 10https://gerrit.wikimedia.org/r/789237 (https://phabricator.wikimedia.org/T307507) (owner: 10Dduvall) [18:30:36] (03Merged) 10jenkins-bot: zuul: [eventgate-wikimedia] Use bespoke pipeline jobs only [integration/config] - 10https://gerrit.wikimedia.org/r/789238 (https://phabricator.wikimedia.org/T307507) (owner: 10Dduvall) [18:31:26] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789238 [18:31:27] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:32:50] if anyone in here is interested in following along on work migrating the blubber project to gitlab and figuring out how we build & publish container images, we'll be having some of those discussions in #wikimedia-gitlab [18:33:23] (03PS1) 10Dduvall: zuul: [eventgate-wikimedia] Fix pipeline job names [integration/config] - 10https://gerrit.wikimedia.org/r/789241 (https://phabricator.wikimedia.org/T307507) [18:34:42] 10GitLab (Infrastructure), 10serviceops: bring new gitlab hardware servers into production - https://phabricator.wikimedia.org/T307142 (10Dzahn) @Jelto gitlab-runner2002, gitlab-runner2003 and gitlab-runner2004 are up and running. Regarding partioning they have / and /srv logical volumes unlike 2001, the VM,... [18:35:24] (03CR) 10Dduvall: [C: 03+2] zuul: [eventgate-wikimedia] Fix pipeline job names [integration/config] - 10https://gerrit.wikimedia.org/r/789241 (https://phabricator.wikimedia.org/T307507) (owner: 10Dduvall) [18:35:26] 10GitLab (Infrastructure), 10serviceops: bring new gitlab hardware servers into production - https://phabricator.wikimedia.org/T307142 (10Dzahn) I also merged https://gerrit.wikimedia.org/r/c/operations/puppet/+/788752 when I noticed we got the 14.9 version installed. [18:38:30] (03Merged) 10jenkins-bot: zuul: [eventgate-wikimedia] Fix pipeline job names [integration/config] - 10https://gerrit.wikimedia.org/r/789241 (https://phabricator.wikimedia.org/T307507) (owner: 10Dduvall) [18:43:53] 10Continuous-Integration-Config, 10Release-Engineering-Team (Doing), 10Release Pipeline, 10Patch-For-Review, 10User-dduvall: Fully deprecate service-pipeline-test and service-pipeline-test-and-publish jobs - https://phabricator.wikimedia.org/T307507 (10dduvall) [18:46:51] (03PS2) 10Daniel Kinzler: Add nifty pre-commit hook [tools/code-utils] - 10https://gerrit.wikimedia.org/r/788384 [18:47:00] (03CR) 10Daniel Kinzler: Add nifty pre-commit hook (039 comments) [tools/code-utils] - 10https://gerrit.wikimedia.org/r/788384 (owner: 10Daniel Kinzler) [18:47:03] (03PS1) 10Dduvall: jjb: [mediawiki/services/eventstreams] Provide bespoke pipeline jobs [integration/config] - 10https://gerrit.wikimedia.org/r/789244 (https://phabricator.wikimedia.org/T307507) [18:47:05] (03PS1) 10Dduvall: zuul: [mediawiki/services/eventstreams] Use bespoke pipeline jobs only [integration/config] - 10https://gerrit.wikimedia.org/r/789245 (https://phabricator.wikimedia.org/T307507) [18:47:42] 10GitLab (Infrastructure), 10serviceops, 10Patch-For-Review: bring new gitlab hardware servers into production - https://phabricator.wikimedia.org/T307142 (10Dzahn) same for gitlab-runner1002, 1003 and 1004 now: {F35103530} [18:47:54] !log creating 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789244 [18:47:55] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:48:41] 10GitLab (Infrastructure), 10serviceops, 10Patch-For-Review: bring new gitlab hardware servers into production - https://phabricator.wikimedia.org/T307142 (10Dzahn) [18:52:14] (03CR) 10Dduvall: [C: 03+2] jjb: [mediawiki/services/eventstreams] Provide bespoke pipeline jobs [integration/config] - 10https://gerrit.wikimedia.org/r/789244 (https://phabricator.wikimedia.org/T307507) (owner: 10Dduvall) [18:52:22] (03CR) 10Dduvall: [C: 03+2] zuul: [mediawiki/services/eventstreams] Use bespoke pipeline jobs only [integration/config] - 10https://gerrit.wikimedia.org/r/789245 (https://phabricator.wikimedia.org/T307507) (owner: 10Dduvall) [18:53:12] sorry for all the noise today. doing migration drudgery [18:54:11] (03Merged) 10jenkins-bot: jjb: [mediawiki/services/eventstreams] Provide bespoke pipeline jobs [integration/config] - 10https://gerrit.wikimedia.org/r/789244 (https://phabricator.wikimedia.org/T307507) (owner: 10Dduvall) [18:54:14] (03Merged) 10jenkins-bot: zuul: [mediawiki/services/eventstreams] Use bespoke pipeline jobs only [integration/config] - 10https://gerrit.wikimedia.org/r/789245 (https://phabricator.wikimedia.org/T307507) (owner: 10Dduvall) [18:54:38] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789245 [18:54:39] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:57:10] (03CR) 10Ahmon Dancy: scap backport: add options stop-before-sync, yes (033 comments) [tools/scap] - 10https://gerrit.wikimedia.org/r/789230 (owner: 10Jeena Huneidi) [19:04:37] what's the procedure for changing wikiversions just on a debug box again? [19:06:41] ^ thcipriani i feel like i bugged you about this recently but now i don't remember the conversational outcome [19:07:42] You could change it in /srv/mediawiki-staging on the deploy server then scap pull on mwdebug. [19:07:46] just `scap pull` followed by `scap update-wikiversions`, maybe? [19:07:56] ah, maybe that is all that's required [19:07:58] scap pull after [19:08:05] update wiki-versions first. [19:08:27] and undo the change after the pull [19:08:40] ack [19:12:38] (03PS1) 10Tchanders: make-release: Start branching SimilarEditors extension for production [tools/release] - 10https://gerrit.wikimedia.org/r/789248 (https://phabricator.wikimedia.org/T306909) [19:19:58] (03CR) 10Ahmon Dancy: [V: 03+1] "Tested with an updated test-scap-backport. Smooth like butter." [tools/scap] - 10https://gerrit.wikimedia.org/r/789230 (owner: 10Jeena Huneidi) [19:24:13] brennen: yeah, update wikiversions.json, scap wikiversions-compile, scap pull on debug (I think...) (sorry for lagged reply) [19:26:27] wikiversions-compile is the bit i didn't remember. [19:27:33] 10GitLab (Project Migration), 10Release-Engineering-Team (GitLab-a-thon 🦊), 10User-brennen, 10User-dduvall: Write a GitLab "Migrating a Project" runbook / manual based on Blubber migration - https://phabricator.wikimedia.org/T307538 (10brennen) 05Openβ†’03In progress p:05Triageβ†’03Medium [19:27:33] that's the bit that auto-generates the wikiversions php file [19:27:39] 10GitLab (Project Migration), 10Release-Engineering-Team (GitLab-a-thon 🦊), 10User-dduvall: Migrate Blubber project to GitLab - https://phabricator.wikimedia.org/T301168 (10brennen) [19:28:19] Can't you just `sudo -u vim` the generated file directly on the mwdebug box? and then scap pull to reset [19:28:56] sure [19:29:00] correction: sudo -u mwdeploy vim [19:29:04] right :) [19:29:50] always more than one way to skin a scap [19:31:54] * thcipriani wonders if "there's more than one way to skin a cat" is a non-US/english idiom [19:33:41] thcipriani: wiktionary says no, it's originally US https://en.wiktionary.org/wiki/there%27s_more_than_one_way_to_skin_a_cat [19:34:35] but https://en.wiktionary.org/wiki/all_roads_lead_to_Rome#English is equivalent and has all the translations [19:34:50] because.. Rome [19:35:14] all trains lead to Rome [19:36:50] (03PS1) 10Ahmon Dancy: test-scap-backport: Pass --yes to scap backport calls [tools/train-dev] - 10https://gerrit.wikimedia.org/r/789255 [19:37:07] huh, I've never thought of those as equivalent idioms. Anyway, maybe I should have just said: TMTOWTDI :D [19:38:01] (03PS2) 10Ahmon Dancy: test-scap-backport: Pass --yes to scap backport calls [tools/train-dev] - 10https://gerrit.wikimedia.org/r/789255 [19:39:17] 'there's more than one way to peel an orange' [19:39:28] really, is there though [19:39:50] There's the one way where you peel it in one spiral unit. [19:39:53] Then the rest where you don't [19:39:55] so two ways. [19:40:00] haha [19:40:11] feels correct. [19:40:19] fair enough:) I'd rather peel oranges than skin cats [19:40:55] kiwi fruit - don't try to peel it, eat it with a spoon like a boiled egg [19:40:58] save a cat, shave a yak! [19:46:46] lately I've been cutting oranges in half and then paring the skin off with a knife [19:47:22] kinda nice because you get a lot of the outer pulp off [19:47:31] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Zuul: Delete .git/logs on zuul-merger git repo to prevent reflog from being created - https://phabricator.wikimedia.org/T307620 (10hashar) [19:48:31] (03PS3) 10Ahmon Dancy: test-scap-backport: Pass --yes to scap backport calls [tools/train-dev] - 10https://gerrit.wikimedia.org/r/789255 [19:54:56] 10Release-Engineering-Team, 10Gerrit (Gerrit 3.4): Depends-On footer processing missing from Gerrit UI after 3.4.4 upgrade - https://phabricator.wikimedia.org/T307621 (10dancy) [19:55:20] (03PS2) 10Jeena Huneidi: scap backport: add options stop-before-sync, yes [tools/scap] - 10https://gerrit.wikimedia.org/r/789230 [19:55:30] (03CR) 10Jeena Huneidi: scap backport: add options stop-before-sync, yes (033 comments) [tools/scap] - 10https://gerrit.wikimedia.org/r/789230 (owner: 10Jeena Huneidi) [19:56:20] (03CR) 10Jeena Huneidi: scap backport: add options stop-before-sync, yes (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/789230 (owner: 10Jeena Huneidi) [19:57:58] (03CR) 10Ahmon Dancy: scap backport: add options stop-before-sync, yes (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/789230 (owner: 10Jeena Huneidi) [20:01:24] 10Release-Engineering-Team, 10Gerrit (Gerrit 3.4): Depends-On footer processing missing from Gerrit UI after 3.4.4 upgrade - https://phabricator.wikimedia.org/T307621 (10hashar) https://gerrit.wikimedia.org/r/admin/plugins shows the Zuul plugin is at `ea30290` which is the tip of the `master` branch at upstrea... [20:06:30] 10Release-Engineering-Team (Doing), 10Patch-For-Review, 10Release, 10Train Deployments: 1.39.0-wmf.10 deployment blockers - https://phabricator.wikimedia.org/T305216 (10brennen) [20:07:16] 10Continuous-Integration-Config, 10Release-Engineering-Team (Doing), 10Release Pipeline, 10Patch-For-Review, 10User-dduvall: Fully deprecate service-pipeline-test and service-pipeline-test-and-publish jobs - https://phabricator.wikimedia.org/T307507 (10dduvall) [20:10:03] (03PS1) 10Dduvall: jjb: [mediawiki/services/change-propagation] Provide bespoke pipeline jobs [integration/config] - 10https://gerrit.wikimedia.org/r/789258 (https://phabricator.wikimedia.org/T307507) [20:10:05] (03PS1) 10Dduvall: zuul: [mediawiki/services/change-propagation] Use bespoke pipeline jobs only [integration/config] - 10https://gerrit.wikimedia.org/r/789259 (https://phabricator.wikimedia.org/T307507) [20:11:02] !log created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789258 [20:11:03] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:11:38] (03CR) 10Dduvall: [C: 03+2] jjb: [mediawiki/services/change-propagation] Provide bespoke pipeline jobs [integration/config] - 10https://gerrit.wikimedia.org/r/789258 (https://phabricator.wikimedia.org/T307507) (owner: 10Dduvall) [20:11:48] (03CR) 10Dduvall: [C: 03+2] zuul: [mediawiki/services/change-propagation] Use bespoke pipeline jobs only [integration/config] - 10https://gerrit.wikimedia.org/r/789259 (https://phabricator.wikimedia.org/T307507) (owner: 10Dduvall) [20:13:51] (03Merged) 10jenkins-bot: jjb: [mediawiki/services/change-propagation] Provide bespoke pipeline jobs [integration/config] - 10https://gerrit.wikimedia.org/r/789258 (https://phabricator.wikimedia.org/T307507) (owner: 10Dduvall) [20:13:53] (03Merged) 10jenkins-bot: zuul: [mediawiki/services/change-propagation] Use bespoke pipeline jobs only [integration/config] - 10https://gerrit.wikimedia.org/r/789259 (https://phabricator.wikimedia.org/T307507) (owner: 10Dduvall) [20:14:35] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789259/1 [20:14:36] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:22:06] !log urbanecm@deployment-mwmaint02:~$ mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=commonswiki --logwiki=metawiki "There'sNoTime" "TheresNoTime" # T307590 [20:22:08] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:22:08] T307590: Stuck rename on Beta Cluster - https://phabricator.wikimedia.org/T307590 [20:24:58] (03PS1) 10Dduvall: jjb: [mediawiki/services/chromium-render] Provide bespoke pipeline jobs [integration/config] - 10https://gerrit.wikimedia.org/r/789264 (https://phabricator.wikimedia.org/T307507) [20:25:02] (03PS1) 10Dduvall: zuul: [mediawiki/services/chromium-render] Use bespoke pipeline jobs only [integration/config] - 10https://gerrit.wikimedia.org/r/789265 (https://phabricator.wikimedia.org/T307507) [20:25:28] !log created 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789264 [20:25:29] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:25:56] (03CR) 10Dduvall: [C: 03+2] jjb: [mediawiki/services/chromium-render] Provide bespoke pipeline jobs [integration/config] - 10https://gerrit.wikimedia.org/r/789264 (https://phabricator.wikimedia.org/T307507) (owner: 10Dduvall) [20:27:07] (03CR) 10Dduvall: [C: 03+2] zuul: [mediawiki/services/chromium-render] Use bespoke pipeline jobs only [integration/config] - 10https://gerrit.wikimedia.org/r/789265 (https://phabricator.wikimedia.org/T307507) (owner: 10Dduvall) [20:28:39] (03Merged) 10jenkins-bot: jjb: [mediawiki/services/chromium-render] Provide bespoke pipeline jobs [integration/config] - 10https://gerrit.wikimedia.org/r/789264 (https://phabricator.wikimedia.org/T307507) (owner: 10Dduvall) [20:29:11] (03Merged) 10jenkins-bot: zuul: [mediawiki/services/chromium-render] Use bespoke pipeline jobs only [integration/config] - 10https://gerrit.wikimedia.org/r/789265 (https://phabricator.wikimedia.org/T307507) (owner: 10Dduvall) [20:31:04] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789265 [20:31:04] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:32:38] (03CR) 10Hashar: Add nifty pre-commit hook (035 comments) [tools/code-utils] - 10https://gerrit.wikimedia.org/r/788384 (owner: 10Daniel Kinzler) [20:33:31] dduvall: kudos on that pipeline migration. I have seen the default jobs to be a source of confusion here and there :] [20:33:45] yeah, same. they confuse me [20:34:18] and sometimes we make changes to pipelinelib without realizing it will effect the older service-pipeline-test* jobs [20:34:28] less tech debt!! :] [20:34:34] and that makes all repos behave the same which is great [20:34:44] yes! :) [20:37:22] 10Beta-Cluster-Infrastructure, 10User-Urbanecm: Stuck rename on Beta Cluster - https://phabricator.wikimedia.org/T307590 (10Urbanecm) 05Openβ†’03Resolved a:03Urbanecm Fixed, but I had to run the script for all the wikis, rather than just the first one. Might be jobs not working at beta? DonΒ¨t have the time... [20:38:25] sleep time [20:39:43] nighty night [20:40:29] 10Continuous-Integration-Config, 10Release-Engineering-Team (Doing), 10Release Pipeline, 10Patch-For-Review, 10User-dduvall: Fully deprecate service-pipeline-test and service-pipeline-test-and-publish jobs - https://phabricator.wikimedia.org/T307507 (10dduvall) [20:42:59] (03PS1) 10Dduvall: jjb: [mediawiki/services/cxserver] Provide bespoke pipeline jobs [integration/config] - 10https://gerrit.wikimedia.org/r/789273 (https://phabricator.wikimedia.org/T307507) [20:43:01] (03PS1) 10Dduvall: zuul: [mediawiki/services/cxserver] Use bespoke pipeline jobs only [integration/config] - 10https://gerrit.wikimedia.org/r/789274 (https://phabricator.wikimedia.org/T307507) [20:44:11] !log creating 4 new jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789273 [20:44:12] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:44:20] (03CR) 10Dduvall: [C: 03+2] jjb: [mediawiki/services/cxserver] Provide bespoke pipeline jobs [integration/config] - 10https://gerrit.wikimedia.org/r/789273 (https://phabricator.wikimedia.org/T307507) (owner: 10Dduvall) [20:44:28] (03CR) 10Dduvall: [C: 03+2] zuul: [mediawiki/services/cxserver] Use bespoke pipeline jobs only [integration/config] - 10https://gerrit.wikimedia.org/r/789274 (https://phabricator.wikimedia.org/T307507) (owner: 10Dduvall) [20:47:34] (03Merged) 10jenkins-bot: jjb: [mediawiki/services/cxserver] Provide bespoke pipeline jobs [integration/config] - 10https://gerrit.wikimedia.org/r/789273 (https://phabricator.wikimedia.org/T307507) (owner: 10Dduvall) [20:47:43] (03Merged) 10jenkins-bot: zuul: [mediawiki/services/cxserver] Use bespoke pipeline jobs only [integration/config] - 10https://gerrit.wikimedia.org/r/789274 (https://phabricator.wikimedia.org/T307507) (owner: 10Dduvall) [20:48:07] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/789274 [20:48:08] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:48:52] 10Continuous-Integration-Config, 10Release-Engineering-Team (Doing), 10Release Pipeline, 10Patch-For-Review, 10User-dduvall: Fully deprecate service-pipeline-test and service-pipeline-test-and-publish jobs - https://phabricator.wikimedia.org/T307507 (10dduvall) [20:49:13] (03PS4) 10Ahmon Dancy: test-scap-backport: Pass --yes to scap backport calls [tools/train-dev] - 10https://gerrit.wikimedia.org/r/789255 [20:59:57] (03PS1) 10Dduvall: jjb: [mediawiki/services/kartotherian] Provide bespoke pipeline jobs [integration/config] - 10https://gerrit.wikimedia.org/r/789277 (https://phabricator.wikimedia.org/T307507) [20:59:59] (03PS1) 10Dduvall: zuul: [mediawiki/services/kartotherian] Use bespoke pipeline jobs only [integration/config] - 10https://gerrit.wikimedia.org/r/789278 (https://phabricator.wikimedia.org/T307507) [21:00:20] (03CR) 10Dduvall: [C: 03+2] jjb: [mediawiki/services/kartotherian] Provide bespoke pipeline jobs [integration/config] - 10https://gerrit.wikimedia.org/r/789277 (https://phabricator.wikimedia.org/T307507) (owner: 10Dduvall) [21:00:25] (03CR) 10Dduvall: [C: 03+2] zuul: [mediawiki/services/kartotherian] Use bespoke pipeline jobs only [integration/config] - 10https://gerrit.wikimedia.org/r/789278 (https://phabricator.wikimedia.org/T307507) (owner: 10Dduvall) [21:00:37] !log created 2 jobs to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789277 [21:00:38] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:02:25] (03PS1) 10Ahmon Dancy: WIP: mirror-repos.sh wait fixes [tools/train-dev] - 10https://gerrit.wikimedia.org/r/789281 [21:02:45] (03Merged) 10jenkins-bot: jjb: [mediawiki/services/kartotherian] Provide bespoke pipeline jobs [integration/config] - 10https://gerrit.wikimedia.org/r/789277 (https://phabricator.wikimedia.org/T307507) (owner: 10Dduvall) [21:02:58] (03CR) 10jerkins-bot: [V: 04-1] zuul: [mediawiki/services/kartotherian] Use bespoke pipeline jobs only [integration/config] - 10https://gerrit.wikimedia.org/r/789278 (https://phabricator.wikimedia.org/T307507) (owner: 10Dduvall) [21:04:20] (03CR) 10Dduvall: [C: 03+2] zuul: [mediawiki/services/kartotherian] Use bespoke pipeline jobs only [integration/config] - 10https://gerrit.wikimedia.org/r/789278 (https://phabricator.wikimedia.org/T307507) (owner: 10Dduvall) [21:04:50] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen), 10Cloud-VPS (Debian Stretch Deprecation), 10Patch-For-Review: Move all Wikimedia CI (WMCS integration project) instances from stretch to buster/bullseye - https://phabricator.wikimedia.org/T252071 (10hashar) I will look at getting... [21:06:27] 10Continuous-Integration-Config: Make PHP 8.1 jobs non voting? (possibly only for css-sanitizer) - https://phabricator.wikimedia.org/T307506 (10Umherirrender) [21:06:54] (03Merged) 10jenkins-bot: zuul: [mediawiki/services/kartotherian] Use bespoke pipeline jobs only [integration/config] - 10https://gerrit.wikimedia.org/r/789278 (https://phabricator.wikimedia.org/T307507) (owner: 10Dduvall) [21:07:56] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789278 [21:07:57] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:10:07] (03CR) 10jerkins-bot: [V: 04-1] test-scap-backport: Pass --yes to scap backport calls [tools/train-dev] - 10https://gerrit.wikimedia.org/r/789255 (owner: 10Ahmon Dancy) [21:11:04] (03PS1) 10Dduvall: jjb: [mediawiki/services/mobileapps] Provide bespoke pipeline jobs [integration/config] - 10https://gerrit.wikimedia.org/r/789284 (https://phabricator.wikimedia.org/T307507) [21:11:06] (03PS1) 10Dduvall: zuul: [mediawiki/services/mobileapps] Use bespoke pipeline jobs only [integration/config] - 10https://gerrit.wikimedia.org/r/789285 (https://phabricator.wikimedia.org/T307507) [21:11:20] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen), 10Cloud-VPS (Debian Stretch Deprecation), 10Patch-For-Review: Move all Wikimedia CI (WMCS integration project) instances from stretch to buster/bullseye - https://phabricator.wikimedia.org/T252071 (10Dzahn) > getting the puppet pat... [21:12:53] (03CR) 10Dduvall: [C: 03+2] jjb: [mediawiki/services/mobileapps] Provide bespoke pipeline jobs [integration/config] - 10https://gerrit.wikimedia.org/r/789284 (https://phabricator.wikimedia.org/T307507) (owner: 10Dduvall) [21:12:57] (03CR) 10Dduvall: [C: 03+2] zuul: [mediawiki/services/mobileapps] Use bespoke pipeline jobs only [integration/config] - 10https://gerrit.wikimedia.org/r/789285 (https://phabricator.wikimedia.org/T307507) (owner: 10Dduvall) [21:14:59] (03CR) 10jerkins-bot: [V: 04-1] WIP: mirror-repos.sh wait fixes [tools/train-dev] - 10https://gerrit.wikimedia.org/r/789281 (owner: 10Ahmon Dancy) [21:16:01] !log created 1 new job to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789284 [21:16:02] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:16:11] (03CR) 10jerkins-bot: [V: 04-1] jjb: [mediawiki/services/mobileapps] Provide bespoke pipeline jobs [integration/config] - 10https://gerrit.wikimedia.org/r/789284 (https://phabricator.wikimedia.org/T307507) (owner: 10Dduvall) [21:16:13] (03CR) 10jerkins-bot: [V: 04-1] zuul: [mediawiki/services/mobileapps] Use bespoke pipeline jobs only [integration/config] - 10https://gerrit.wikimedia.org/r/789285 (https://phabricator.wikimedia.org/T307507) (owner: 10Dduvall) [21:17:43] (03PS5) 10Ahmon Dancy: test-scap-backport: Pass --yes to scap backport calls [tools/train-dev] - 10https://gerrit.wikimedia.org/r/789255 [21:20:05] (03PS2) 10Ahmon Dancy: WIP: mirror-repos.sh wait fixes [tools/train-dev] - 10https://gerrit.wikimedia.org/r/789281 [21:20:39] (03PS2) 10Dduvall: jjb: [mediawiki/services/mobileapps] Provide bespoke pipeline jobs [integration/config] - 10https://gerrit.wikimedia.org/r/789284 (https://phabricator.wikimedia.org/T307507) [21:20:41] (03PS2) 10Dduvall: zuul: [mediawiki/services/mobileapps] Use bespoke pipeline jobs only [integration/config] - 10https://gerrit.wikimedia.org/r/789285 (https://phabricator.wikimedia.org/T307507) [21:21:26] (03CR) 10Dduvall: [C: 03+2] jjb: [mediawiki/services/mobileapps] Provide bespoke pipeline jobs [integration/config] - 10https://gerrit.wikimedia.org/r/789284 (https://phabricator.wikimedia.org/T307507) (owner: 10Dduvall) [21:21:40] (03CR) 10Dduvall: [C: 03+2] zuul: [mediawiki/services/mobileapps] Use bespoke pipeline jobs only [integration/config] - 10https://gerrit.wikimedia.org/r/789285 (https://phabricator.wikimedia.org/T307507) (owner: 10Dduvall) [21:23:17] PROBLEM - Work requests waiting in Zuul Gearman server on contint2001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [400.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/d/000000322/zuul-gearman?orgId=1&viewPanel=10 [21:24:02] (03Merged) 10jenkins-bot: jjb: [mediawiki/services/mobileapps] Provide bespoke pipeline jobs [integration/config] - 10https://gerrit.wikimedia.org/r/789284 (https://phabricator.wikimedia.org/T307507) (owner: 10Dduvall) [21:24:07] (03CR) 10jerkins-bot: [V: 04-1] zuul: [mediawiki/services/mobileapps] Use bespoke pipeline jobs only [integration/config] - 10https://gerrit.wikimedia.org/r/789285 (https://phabricator.wikimedia.org/T307507) (owner: 10Dduvall) [21:26:08] (03CR) 10Dduvall: [C: 03+2] zuul: [mediawiki/services/mobileapps] Use bespoke pipeline jobs only [integration/config] - 10https://gerrit.wikimedia.org/r/789285 (https://phabricator.wikimedia.org/T307507) (owner: 10Dduvall) [21:28:42] (03Merged) 10jenkins-bot: zuul: [mediawiki/services/mobileapps] Use bespoke pipeline jobs only [integration/config] - 10https://gerrit.wikimedia.org/r/789285 (https://phabricator.wikimedia.org/T307507) (owner: 10Dduvall) [21:29:06] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/789285 [21:29:07] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:30:01] RECOVERY - Work requests waiting in Zuul Gearman server on contint2001 is OK: OK: Less than 100.00% above the threshold [200.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/d/000000322/zuul-gearman?orgId=1&viewPanel=10 [21:34:34] 10Continuous-Integration-Config, 10Release-Engineering-Team (Doing), 10Release Pipeline, 10Patch-For-Review, 10User-dduvall: Fully deprecate service-pipeline-test and service-pipeline-test-and-publish jobs - https://phabricator.wikimedia.org/T307507 (10dduvall) [21:38:17] (03PS3) 10Ahmon Dancy: WIP: mirror-repos.sh wait fixes [tools/train-dev] - 10https://gerrit.wikimedia.org/r/789281 [21:40:17] 10Continuous-Integration-Config: Make PHP 8.1 jobs non voting? (possibly only for css-sanitizer) - https://phabricator.wikimedia.org/T307506 (10Umherirrender) [21:47:22] (03PS4) 10Ahmon Dancy: mirror-repos.sh: fix subprocess error handling [tools/train-dev] - 10https://gerrit.wikimedia.org/r/789281 (https://phabricator.wikimedia.org/T307495) [21:47:23] mutante: now that i'm back from leave i can help get https://gerrit.wikimedia.org/r/c/operations/puppet/+/768774 merged and verified whenever works for you [21:49:07] dduvall: alright! How about.. tomorrow right after the staff meeting? [21:51:15] sounds good to me [21:51:29] thanks! [21:52:07] dduvall: ok, I sent calendar event [21:53:21] accepted :) [21:54:18] ;) [22:14:01] 10Continuous-Integration-Config, 10Patch-For-Review: Make PHP 8.1 jobs non voting? (possibly only for css-sanitizer) - https://phabricator.wikimedia.org/T307506 (10Umherirrender) [22:19:20] 10Continuous-Integration-Config, 10Patch-For-Review: Make PHP 8.1 jobs non voting? (possibly only for css-sanitizer) - https://phabricator.wikimedia.org/T307506 (10Umherirrender) (Sorry for the spam from jenkins/gerrit and phab by rerunning jobs) [22:32:54] PROBLEM - Work requests waiting in Zuul Gearman server on contint2001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [400.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/d/000000322/zuul-gearman?orgId=1&viewPanel=10 [22:35:18] RECOVERY - Work requests waiting in Zuul Gearman server on contint2001 is OK: OK: Less than 100.00% above the threshold [200.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/d/000000322/zuul-gearman?orgId=1&viewPanel=10 [23:18:23] 10Continuous-Integration-Config, 10PHP 8.1 support, 10Patch-For-Review: Address PHP 8.1 job failures on various PHP libs - https://phabricator.wikimedia.org/T307506 (10Reedy)