[00:29:06] 10Release-Engineering-Team, 10SRE, 10serviceops-collab: Redirect revisions from svn.wikimedia.org to https://phabricator.wikimedia.org/rSVN - https://phabricator.wikimedia.org/T119846 (10Dzahn) [00:29:35] 10Release-Engineering-Team, 10SRE, 10serviceops-collab: Redirect revisions from svn.wikimedia.org to https://phabricator.wikimedia.org/rSVN - https://phabricator.wikimedia.org/T119846 (10Dzahn) I think we should either decline this OR redirect to gitlab OR to gerrit, just definitely not to Phabricator anymore. [00:30:41] 10Deployments, 10Infrastructure-Foundations, 10SRE, 10serviceops-radar: Make l10nupdate user a system user - https://phabricator.wikimedia.org/T120585 (10Dzahn) [00:31:03] 10Deployments, 10Infrastructure-Foundations, 10SRE, 10serviceops-radar: Make l10nupdate user a system user - https://phabricator.wikimedia.org/T120585 (10Dzahn) Probably this means it should be created with `systemd::sysuser` in puppet nowadays. [00:43:44] 10Diffusion, 10Release-Engineering-Team, 10SRE, 10serviceops-collab: svn.wikimedia.org redirects to Diffusion main page, hence hard to find e.g. "flexbisonparse" - https://phabricator.wikimedia.org/T140594 (10Dzahn) [00:45:31] 10Release-Engineering-Team, 10Infrastructure-Foundations, 10SRE: Enforce reference to Phabricator task for all commits to modules/admin/data/data.yaml - https://phabricator.wikimedia.org/T142827 (10Dzahn) [00:55:25] 10Release-Engineering-Team, 10SRE, 10serviceops-collab: Redirect revisions from svn.wikimedia.org to https://phabricator.wikimedia.org/rSVN - https://phabricator.wikimedia.org/T119846 (10bd808) >>! In T119846#8434042, @Dzahn wrote: > I think we should either decline this OR redirect to gitlab OR to gerrit, j... [02:59:24] (03CR) 10Ladsgroup: jjb: Make wikimedia-portals-build job rebase (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/856033 (owner: 10Ladsgroup) [09:04:18] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Priority Backlog 📥), 10serviceops-collab: Upgrade jwt-authorizer on all registry hosts - https://phabricator.wikimedia.org/T324037 (10jijiki) [09:29:46] 10Release-Engineering-Team, 10Infrastructure-Foundations, 10SRE: Enforce reference to Phabricator task for all commits to modules/admin/data/data.yaml - https://phabricator.wikimedia.org/T142827 (10MoritzMuehlenhoff) 05Open→03Declined Mid-term data.yaml will be generated via the IDM which will include pr... [09:41:16] 10Phabricator: Remove members of #Triagers not active in any Phab tasks for >18 months - https://phabricator.wikimedia.org/T324189 (10Aklapper) p:05Triage→03Low [09:44:18] 10Phabricator: Remove members of #Triagers not active in any Phab tasks for >18 months - https://phabricator.wikimedia.org/T324189 (10Aklapper) 05Open→03Resolved Removed 10 inactive members via API in https://phabricator.wikimedia.org/project/manage/13/#91134 [09:46:33] (03PS1) 10Slyngshede: Add SRE project bitu-ldap [integration/config] - 10https://gerrit.wikimedia.org/r/862836 [10:12:09] 10Phabricator: Remove members of #acl*project-admins not active in any Phab tasks for >18 months - https://phabricator.wikimedia.org/T324194 (10Aklapper) p:05Triage→03Low [10:14:11] 10Phabricator: Remove members of #acl*project-admins not active in any Phab tasks for >18 months - https://phabricator.wikimedia.org/T324194 (10Aklapper) 05Open→03Resolved Removed 3 inactive members via API in https://phabricator.wikimedia.org/project/manage/1776/#91135 [10:43:58] 10Release-Engineering-Team (Seen), 10MW-on-K8s, 10SRE, 10Traffic, and 2 others: Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536 (10Joe) For the record, we decided to start with option 3 and we're starting with rollout phase 1, specifically we'll move test2.wikipedia.org... [13:07:58] 10Phabricator, 10Release-Engineering-Team, 10translatewiki.net, 10Language-Team (Language-2022-October-December), 10Unplanned-Sprint-Work: Phabricator translations can no more be received due to decommissioning of git-ssh - https://phabricator.wikimedia.org/T321350 (10Nikerabbit) L10n-bot doesn't do a di... [14:15:55] (03PS1) 10Jforrester: Zuul: [css-sanitizer] Re-disable PHP 8.1 job, still broken [integration/config] - 10https://gerrit.wikimedia.org/r/862926 (https://phabricator.wikimedia.org/T311451) [14:17:38] (03CR) 10Jforrester: [C: 03+2] Zuul: [css-sanitizer] Re-disable PHP 8.1 job, still broken [integration/config] - 10https://gerrit.wikimedia.org/r/862926 (https://phabricator.wikimedia.org/T311451) (owner: 10Jforrester) [14:19:44] (03Merged) 10jenkins-bot: Zuul: [css-sanitizer] Re-disable PHP 8.1 job, still broken [integration/config] - 10https://gerrit.wikimedia.org/r/862926 (https://phabricator.wikimedia.org/T311451) (owner: 10Jforrester) [14:20:57] !log Zuul: [css-sanitizer] Re-disable PHP 8.1 job, still broken T311451 [14:20:59] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:20:59] T311451: Re-enable PHP 8.1 CI on css-sanitizer - https://phabricator.wikimedia.org/T311451 [14:22:12] 10Continuous-Integration-Config, 10css-sanitizer, 10PHP 8.1 support: Re-enable PHP 8.1 CI on css-sanitizer - https://phabricator.wikimedia.org/T311451 (10Jdforrester-WMF) a:05Jdforrester-WMF→03None [14:24:20] (03PS1) 10Jforrester: Zuul: [mediawiki/extensions/WatchAnalyics] Add basic CI [integration/config] - 10https://gerrit.wikimedia.org/r/862929 [14:24:31] (03CR) 10Jforrester: [C: 03+2] Zuul: [mediawiki/extensions/WatchAnalyics] Add basic CI [integration/config] - 10https://gerrit.wikimedia.org/r/862929 (owner: 10Jforrester) [14:26:26] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/WatchAnalyics] Add basic CI [integration/config] - 10https://gerrit.wikimedia.org/r/862929 (owner: 10Jforrester) [14:27:16] !log Zuul: [mediawiki/extensions/WatchAnalyics] Add basic CI [14:27:17] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:40:45] (03PS1) 10Sohom Datta: Make the Wikisource extension depend on ProofreadPage [integration/config] - 10https://gerrit.wikimedia.org/r/862936 [14:43:32] (03CR) 10Sohom Datta: "Per I4d1839fcebbf31a82625bba429526ad96804ada4" [integration/config] - 10https://gerrit.wikimedia.org/r/862936 (owner: 10Sohom Datta) [14:57:42] 10Gerrit, 10Release-Engineering-Team (GitLab III: GitLab in LA 🪃), 10SRE-OnFire, 10serviceops-collab, and 2 others: gerrit1001 running out of space on / - https://phabricator.wikimedia.org/T323262 (10hashar) Sorry for lack of update. I did dig into the Gerrit cache which are backed up by H2 Database. Some... [15:05:10] 10Gerrit, 10Sustainability (Incident Followup): Investigate Gerrit h2 cache being way too large - https://phabricator.wikimedia.org/T323754 (10hashar) [15:13:31] 10Gerrit, 10Release-Engineering-Team (GitLab III: GitLab in LA 🪃), 10SRE-OnFire, 10serviceops-collab, and 2 others: gerrit1001 running out of space on / - https://phabricator.wikimedia.org/T323262 (10hashar) @Jelto thank you for the excellent analysis about monitoring. I will look at integrating that to th... [15:14:11] (03CR) 10Jdrewniak: jjb: Make wikimedia-portals-build job rebase (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/856033 (owner: 10Ladsgroup) [15:19:27] (03CR) 10Umherirrender: "The extension is called "WatchAnaly*t*ics"" [integration/config] - 10https://gerrit.wikimedia.org/r/862929 (owner: 10Jforrester) [16:21:21] 10Release-Engineering-Team, 10SRE, 10serviceops-collab: Redirect revisions from svn.wikimedia.org to https://static-codereview.wikimedia.org - https://phabricator.wikimedia.org/T119846 (10Krinkle) [16:22:18] 10Release-Engineering-Team, 10SRE, 10serviceops-collab: Redirect revisions from svn.wikimedia.org to https://static-codereview.wikimedia.org - https://phabricator.wikimedia.org/T119846 (10Krinkle) I've boldly updated the task description to suggest targetting instead... [17:16:17] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Priority Backlog 📥), 10serviceops-collab: Upgrade jwt-authorizer on all registry hosts - https://phabricator.wikimedia.org/T324037 (10dduvall) 05Open→03Resolved a:03dduvall Done. See T322691#8433642 [17:16:20] 10Release-Engineering-Team (Priority Backlog 📥), 10serviceops, 10Release Pipeline (Blubber): Buildkit erroring with "cannot reuse body, request must be retried" upon multi-platform push - https://phabricator.wikimedia.org/T322453 (10dduvall) [17:17:32] (03PS2) 10Jforrester: Zuul: [mediawiki/extensions/Wikisource] Add ProofreadPage dependency [integration/config] - 10https://gerrit.wikimedia.org/r/862936 (owner: 10Sohom Datta) [17:18:25] (03PS2) 10Jforrester: Zuul: [operations/software/bitu-ldap] Add basic tox CI [integration/config] - 10https://gerrit.wikimedia.org/r/862836 (owner: 10Slyngshede) [17:18:32] (03CR) 10Jforrester: [C: 03+2] Zuul: [operations/software/bitu-ldap] Add basic tox CI [integration/config] - 10https://gerrit.wikimedia.org/r/862836 (owner: 10Slyngshede) [17:21:14] (03Merged) 10jenkins-bot: Zuul: [operations/software/bitu-ldap] Add basic tox CI [integration/config] - 10https://gerrit.wikimedia.org/r/862836 (owner: 10Slyngshede) [17:21:44] (03CR) 10Jforrester: [C: 03+2] Zuul: [mediawiki/extensions/Wikisource] Add ProofreadPage dependency [integration/config] - 10https://gerrit.wikimedia.org/r/862936 (owner: 10Sohom Datta) [17:22:09] !log [17:22:09] James_F: Message missing. Nothing logged. [17:22:12] !log Zuul: [operations/software/bitu-ldap] Add basic tox CI [17:22:13] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:22:14] Bah. [17:23:36] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/Wikisource] Add ProofreadPage dependency [integration/config] - 10https://gerrit.wikimedia.org/r/862936 (owner: 10Sohom Datta) [17:25:52] !log Zuul: [mediawiki/extensions/Wikisource] Add ProofreadPage dependency [17:25:53] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:08:22] (03PS1) 10Umherirrender: zuul: [mediawiki/extensions/ImageSuggestions] Add phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/863015 [18:52:29] 10GitLab (Integrations), 10Phabricator, 10Release-Engineering-Team (GitLab III: GitLab in LA 🪃), 10User-brennen: Sandbox task for gitlab-phabricator comment integration - https://phabricator.wikimedia.org/T324164 (10brennen) {"object_kind": "push", "event_name": "push", "before": "a9837c91ba90d516587bcf8f7... [19:21:44] 10GitLab (Integrations), 10Phabricator, 10Release-Engineering-Team (GitLab III: GitLab in LA 🪃), 10User-brennen: Sandbox task for gitlab-phabricator comment integration - https://phabricator.wikimedia.org/T324164 (10brennen) {"object_kind": "build", "ref": "test-webhooks", "tag": false, "before_sha": "0000... [19:21:52] 10GitLab (Integrations), 10Phabricator, 10Release-Engineering-Team (GitLab III: GitLab in LA 🪃), 10User-brennen: Sandbox task for gitlab-phabricator comment integration - https://phabricator.wikimedia.org/T324164 (10brennen) {"object_kind": "build", "ref": "test-webhooks", "tag": false, "before_sha": "0000... [19:22:00] 10GitLab (Integrations), 10Phabricator, 10Release-Engineering-Team (GitLab III: GitLab in LA 🪃), 10User-brennen: Sandbox task for gitlab-phabricator comment integration - https://phabricator.wikimedia.org/T324164 (10brennen) {"object_kind": "build", "ref": "test-webhooks", "tag": false, "before_sha": "0000... [19:22:08] 10GitLab (Integrations), 10Phabricator, 10Release-Engineering-Team (GitLab III: GitLab in LA 🪃), 10User-brennen: Sandbox task for gitlab-phabricator comment integration - https://phabricator.wikimedia.org/T324164 (10brennen) {"object_kind": "build", "ref": "test-webhooks", "tag": false, "before_sha": "0000... [19:22:16] 10GitLab (Integrations), 10Phabricator, 10Release-Engineering-Team (GitLab III: GitLab in LA 🪃), 10User-brennen: Sandbox task for gitlab-phabricator comment integration - https://phabricator.wikimedia.org/T324164 (10brennen) {"object_kind": "build", "ref": "test-webhooks", "tag": false, "before_sha": "0000... [19:22:24] 10GitLab (Integrations), 10Phabricator, 10Release-Engineering-Team (GitLab III: GitLab in LA 🪃), 10User-brennen: Sandbox task for gitlab-phabricator comment integration - https://phabricator.wikimedia.org/T324164 (10brennen) {"object_kind": "push", "event_name": "push", "before": "0000000000000000000000000... [19:22:32] 10GitLab (Integrations), 10Phabricator, 10Release-Engineering-Team (GitLab III: GitLab in LA 🪃), 10User-brennen: Sandbox task for gitlab-phabricator comment integration - https://phabricator.wikimedia.org/T324164 (10brennen) {"object_kind": "pipeline", "object_attributes": {"id": 7939, "ref": "test-webhook... [19:25:29] 10GitLab (Integrations), 10Phabricator, 10Release-Engineering-Team (GitLab III: GitLab in LA 🪃), 10User-brennen: Sandbox task for gitlab-phabricator comment integration - https://phabricator.wikimedia.org/T324164 (10brennen) {"object_kind": "push", "event_name": "push", "before": "a9837c91ba90d516587bcf8f7... [19:56:13] 10Beta-Cluster-Infrastructure: File on betacommons shows usage on production and links nonexistent beta project - https://phabricator.wikimedia.org/T301997 (10matmarex) This is the full list of projects that exist on the beta cluster: https://gerrit.wikimedia.org/g/operations/mediawiki-config/+/32d95974a38fa37c2... [20:29:06] 10GitLab (Integrations), 10Phabricator, 10Release-Engineering-Team (GitLab III: GitLab in LA 🪃), 10User-brennen: Sandbox task for gitlab-phabricator comment integration - https://phabricator.wikimedia.org/T324164 (10brennen) Commit 9bece71c5bdea61f3ceececa891cac114622581b merged by brennen: [brennen/test@... [20:42:52] gitlab.wikimedia.org just had a version upgrade [20:52:19] Great work mutante [20:55:06] thx mutante [21:13:48] 10GitLab: Gitlab message after push point to a 404 merge request URL when not logged in - https://phabricator.wikimedia.org/T324262 (10hashar) [21:14:08] 10GitLab: Gitlab message after push point to a 404 merge request URL when not logged in - https://phabricator.wikimedia.org/T324262 (10hashar) (I filed that one before I forget, I haven't looked into upstream issue tracker) [21:52:35] 10Beta-Cluster-Infrastructure: File on betacommons shows usage on production and links nonexistent beta project - https://phabricator.wikimedia.org/T301997 (10Reedy) `lang=sql MariaDB [commonswiki]> SELECT count(*) FROM globalimagelinks; +----------+ | count(*) | +----------+ | 479567 | +----------+ 1 row in s... [22:41:39] (Queue (Jenkins jobs + Zuul functions) alert) firing: Queue (Jenkins jobs + Zuul functions) alert - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert [22:51:39] (Queue (Jenkins jobs + Zuul functions) alert) firing: (2) Queue (Jenkins jobs + Zuul functions) alert - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert [23:02:28] 10Release-Engineering-Team, 10Scap: scap backport does not handle Depends-On header correctly - https://phabricator.wikimedia.org/T324275 (10Urbanecm) [23:05:07] 10Release-Engineering-Team, 10Scap: scap backport does not handle Depends-On header correctly - https://phabricator.wikimedia.org/T324275 (10Urbanecm) This is similar to T323277 I filled a few days ago, but the underlying issue seems to be different (in that case, the hash was correct, but scap backport wasn't... [23:11:39] (Queue (Jenkins jobs + Zuul functions) alert) resolved: Queue (Jenkins jobs + Zuul functions) alert - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert