[00:00:37] (DatasourceError) firing: Queue (Jenkins jobs + Zuul functions) alert - https://grafana.wikimedia.org/alerting/grafana/iS0FSjJ4z/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [00:05:52] (DatasourceError) resolved: Queue (Jenkins jobs + Zuul functions) alert - https://grafana.wikimedia.org/alerting/grafana/iS0FSjJ4z/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [08:35:12] (03CR) 10Hashar: [C: 03+2] "Last time I forgot to rebase the deployment repository and ended up NOT deploying this change. I have redone it properly this time :)" [integration/docroot] - 10https://gerrit.wikimedia.org/r/965840 (https://phabricator.wikimedia.org/T327604) (owner: 10Gergő Tisza) [08:37:55] (03PS1) 10Hslater: Add OOJSPlus as depdendency to BlueSpiceBookshelf [integration/config] - 10https://gerrit.wikimedia.org/r/973719 [08:39:01] (03PS2) 10Hslater: Add OOJSPlus as depdendency to BlueSpiceBookshelf [integration/config] - 10https://gerrit.wikimedia.org/r/973719 [09:15:15] (03CR) 10Hashar: [C: 03+2] Replace WikimediaUI Base with Codex design tokens (032 comments) [integration/docroot] - 10https://gerrit.wikimedia.org/r/969496 (https://phabricator.wikimedia.org/T331403) (owner: 10VolkerE) [09:16:00] (03Merged) 10jenkins-bot: Replace WikimediaUI Base with Codex design tokens [integration/docroot] - 10https://gerrit.wikimedia.org/r/969496 (https://phabricator.wikimedia.org/T331403) (owner: 10VolkerE) [09:25:31] (03CR) 10Hashar: [C: 03+2] Add OOJSPlus as depdendency to BlueSpiceBookshelf [integration/config] - 10https://gerrit.wikimedia.org/r/973719 (owner: 10Hslater) [09:27:17] (03Merged) 10jenkins-bot: Add OOJSPlus as depdendency to BlueSpiceBookshelf [integration/config] - 10https://gerrit.wikimedia.org/r/973719 (owner: 10Hslater) [09:27:31] (03CR) 10Hashar: [C: 03+2] "Deployed" [integration/config] - 10https://gerrit.wikimedia.org/r/973719 (owner: 10Hslater) [09:33:55] 10Release-Engineering-Team (Quid Pro Crow 🦃), 10Release, 10Train Deployments: 1.42.0-wmf.4 deployment blockers - https://phabricator.wikimedia.org/T350080 (10jnuche) Confirmed it's possible to log into https://horizon.wikimedia.org after deploying https://gerrit.wikimedia.org/r/973247 and setting `labswiki`... [09:40:13] 10Release-Engineering-Team (Quid Pro Crow 🦃), 10Patch-For-Review, 10Release, 10Train Deployments: 1.42.0-wmf.4 deployment blockers - https://phabricator.wikimedia.org/T350080 (10jnuche) [09:40:22] 10Release-Engineering-Team (Quid Pro Crow 🦃), 10Patch-For-Review, 10Release, 10Train Deployments: 1.42.0-wmf.4 deployment blockers - https://phabricator.wikimedia.org/T350080 (10jnuche) 05Open→03Resolved [09:41:38] 10Gerrit: Unable to add/save SSH keys - https://phabricator.wikimedia.org/T350753 (10hashar) 05Stalled→03Resolved a:03hashar Head to https://gerrit.wikimedia.org/r/settings/#SSHKeys * Paste the public part of the key in the field {nav New SSH Key} * Click {nav ADD NEW SSH KEY} That should add the key to... [09:50:16] 10GitLab (Pipeline Services Migration🐤), 10Wikimedia-Design, 10collaboration-services: move design.wikimedia.org to kubernetes - https://phabricator.wikimedia.org/T350791 (10Volker_E) [09:55:12] 10Release-Engineering-Team (Quid Pro Crow 🦃), 10Release, 10Train Deployments: 1.42.0-wmf.5 deployment blockers - https://phabricator.wikimedia.org/T350081 (10DAlangi_WMF) ##### Risky Patch! 🚂🔥 * **Change**: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CentralAuth/+/967880 * **Summary**: Last week... [10:38:07] 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥): Uninstall Harbormaster (Phabricator application) - https://phabricator.wikimedia.org/T348115 (10Aklapper) [10:38:37] 10Release-Engineering-Team (Seen), 10MW-on-K8s, 10SRE, 10Traffic, and 2 others: Migrate mobileapps to k8s - https://phabricator.wikimedia.org/T350846 (10Joe) p:05Triage→03High [10:46:27] (03CR) 10Hashar: [C: 04-1] Remove acme-chief project (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/972910 (https://phabricator.wikimedia.org/T347623) (owner: 10BCornwall) [10:48:44] (03PS2) 10Hashar: Archive operations/software/purged [integration/config] - 10https://gerrit.wikimedia.org/r/973269 (https://phabricator.wikimedia.org/T347623) (owner: 10BCornwall) [10:54:46] (03CR) 10Hashar: [C: 03+2] Archive operations/software/purged [integration/config] - 10https://gerrit.wikimedia.org/r/973269 (https://phabricator.wikimedia.org/T347623) (owner: 10BCornwall) [10:56:31] (03Merged) 10jenkins-bot: Archive operations/software/purged [integration/config] - 10https://gerrit.wikimedia.org/r/973269 (https://phabricator.wikimedia.org/T347623) (owner: 10BCornwall) [10:58:49] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Priority Backlog 📥), 10collaboration-services: Migrate to using new GitLab CI runner authentication scheme - https://phabricator.wikimedia.org/T344951 (10Jelto) [10:59:04] 10GitLab (Infrastructure), 10Data-Persistence-Backup, 10collaboration-services, 10Patch-For-Review, 10User-brennen: Backups for GitLab - https://phabricator.wikimedia.org/T274463 (10CodeReviewBot) jynus opened https://gitlab.wikimedia.org/repos/sre/wmfbackups/-/merge_requests/3 WIP GitLab backups [11:03:40] 10GitLab (Project Migration), 10Traffic, 10Patch-For-Review: Migrate Traffic repositories from Gerrit to Gitlab - https://phabricator.wikimedia.org/T347623 (10hashar) [11:07:43] (03PS2) 10Hashar: Archive operations/software/knead-wikidough [integration/config] - 10https://gerrit.wikimedia.org/r/973220 (https://phabricator.wikimedia.org/T347623) (owner: 10BCornwall) [11:08:16] (03CR) 10Hashar: [C: 03+2] "I have amended the change to mark it as archived in CI rather than doing a plain removal :)" [integration/config] - 10https://gerrit.wikimedia.org/r/973220 (https://phabricator.wikimedia.org/T347623) (owner: 10BCornwall) [11:09:58] (03Merged) 10jenkins-bot: Archive operations/software/knead-wikidough [integration/config] - 10https://gerrit.wikimedia.org/r/973220 (https://phabricator.wikimedia.org/T347623) (owner: 10BCornwall) [11:16:17] 10GitLab (Project Migration), 10Traffic, 10Patch-For-Review: Migrate Traffic repositories from Gerrit to Gitlab - https://phabricator.wikimedia.org/T347623 (10hashar) [12:07:58] 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review: Uninstall Harbormaster (Phabricator application) - https://phabricator.wikimedia.org/T348115 (10CodeReviewBot) aklapper opened https://gitlab.wikimedia.org/repos/phabricator/deployment/-/merge_requests/27 Remove unused "Harb... [12:18:37] 10Release-Engineering-Team (Seen), 10MW-on-K8s, 10SRE, 10Traffic, and 2 others: Migrate mobileapps to k8s - https://phabricator.wikimedia.org/T350846 (10Joe) I decided we should move about 10% of the mobileapps traffic at a time; that means about 300 rps, which I think we should be able to serve moving ove... [13:12:40] 10Phabricator (Upstream), 10Upstream: Central task information is put in periphery by Maniphest - https://phabricator.wikimedia.org/T134398 (10Aklapper) 05Open→03Declined Declining to reflect reality, as there are no plans to change the current layout in upstream as far as I know. [13:14:59] 10Phabricator (Upstream), 10Upstream: Add custom policy to access denied message - https://phabricator.wikimedia.org/T231734 (10Aklapper) 05Open→03Resolved In my understanding this seems resolved; link nowadays says: ` Access Denied: Restricted Task You do not have permission to view this object. Users... [13:16:57] 10Phabricator (Upstream), 10Upstream: Typeahead shows archived projects as if they are active - https://phabricator.wikimedia.org/T86219 (10Aklapper) 05Open→03Resolved Looks resolved nowadays: {F41502332} [13:56:14] 10Phabricator, 10Wikimedia-Phabricator-Extensions: Remove unused custom /src/elasticsearch/ code - https://phabricator.wikimedia.org/T351085 (10Aklapper) p:05Triage→03Low [14:19:48] 10Phabricator (Upstream), 10Upstream: Project tags can not be found via the fulltext search index - https://phabricator.wikimedia.org/T92630 (10Aklapper) No idea if still wanted or needed, however pasting the related draft patch in Differential here before we may switch that off: ` diff --git a/src/application... [14:25:57] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Priority Backlog 📥), 10collaboration-services: Migrate to using new GitLab CI runner authentication scheme - https://phabricator.wikimedia.org/T344951 (10Jelto) The converted Shared Runner (https://gitlab.wikimedia.org/admin/runners/1479#/) and Trusted... [14:51:26] 10Phabricator, 10Wikimedia-Phabricator-Extensions: The list of linked Gerrit patches in Phabricator tasks should not have clickable repo/branch links - https://phabricator.wikimedia.org/T349138 (10Aklapper) p:05Triage→03Low > the repo link links to the list of all patches in that repo Huh? It does not her... [14:52:33] (03PS11) 10Hashar: Plugin to process Puppet Catalog Compiler results [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/969981 [15:09:37] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Priority Backlog 📥), 10collaboration-services, 10Patch-For-Review: Migrate to using new GitLab CI runner authentication scheme - https://phabricator.wikimedia.org/T344951 (10CodeReviewBot) jelto opened https://gitlab.wikimedia.org/repos/releng/gitlab... [15:49:36] 10Diffusion, 10Phabricator, 10Security: Diffusion 'view raw file' policy and behavior is unclear and inconsistent (PHP files displayed in browser; JS files offer downloading file) - https://phabricator.wikimedia.org/T149152 (10Aklapper) T117621 mentioned in the already existing code that D428 was written aga... [15:58:56] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Patch-For-Review: integration/config Remove Java 8 or migrate from Buster to Bullseye - https://phabricator.wikimedia.org/T350587 (10hashar) [15:59:21] (03CR) 10Hashar: [C: 03+2] zuul: document Refinery is sticking to Java 8 [integration/config] - 10https://gerrit.wikimedia.org/r/973089 (https://phabricator.wikimedia.org/T350587) (owner: 10Hashar) [16:00:43] (03Merged) 10jenkins-bot: zuul: document Refinery is sticking to Java 8 [integration/config] - 10https://gerrit.wikimedia.org/r/973089 (https://phabricator.wikimedia.org/T350587) (owner: 10Hashar) [16:00:44] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Patch-For-Review: Remove Java 8 images from integration/config - https://phabricator.wikimedia.org/T350587 (10hashar) [16:06:07] 10Release-Engineering-Team (Priority Backlog 📥): Train blockers task throws exceptions on strange version numbers - https://phabricator.wikimedia.org/T304540 (10Aklapper) @dduvall: Would you be willing and have time to transfer your patch in D1200 into a MR in https://gitlab.wikimedia.org/repos/phabricator/exten... [16:10:19] 10Phabricator, 10Release-Engineering-Team (Quid Pro Crow 🦃): Triage and close remaining open patches in Differential - https://phabricator.wikimedia.org/T351105 (10Aklapper) p:05Triage→03Medium [16:11:22] 10GitLab (Pipeline Services Migration🐤), 10Wikimedia-Design, 10collaboration-services, 10Design: move design.wikimedia.org to kubernetes - https://phabricator.wikimedia.org/T350791 (10LSobanski) p:05Triage→03Medium [16:12:53] 10Phabricator, 10Release-Engineering-Team (Quid Pro Crow 🦃): Triage and close remaining open patches in Differential - https://phabricator.wikimedia.org/T351105 (10Aklapper) a:03Aklapper As I only retroactively decided to make this a separate subtask: We are down to one remaining task, to transfer to GitLab... [16:14:03] 10GitLab (Pipeline Services Migration🐤), 10collaboration-services: move os-reports.wikimedia.org to kubernetes - https://phabricator.wikimedia.org/T350794 (10LSobanski) p:05Triage→03Medium [16:14:08] 10GitLab (Pipeline Services Migration🐤), 10collaboration-services: move commons-query.wikimedia.org and query.wikidata.org to kubernetes - https://phabricator.wikimedia.org/T350793 (10LSobanski) p:05Triage→03Medium [16:14:14] 10GitLab (Pipeline Services Migration🐤), 10collaboration-services: move security.wikimedia.org to kubernetes - https://phabricator.wikimedia.org/T350796 (10LSobanski) p:05Triage→03Medium [16:14:27] 10Phabricator, 10Release-Engineering-Team (Quid Pro Crow 🦃), 10Patch-For-Review: Uninstall Harbormaster (Phabricator application) - https://phabricator.wikimedia.org/T348115 (10Aklapper) [16:15:39] 10Phabricator (phabricator-next), 10Release-Engineering-Team (Quid Pro Crow 🦃), 10Patch-For-Review, 10User-brennen: Deploy Phabricator/Phorge 2023-11-14 - https://phabricator.wikimedia.org/T350876 (10Aklapper) [16:17:51] 10Phabricator, 10Release-Engineering-Team (Quid Pro Crow 🦃): Disable inactive "jenkins" bot account in Phabricator - https://phabricator.wikimedia.org/T350850 (10Aklapper) 05Open→03Resolved Account disabled [17:00:25] 10Gerrit: Investigate changes having a wrong server id - https://phabricator.wikimedia.org/T323259 (10hashar) Tyler found `1003873@173816e5-2b9a-37c3-8a2e-48639d4f1153` in /srv/git/operations/software/gerrit/plugins/quota.git . That is the instance id from Google Gerrit instance. [17:13:12] (03CR) 10BCornwall: Remove acme-chief project (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/972910 (https://phabricator.wikimedia.org/T347623) (owner: 10BCornwall) [17:13:30] (03PS2) 10BCornwall: Archive acme-chief/certcentral projects [integration/config] - 10https://gerrit.wikimedia.org/r/972910 (https://phabricator.wikimedia.org/T347623) [17:17:25] (03CR) 10Lucas Werkmeister (WMDE): CoreComposerTest: Remove ancient PHP file extensions (031 comment) [integration/quibble] - 10https://gerrit.wikimedia.org/r/883885 (owner: 10Lucas Werkmeister (WMDE)) [17:17:40] 10GitLab (Project Migration), 10Traffic, 10Patch-For-Review: Migrate Traffic repositories from Gerrit to Gitlab - https://phabricator.wikimedia.org/T347623 (10BCornwall) >>! In T347623#9322202, @LSobanski wrote: > @BCornwall For operations/software/varnish, looks like it should just be archived and not migra... [17:18:32] (03PS3) 10Jforrester: Docker, jjb, Zuul: Archive acme-chief and certcentral projects [integration/config] - 10https://gerrit.wikimedia.org/r/972910 (https://phabricator.wikimedia.org/T347623) (owner: 10BCornwall) [17:18:39] (03CR) 10Jforrester: [C: 03+2] Docker, jjb, Zuul: Archive acme-chief and certcentral projects [integration/config] - 10https://gerrit.wikimedia.org/r/972910 (https://phabricator.wikimedia.org/T347623) (owner: 10BCornwall) [17:20:44] (03Merged) 10jenkins-bot: Docker, jjb, Zuul: Archive acme-chief and certcentral projects [integration/config] - 10https://gerrit.wikimedia.org/r/972910 (https://phabricator.wikimedia.org/T347623) (owner: 10BCornwall) [17:24:04] !log Zuul: Archive acme-chief and certcentral projects for T347623 [17:24:07] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:24:07] T347623: Migrate Traffic repositories from Gerrit to Gitlab - https://phabricator.wikimedia.org/T347623 [17:24:59] !log jjb: Manually deleted acme-chief-tox-docker for T347623 [17:25:02] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:25:52] (03CR) 10Jforrester: jjb: use Java 17 for extensions codehealth jobs (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/971920 (https://phabricator.wikimedia.org/T349983) (owner: 10Hashar) [17:31:15] 10Gerrit, 10Release-Engineering-Team: Add MoTD to gerrit - https://phabricator.wikimedia.org/T351109 (10thcipriani) [17:34:29] (03CR) 10Pwangai: jjb: use Java 17 for extensions codehealth jobs (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/971920 (https://phabricator.wikimedia.org/T349983) (owner: 10Hashar) [17:42:54] 10Release-Engineering-Team (Seen), 10MW-on-K8s, 10SRE, 10Traffic, 10serviceops: Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536 (10Krinkle) [18:00:56] 10Release-Engineering-Team (Priority Backlog 📥): Decrease number of open Phab tickets with assignee field set for more than two years (aka cookie licking) (Q3/2023 edition) - https://phabricator.wikimedia.org/T332676 (10Aklapper) [18:02:36] brennen: If you find 3min, would also love to merge/deploy the trivial Phab one-liners in https://gitlab.wikimedia.org/repos/phabricator/extensions/-/merge_requests/20/ and https://gitlab.wikimedia.org/repos/phabricator/deployment/-/merge_requests/27/ - TIA [18:21:18] (03CR) 10Jforrester: [C: 03+2] jjb: use Java 17 for extensions codehealth jobs [integration/config] - 10https://gerrit.wikimedia.org/r/971920 (https://phabricator.wikimedia.org/T349983) (owner: 10Hashar) [18:23:03] (03Merged) 10jenkins-bot: jjb: use Java 17 for extensions codehealth jobs [integration/config] - 10https://gerrit.wikimedia.org/r/971920 (https://phabricator.wikimedia.org/T349983) (owner: 10Hashar) [18:25:20] 10Continuous-Integration-Config, 10SonarQube Bot, 10Patch-For-Review, 10Quality-and-Test-Engineering-Team (SonarCloud Admin): Upgrade SonarQube / codehealth jobs to use Java 17 - https://phabricator.wikimedia.org/T349983 (10Jdforrester-WMF) 05Open→03Resolved a:03Jdforrester-WMF >>! In T349983#9310173... [19:06:58] 10Phabricator, 10Release-Engineering-Team (Radar), 10DBA, 10Data-Persistence-Backup, and 2 others: phabricator->phorge migration - database handling - https://phabricator.wikimedia.org/T335080 (10Dzahn) Should the VM phab-test1001 be deleted at this point or does it still have value? (T351115) [19:25:17] 10Continuous-Integration-Config, 10SonarQube Bot, 10Quality-and-Test-Engineering-Team (SonarCloud Admin): Upgrade SonarQube / codehealth jobs to use Java 17 - https://phabricator.wikimedia.org/T349983 (10Jdforrester-WMF) a:05Jdforrester-WMF→03hashar [23:00:39] 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥), 10Wikimedia-Phabricator-Extensions, 10Patch-For-Review, 10User-DannyS712: In train deployment tasks, link to create a subtask for a blocker should prefill unbreak now - https://phabricator.wikimedia.org/T285733 (10CodeReviewBot) brennen merg... [23:00:58] 10Phabricator, 10Release-Engineering-Team (Quid Pro Crow 🦃), 10Patch-For-Review: Uninstall Harbormaster (Phabricator application) - https://phabricator.wikimedia.org/T348115 (10CodeReviewBot) brennen merged https://gitlab.wikimedia.org/repos/phabricator/deployment/-/merge_requests/27 Remove unused "Harborma... [23:43:13] 10Scap: scap update-interwiki-cache is broken - https://phabricator.wikimedia.org/T347982 (10Krinkle) @Ladsgroup How hard would it be to run the script to create the updated interwiki map locally instead of in prod? If feasible, it could be automated as part of the creationg one of the wmf-config commits that yo... [23:43:20] 10Scap, 10MediaWiki-Platform-Team (Radar): scap update-interwiki-cache is broken - https://phabricator.wikimedia.org/T347982 (10Krinkle)