[00:00:38] 10Continuous-Integration-Config, 10Codex, 10Design-Systems-Team, 10Test-Coverage, 10User-DannyS712: Public codex code coverage report - https://phabricator.wikimedia.org/T303899 (10STHart) p:05Triage→03Low [01:22:24] bd808: Yeah. After the commit was merged I realized that a better commit message summary would be "add ipv6 support" [01:22:44] It is very cool code and it was fun playing with it. [06:37:51] 10Release-Engineering-Team, 10Scap, 10serviceops, 10User-brennen: Deploy Scap version 4.7.0 - https://phabricator.wikimedia.org/T306827 (10JMeybohm) a:03JMeybohm [06:55:49] 10Release-Engineering-Team, 10Scap, 10serviceops, 10User-brennen: Deploy Scap version 4.7.0 - https://phabricator.wikimedia.org/T306827 (10JMeybohm) Rolled out to canaries + deploy1002, still super slow as introduced with T305949 [07:05:29] (03PS2) 10Elukey: Remove the --global option from git lfs calls [tools/scap] - 10https://gerrit.wikimedia.org/r/785154 (https://phabricator.wikimedia.org/T303801) [07:06:06] (03CR) 10Elukey: Remove the --global option from git lfs calls (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/785154 (https://phabricator.wikimedia.org/T303801) (owner: 10Elukey) [07:13:53] 10Release-Engineering-Team (Radar), 10MW-on-K8s, 10serviceops, 10Kubernetes, 10Patch-For-Review: Kubernetes credentials on deployment servers should be available to deployers, not all users - https://phabricator.wikimedia.org/T305729 (10JMeybohm) >>! In T305729#7856071, @dancy wrote: > * `WARNING: Kubern... [07:23:46] 10Beta-Cluster-Infrastructure, 10ORES, 10Machine-Learning-Team (Active Tasks): Upgrade deployment-ores01 host to Buster - https://phabricator.wikimedia.org/T306053 (10elukey) [07:26:13] 10Beta-Cluster-Infrastructure, 10ORES, 10Machine-Learning-Team (Active Tasks): Upgrade deployment-ores01 host to Buster - https://phabricator.wikimedia.org/T306053 (10elukey) We created deployment-ores02 as part of T303801 with Debian Buster, once we'll finish the task we should be able to delete deployment-... [07:43:44] 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Debian Stretch Deprecation): Cloud VPS "deployment-prep" project Stretch deprecation - https://phabricator.wikimedia.org/T306068 (10hashar) [07:56:04] (03Abandoned) 10Hashar: Add jjb entry for image-suggestions service [integration/config] - 10https://gerrit.wikimedia.org/r/779451 (https://phabricator.wikimedia.org/T303408) (owner: 10Hnowlan) [08:07:22] (03CR) 10Hashar: [C: 03+2] "INFO:jenkins_jobs.builder:Number of jobs generated: 4" [integration/config] - 10https://gerrit.wikimedia.org/r/785204 (https://phabricator.wikimedia.org/T306668) (owner: 10BryanDavis) [08:09:18] (03Merged) 10jenkins-bot: striker: Switch tests to PipelineLib [integration/config] - 10https://gerrit.wikimedia.org/r/785204 (https://phabricator.wikimedia.org/T306668) (owner: 10BryanDavis) [08:09:51] !log Reloading Zuul for https://gerrit.wikimedia.org/r/785204 [08:09:52] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [08:10:44] (03CR) 10Hashar: "I have done a recheck on https://gerrit.wikimedia.org/r/c/labs/striker/+/785203" [integration/config] - 10https://gerrit.wikimedia.org/r/785204 (https://phabricator.wikimedia.org/T306668) (owner: 10BryanDavis) [08:15:18] (03CR) 10Hashar: [C: 03+2] layout: [mediawiki/extensions/CampaignEvents] Add apitests [integration/config] - 10https://gerrit.wikimedia.org/r/780824 (owner: 10Daimona Eaytoy) [08:17:37] (03Merged) 10jenkins-bot: layout: [mediawiki/extensions/CampaignEvents] Add apitests [integration/config] - 10https://gerrit.wikimedia.org/r/780824 (owner: 10Daimona Eaytoy) [08:23:04] Hi [08:30:13] !log Reloading Zuul for https://gerrit.wikimedia.org/r/780824 [08:30:14] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:22:32] PROBLEM - Check systemd state on doc1001 is CRITICAL: CRITICAL - degraded: The following units failed: rsync-doc-doc1002.eqiad.wmnet.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [10:56:31] (03PS1) 10QChris: Allow “Gerrit Managers” to import history [extensions/PhotoSwipe] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/786293 [10:56:33] (03CR) 10QChris: [V: 03+2 C: 03+2] Allow “Gerrit Managers” to import history [extensions/PhotoSwipe] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/786293 (owner: 10QChris) [10:56:37] (03PS1) 10QChris: Import done. Revoke import grants [extensions/PhotoSwipe] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/786294 [10:56:39] (03CR) 10QChris: [V: 03+2 C: 03+2] Import done. Revoke import grants [extensions/PhotoSwipe] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/786294 (owner: 10QChris) [11:18:55] RECOVERY - Check systemd state on doc1001 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [11:37:25] 10Release-Engineering-Team (🌱 Spring Cleaning — April 2022), 10Release, 10Train Deployments: 1.39.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T305215 (10AlexisJazz) T306882 might be something. [11:44:42] 10Release-Engineering-Team (🌱 Spring Cleaning — April 2022), 10Release, 10Train Deployments: 1.39.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T305215 (10Urbanecm) [11:48:19] 10Release-Engineering-Team (🌱 Spring Cleaning — April 2022), 10Release, 10Train Deployments: 1.39.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T305215 (10Urbanecm) [11:48:38] 10Release-Engineering-Team (🌱 Spring Cleaning — April 2022), 10Release, 10Train Deployments: 1.39.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T305215 (10Urbanecm) [11:49:30] 10Phabricator, 10User-AKlapper: Establish a workflow that scales for requesting Phab 2FA resets - https://phabricator.wikimedia.org/T306708 (10Aklapper) [11:53:33] 10GitLab (Integrations), 10GitLab-Test, 10Phabricator, 10Release-Engineering-Team (Next), and 2 others: Experiment with GitLab-Phabricator integration - https://phabricator.wikimedia.org/T265617 (10kostajh) >>! In T265617#7413071, @mmodell wrote: > So one thing I think that we need is the equivalent to the... [12:07:17] 10Release-Engineering-Team (🌱 Spring Cleaning — April 2022), 10Release, 10Train Deployments: 1.39.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T305215 (10Urbanecm) [12:30:24] (03CR) 10Klausman: [C: 03+1] Remove the --global option from git lfs calls [tools/scap] - 10https://gerrit.wikimedia.org/r/785154 (https://phabricator.wikimedia.org/T303801) (owner: 10Elukey) [12:33:22] !log Manually pruned dangling docker images on contint1001 and contint2001 [12:33:23] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [12:55:32] 10Beta-Cluster-Infrastructure, 10ORES, 10Machine-Learning-Team (Active Tasks): Upgrade deployment-ores01 host to Buster - https://phabricator.wikimedia.org/T306053 (10Jdforrester-WMF) >>! In T306053#7879658, @elukey wrote: > We created deployment-ores02 as part of T303801 with Debian Buster, once we'll finis... [13:20:27] (03PS1) 10Tchanders: layout: Install CI for mediawiki/extensions/SimilarEditors [integration/config] - 10https://gerrit.wikimedia.org/r/786308 (https://phabricator.wikimedia.org/T306897) [13:29:24] (03PS2) 10Jforrester: Zuul: [mediawiki/extensions/SimilarEditors] Install basic prod CI [integration/config] - 10https://gerrit.wikimedia.org/r/786308 (https://phabricator.wikimedia.org/T306897) (owner: 10Tchanders) [13:30:53] (03PS3) 10Jforrester: Zuul: [mediawiki/extensions/SimilarEditors] Install basic prod CI [integration/config] - 10https://gerrit.wikimedia.org/r/786308 (https://phabricator.wikimedia.org/T306897) (owner: 10Tchanders) [13:31:19] (03CR) 10Jforrester: [C: 03+2] Zuul: [mediawiki/extensions/SimilarEditors] Install basic prod CI [integration/config] - 10https://gerrit.wikimedia.org/r/786308 (https://phabricator.wikimedia.org/T306897) (owner: 10Tchanders) [13:34:28] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/SimilarEditors] Install basic prod CI [integration/config] - 10https://gerrit.wikimedia.org/r/786308 (https://phabricator.wikimedia.org/T306897) (owner: 10Tchanders) [13:37:48] !log Zuul: [mediawiki/extensions/SimilarEditors] Install basic prod CI for T306897 [13:38:18] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:38:19] T306897: Add CI for SimilarEditors - https://phabricator.wikimedia.org/T306897 [13:38:55] 10Continuous-Integration-Infrastructure, 10MinervaNeue, 10Vector, 10Accessibility, and 3 others: Add automated accessibility tests in CI to generate accessibility benchmarks for Skins - https://phabricator.wikimedia.org/T301184 (10zeljkofilipin) a:05zeljkofilipin→03None [13:46:25] !log Deleting deployment-mx02.deployment-prep.eqiad1.wikimedia.cloud for T306068 [13:46:27] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:46:27] T306068: Cloud VPS "deployment-prep" project Stretch deprecation - https://phabricator.wikimedia.org/T306068 [13:47:04] 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Debian Stretch Deprecation): Cloud VPS "deployment-prep" project Stretch deprecation - https://phabricator.wikimedia.org/T306068 (10Jdforrester-WMF) [13:49:38] 10Beta-Cluster-Infrastructure: Beta cluster has reached its quota - https://phabricator.wikimedia.org/T257118 (10Jdforrester-WMF) 05Open→03Resolved >>! In T257118#6947852, @Peachey88 wrote: > The project is now down to 67 / 100 instances, Should the task now be closed off. Yes. [13:49:42] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Radar), 10Code-Stewardship-Reviews: deployment-prep: Code stewardship request - https://phabricator.wikimedia.org/T215217 (10Jdforrester-WMF) [14:47:34] hashar: thanks for the merge on my striker -> pipelinelib stuff. I'm also sort of amazed I got all the changes correct to make it work the first time. :) [14:50:15] 10Release-Engineering-Team (Priority Backlog 📥), 10Scap, 10Infrastructure-Foundations, 10serviceops, 10Patch-For-Review: Use scap to deploy itself to scap targets - https://phabricator.wikimedia.org/T303559 (10jnuche) [15:32:34] (03CR) 10Ahmon Dancy: [C: 03+2] Remove the --global option from git lfs calls [tools/scap] - 10https://gerrit.wikimedia.org/r/785154 (https://phabricator.wikimedia.org/T303801) (owner: 10Elukey) [15:34:42] 10Release-Engineering-Team (Priority Backlog 📥), 10Scap, 10Infrastructure-Foundations, 10serviceops, 10Patch-For-Review: Use scap to deploy itself to scap targets - https://phabricator.wikimedia.org/T303559 (10jnuche) [15:36:45] (03Merged) 10jenkins-bot: Remove the --global option from git lfs calls [tools/scap] - 10https://gerrit.wikimedia.org/r/785154 (https://phabricator.wikimedia.org/T303801) (owner: 10Elukey) [15:40:00] !log train 1.39.0-wmf.9 (T305215): no current blockers - expect to start train ops after the toolhub deployment window wraps, so some time after 17:00 UTC; taking a pre-train stroll-around-the-block break before that. [15:40:02] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:40:02] T305215: 1.39.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T305215 [15:49:40] (Queue (Jenkins jobs + Zuul functions) alert) firing: Queue (Jenkins jobs + Zuul functions) alert - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert [15:59:40] (Queue (Jenkins jobs + Zuul functions) alert) firing: (2) Queue (Jenkins jobs + Zuul functions) alert - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert [16:05:33] 10Release-Engineering-Team, 10Scap: scap deploy slower than before - https://phabricator.wikimedia.org/T306915 (10dancy) [16:06:48] 10Release-Engineering-Team, 10Scap, 10serviceops, 10User-brennen: Deploy Scap version 4.7.0 - https://phabricator.wikimedia.org/T306827 (10dancy) >>! In T306827#7879622, @JMeybohm wrote: > Rolled out to canaries + deploy1002, still super slow as introduced with T305949 I created T306915 for further conver... [16:14:40] (Queue (Jenkins jobs + Zuul functions) alert) resolved: Queue (Jenkins jobs + Zuul functions) alert - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert [16:23:36] 10Release-Engineering-Team, 10Scap: scap deploy slower than before - https://phabricator.wikimedia.org/T306915 (10dancy) [16:28:04] 10Release-Engineering-Team, 10Scap: scap deploy slower than before - https://phabricator.wikimedia.org/T306915 (10dancy) [16:33:43] 10Release-Engineering-Team, 10Scap: scap deploy slower than before - https://phabricator.wikimedia.org/T306915 (10dancy) [16:34:21] 10Release-Engineering-Team, 10Scap: scap deploy slower than before - https://phabricator.wikimedia.org/T306915 (10dancy) 05Open→03Invalid Based on the information from the logs and the fact that scap hasn't been modified in a way that would affect the operations that happened, I'm declaring this to be not... [16:59:28] 10Release-Engineering-Team (🌱 Spring Cleaning — April 2022), 10Patch-For-Review, 10Release, 10Train Deployments: 1.39.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T305215 (10brennen) (Above double-sync due to having forgotten a `SCAP=scap` while using a local checkout of scap.) [17:19:58] 10Release-Engineering-Team (🌱 Spring Cleaning — April 2022), 10Patch-For-Review, 10Release, 10Train Deployments: 1.39.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T305215 (10brennen) Had some hosts time out during `deploy-promote --yes testwikis 1.39.0-wmf.9`: {P26581} Per @Dzahn, thes... [17:23:29] 10Release-Engineering-Team (🌱 Spring Cleaning — April 2022), 10Patch-For-Review, 10Release, 10Train Deployments: 1.39.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T305215 (10Dzahn) Yea, I picked mw1362 and wtp1046 an mw2309.. they are pooled, are in conftool-data, could "scap pull" direc... [18:09:31] bd808: sometime it feels like magic [18:47:21] 10Release-Engineering-Team (🌱 Spring Cleaning — April 2022), 10Patch-For-Review, 10Release, 10Train Deployments: 1.39.0-wmf.8 deployment blockers - https://phabricator.wikimedia.org/T305214 (10thcipriani) 05Open→03Resolved [19:33:10] hello! can I get an extra backport window after the UTC late one? I need to backport some patches soonish; the previous attempt didn't go well (I failed to identify a dependency within the patch and it broke the canary) and the late window is already full [19:40:29] thcipriani: I think you are the person to ask ^ that question from these days? [19:42:09] tgr: Sure. I can help with that if the regular backport window deployers are not available after the normal window. [19:43:48] dancy: thx! I can deploy, just wanted to make sure people are aware, since it already almost caused problems once [19:49:23] 10Release-Engineering-Team (🌱 Spring Cleaning — April 2022), 10Patch-For-Review, 10Release, 10Train Deployments: 1.39.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T305215 (10Dzahn) The timeout issues were gone when Brennen repeated the sync a little while later. [19:49:24] tgr: seems like it shouldn't be a problem, coordinate with folks doing the utc late window and that should work fine :) [19:49:48] thanks! [20:24:26] (03CR) 10Ahmon Dancy: "recheck" [releng/phatality] - 10https://gerrit.wikimedia.org/r/772916 (https://phabricator.wikimedia.org/T304736) (owner: 10Krinkle) [21:23:36] 10Beta-Cluster-Infrastructure, 10Wikimedia-Site-requests, 10Growth-Team (Current Sprint), 10Patch-For-Review: Reopen beta eswiki - https://phabricator.wikimedia.org/T306833 (10Tgr) This is done. @Etonkovidova let me know if you'd like anything to be configured differently. (In theory it's the same as real... [21:42:23] 10GitLab (Infrastructure), 10DC-Ops, 10SRE, 10ops-codfw: Q3:(Need By: TBD) rack/setup/install gitlab200[2|3] and gitlab-runner200[2|3|4] - https://phabricator.wikimedia.org/T301183 (10Papaul) [22:05:55] 10GitLab (Infrastructure), 10DC-Ops, 10SRE, 10ops-codfw: Q3:(Need By: TBD) rack/setup/install gitlab200[2|3] and gitlab-runner200[2|3|4] - https://phabricator.wikimedia.org/T301183 (10Papaul) [23:22:47] (03PS18) 10Jeena Huneidi: backport: compare fetched commits with backports [tools/scap] - 10https://gerrit.wikimedia.org/r/773897 (https://phabricator.wikimedia.org/T295495) [23:24:16] (03CR) 10Jeena Huneidi: "Made a few changes to reduce requests to gerrit." [tools/scap] - 10https://gerrit.wikimedia.org/r/773897 (https://phabricator.wikimedia.org/T295495) (owner: 10Jeena Huneidi) [23:27:47] (03PS19) 10Jeena Huneidi: backport: compare fetched commits with backports [tools/scap] - 10https://gerrit.wikimedia.org/r/773897 (https://phabricator.wikimedia.org/T295495)