[00:16:28] (InstanceDown) firing: Project tf-infra-test instance tf-infra-test is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [00:21:28] (InstanceDown) resolved: Project tf-infra-test instance tf-infra-test is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [02:19:53] 10Wikibugs: Report GitLab merge request events to IRC - https://phabricator.wikimedia.org/T362500 (10bd808) 03NEW [02:42:47] (CloudVPSDesignateLeaks) firing: (2) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [02:47:41] (CloudVPSDesignateLeaks) firing: (3) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [02:52:41] (CloudVPSDesignateLeaks) firing: (3) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [02:57:41] (CloudVPSDesignateLeaks) resolved: (3) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [03:15:17] 10Wikibugs: Report GitLab merge request events to IRC - https://phabricator.wikimedia.org/T362500#9712180 (10bd808) Quick mockup using `jq` of data likely to be used in building messages: `lang=shell-session $ grep '"object_kind": "merge_request"' events.log | grep "bd808" | jq '{"action": .object_attributes.act... [03:33:47] 10Wikibugs, 10GitLab (Integrations), 10Release-Engineering-Team (Priority Backlog 📥): Connect WikiBugs IRC bot to Wikimedia GitLab - https://phabricator.wikimedia.org/T288381#9712181 (10bd808) Getting [[https://docs.gitlab.com/ee/user/project/integrations/webhook_events.html#comment-events|comment]], [[https... [06:12:41] (CloudVPSDesignateLeaks) firing: (2) Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:17:41] (CloudVPSDesignateLeaks) firing: (3) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:22:41] (CloudVPSDesignateLeaks) firing: (3) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:27:41] (CloudVPSDesignateLeaks) resolved: (3) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:56:52] 06cloud-services-team, 10VPS-project-devtools, 06collaboration-services, 13Patch-For-Review, and 2 others: Update devtools project puppetmaster - https://phabricator.wikimedia.org/T360470#9712252 (10Jelto) Puppet runs on some machines which use the new puppetmaster in devtools fail, here example of `gitlab... [08:24:06] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api [08:24:10] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [08:24:40] !log dcaro@urcuchillay toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api [08:24:42] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [09:02:42] !log dcaro@urcuchillay tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api [09:02:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [09:03:15] !log dcaro@urcuchillay tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api [09:03:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [09:15:09] 10cloud-services-team (FY2023/2024-Q3-Q4), 05Cloud-Services-Origin-User, 07Cloud-Services-Worktype-Unplanned: [tools,harbor] failing to push images to harbor - https://phabricator.wikimedia.org/T362515 (10dcaro) 03NEW p:05Triage→03High [09:16:57] 10cloud-services-team (FY2023/2024-Q3-Q4), 05Cloud-Services-Origin-User, 07Cloud-Services-Worktype-Unplanned: [tools,harbor] failing to push images to harbor - https://phabricator.wikimedia.org/T362515#9712562 (10dcaro) [09:30:13] 10cloud-services-team (FY2023/2024-Q3-Q4), 05Cloud-Services-Origin-User, 07Cloud-Services-Worktype-Unplanned: [tools,harbor] failing to push images to harbor - https://phabricator.wikimedia.org/T362515#9712596 (10dcaro) Restarting all the containers helped with the image pushing, that's restored. For the U... [09:32:04] 10cloud-services-team (FY2023/2024-Q3-Q4), 05Cloud-Services-Origin-User, 07Cloud-Services-Worktype-Unplanned: [tools,harbor] failing to push images to harbor - https://phabricator.wikimedia.org/T362515#9712602 (10dcaro) It seems that the tmp directory was full on proxy-03.project-proxy.eqiad1.wikimedia.cloud... [09:42:41] (CloudVPSDesignateLeaks) firing: (2) Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [09:47:41] (CloudVPSDesignateLeaks) firing: (3) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [09:50:00] 10cloud-services-team (FY2023/2024-Q3-Q4), 05Cloud-Services-Origin-User, 07Cloud-Services-Worktype-Unplanned: [tools,harbor] failing to push images to harbor - https://phabricator.wikimedia.org/T362515#9712668 (10dcaro) See {T354116} for previous instances of this same issue [09:52:40] 06cloud-services-team, 10Toolforge (Toolforge iteration 08): Harbor uploads sometimes fail due to tmpfs space on project-proxy - https://phabricator.wikimedia.org/T354116#9712676 (10dcaro) I'm thinking that it might be another cache that fills up, we might want to try setting: https://nginx.org/en/docs/http/n... [09:52:41] (CloudVPSDesignateLeaks) firing: (3) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [09:53:30] 10cloud-services-team (FY2023/2024-Q3-Q4), 05Cloud-Services-Origin-User, 07Cloud-Services-Worktype-Unplanned: 14[tools,harbor] failing to push images to harbor - 14https://phabricator.wikimedia.org/T362515#9712678 (10dcaro) 05Open→03Resolved 14Closing this, will work on the tmp space on T354116 [09:57:41] (CloudVPSDesignateLeaks) resolved: (3) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [10:51:14] !log dcaro@cloudcumin1001 owidm START - Cookbook wmcs.openstack.quota_increase (T361895) [10:51:18] T361895: owidm storage quota request - https://phabricator.wikimedia.org/T361895 [10:51:22] !log dcaro@cloudcumin1001 owidm END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) (T361895) [10:58:15] 10Cloud-VPS (Quota-requests): 14owidm storage quota request - 14https://phabricator.wikimedia.org/T361895#9712800 (10dcaro) 05Open→03Resolved a:03dcaro 14Done, @Tim-moody let me know if you have any issues. [10:58:36] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder [10:58:46] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder [10:59:21] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder [10:59:32] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder [11:02:16] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers [11:02:26] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers [11:03:43] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers [11:03:53] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers [11:06:10] 10Toolforge (Toolforge iteration 08): [maintain-kubeusers] Increment default services quota - https://phabricator.wikimedia.org/T362520 (10taavi) 03NEW p:05Triage→03High [11:10:02] 10Toolforge (Toolforge iteration 08), 13Patch-For-Review: [api-gateway] Add a python server to serve consolidated openapi docs - https://phabricator.wikimedia.org/T362299#9712838 (10aborrero) >>! In T362299#9706660, @aborrero wrote: > > I think I misread the earlier comment, I thought you were suggesting to r... [11:20:08] (03PS2) 10Majavah: vps: remove_instance: Silence metricsinfra alerts before deleting [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1018272 (https://phabricator.wikimedia.org/T360932) [11:20:08] (03PS1) 10Majavah: build: Require Spicerack 8.5 or newer [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1019723 [11:21:59] 10Toolforge: toolforge jobs api logs internal datetime error - https://phabricator.wikimedia.org/T362521 (10aborrero) 03NEW [11:22:56] (03CR) 10CI reject: [V:04-1] build: Require Spicerack 8.5 or newer [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1019723 (owner: 10Majavah) [11:23:05] (03CR) 10CI reject: [V:04-1] vps: remove_instance: Silence metricsinfra alerts before deleting [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1018272 (https://phabricator.wikimedia.org/T360932) (owner: 10Majavah) [11:24:21] (03PS2) 10Majavah: build: Require Spicerack 8.5 or newer [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1019723 [11:24:21] (03PS3) 10Majavah: vps: remove_instance: Silence metricsinfra alerts before deleting [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1018272 (https://phabricator.wikimedia.org/T360932) [11:24:21] (03PS1) 10Majavah: proxy: Replace ALERTMANAGER_URLS [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1019724 [11:27:08] (03CR) 10CI reject: [V:04-1] build: Require Spicerack 8.5 or newer [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1019723 (owner: 10Majavah) [11:27:13] 10Toolforge: [jobs-api] logs internal datetime error - https://phabricator.wikimedia.org/T362521#9712872 (10dcaro) [11:27:16] 10Toolforge: [jobs-api] logs internal datetime error - https://phabricator.wikimedia.org/T362521#9712873 (10dcaro) p:05Triage→03Medium [11:27:18] (03CR) 10CI reject: [V:04-1] proxy: Replace ALERTMANAGER_URLS [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1019724 (owner: 10Majavah) [11:27:22] (03CR) 10CI reject: [V:04-1] vps: remove_instance: Silence metricsinfra alerts before deleting [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1018272 (https://phabricator.wikimedia.org/T360932) (owner: 10Majavah) [11:28:27] 10Toolforge: [lima-kilo] toolforge_deploy_mr.py confusing message - https://phabricator.wikimedia.org/T362389#9712876 (10dcaro) I think it just checks the exit code, and probably it's 0 xd [11:28:48] 10Toolforge: [lima-kilo] toolforge_deploy_mr.py confusing message - https://phabricator.wikimedia.org/T362389#9712883 (10dcaro) p:05Triage→03Low [11:31:10] 10Toolforge: Rust buildservice failed to clone a repository from GitHub - https://phabricator.wikimedia.org/T362404#9712890 (10dcaro) Yep, I have seen connectivity issues towards github in the last few days generally on cloud VMs (toolforge included). [11:31:24] (03PS3) 10Majavah: build: Require Spicerack 8.5 or newer [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1019723 [11:31:25] (03PS4) 10Majavah: vps: remove_instance: Silence metricsinfra alerts before deleting [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1018272 (https://phabricator.wikimedia.org/T360932) [11:31:25] (03PS1) 10Majavah: build: Update Black [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1019727 [11:31:25] (03PS1) 10Majavah: build: Remove explicit types-requests pin [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1019728 [11:33:16] (03PS2) 10Majavah: build: Update Black [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1019727 [11:33:16] (03PS2) 10Majavah: proxy: Replace ALERTMANAGER_URLS [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1019724 [11:33:16] (03PS4) 10Majavah: build: Require Spicerack 8.5 or newer [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1019723 [11:33:16] (03PS2) 10Majavah: build: Remove explicit types-requests pin [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1019728 [11:33:17] (03PS5) 10Majavah: vps: remove_instance: Silence metricsinfra alerts before deleting [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1018272 (https://phabricator.wikimedia.org/T360932) [11:36:21] (03CR) 10CI reject: [V:04-1] build: Update Black [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1019727 (owner: 10Majavah) [11:37:06] (03CR) 10David Caro: [C:03+1] "LGTM" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1019727 (owner: 10Majavah) [11:37:55] (03CR) 10Majavah: [V:03+2 C:03+2] "Force-merging since there's a circular dependency between this one and Icbe020fb229b80d6ca4de4456343e6e04c1743a1." [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1019727 (owner: 10Majavah) [11:39:54] (03CR) 10David Caro: [C:03+1] proxy: Replace ALERTMANAGER_URLS [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1019724 (owner: 10Majavah) [11:40:31] (03CR) 10Majavah: [C:03+2] proxy: Replace ALERTMANAGER_URLS [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1019724 (owner: 10Majavah) [11:42:30] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api [11:42:41] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api [11:44:09] (03Merged) 10jenkins-bot: proxy: Replace ALERTMANAGER_URLS [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1019724 (owner: 10Majavah) [11:49:58] (03PS6) 10Majavah: vps: remove_instance: Silence metricsinfra alerts before deleting [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1018272 (https://phabricator.wikimedia.org/T360932) [11:54:35] 10Toolforge (Toolforge iteration 08), 13Patch-For-Review: [api-gateway] Add a python server to serve consolidated openapi docs - https://phabricator.wikimedia.org/T362299#9712930 (10dcaro) >>! In T362299#9712838, @aborrero wrote: >>>! In T362299#9706660, @aborrero wrote: >> >> I think I misread the earlier co... [12:12:41] (CloudVPSDesignateLeaks) firing: (2) Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [12:17:41] (CloudVPSDesignateLeaks) firing: (3) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [12:22:41] (CloudVPSDesignateLeaks) firing: (3) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [12:27:41] (CloudVPSDesignateLeaks) resolved: (3) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [12:33:32] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [labs/tools/massmailer] - 10https://gerrit.wikimedia.org/r/1019754 (owner: 10L10n-bot) [12:39:17] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Cloud-VPS, 10Cumin, 06Infrastructure-Foundations, 13Patch-For-Review: [cumin] [openstack] Openstack backend fails when project is not set - https://phabricator.wikimedia.org/T346453#9713036 (10fnegri) 05In progress→03Stalled Last week I tried reproducing t... [12:46:59] 10cloud-services-team (FY2023/2024-Q3-Q4), 06Infrastructure-Foundations, 10Spicerack, 10SRE-tools, and 2 others: Remove elasticsearch-curator dependency from Spicerack/Elastic cookbooks - https://phabricator.wikimedia.org/T361647#9713106 (10Gehel) [12:47:28] (03CR) 10Nikerabbit: [V:03+2] Localisation updates from https://translatewiki.net. [labs/tools/massmailer] - 10https://gerrit.wikimedia.org/r/1019754 (owner: 10L10n-bot) [12:54:20] 06cloud-services-team, 06Infrastructure-Foundations, 06SRE, 10vm-requests: Site: 1 VM for codfw1dev bitu deployment - https://phabricator.wikimedia.org/T362128#9713149 (10MoritzMuehlenhoff) You don't need 4G of RAM, 2 should be perfectly fine. Also, let's not use idm2001-dev, that's too confusin with the... [12:54:30] 06cloud-services-team, 06Infrastructure-Foundations, 06SRE, 10vm-requests: Site: 1 VM for codfw1dev bitu deployment - https://phabricator.wikimedia.org/T362128#9713152 (10MoritzMuehlenhoff) p:05Triage→03Medium [12:55:10] 06cloud-services-team, 06Infrastructure-Foundations, 06SRE, 10vm-requests: Site: 1 VM for codfw1dev bitu deployment - https://phabricator.wikimedia.org/T362128#9713154 (10SLyngshede-WMF) a:03SLyngshede-WMF [13:14:41] (CloudVPSDesignateLeaks) firing: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:24:41] (CloudVPSDesignateLeaks) resolved: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:29:39] 10cloud-services-team (FY2023/2024-Q3-Q4), 06Infrastructure-Foundations, 10Spicerack, 10SRE-tools, 13Patch-For-Review: spicerack: tox fails to install PyYAML using python 3.11 on bookworm - https://phabricator.wikimedia.org/T345337#9713348 (10Volans) With the above patch I think the issue should be solve... [13:36:20] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api [13:36:31] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api [13:38:03] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api [13:38:14] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api [13:38:24] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api [13:38:28] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api [13:38:57] 10cloud-services-team (FY2023/2024-Q3-Q4), 06Infrastructure-Foundations, 10Spicerack, 10SRE-tools, 13Patch-For-Review: spicerack: tox fails to install PyYAML using python 3.11 on bookworm - https://phabricator.wikimedia.org/T345337#9713370 (10fnegri) Works for me! 🎉 ` spicerack (master) $ docker run --r... [13:40:06] 10cloud-services-team (FY2023/2024-Q3-Q4), 06Infrastructure-Foundations, 10Spicerack, 10SRE-tools, 13Patch-For-Review: spicerack: tox fails to install PyYAML using python 3.11 on bookworm - https://phabricator.wikimedia.org/T345337#9713373 (10fnegri) `pip install wikimedia-spicerack` is also working fine... [13:41:24] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api [13:41:35] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api [13:42:55] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api [13:43:06] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api [14:00:12] 10Tools, 10Wikidata: 14wikidata-todo: important_blank_items throws uncaught exception ToolforgeCommon::getSQL - 14https://phabricator.wikimedia.org/T305788#9713447 (10Magnus) 05Open→03Declined 14This tool would not scale to today's Wikidata size [14:09:18] 10cloud-services-team (FY2023/2024-Q3-Q4), 06Infrastructure-Foundations, 10Spicerack, 10SRE-tools, 13Patch-For-Review: 14spicerack: tox fails to install PyYAML using python 3.11 on bookworm - 14https://phabricator.wikimedia.org/T345337#9713473 (10Volans) 05Stalled→03Resolved 14Resolving then, th... [14:13:22] 06cloud-services-team, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Project: [wm-lol]Add a shortcut for tables to https://www.mediawiki.org/w/index.php?title=Manual:Database_layout/diagram&action=render - https://phabricator.wikimedia.org/T299232#9713522 (10dcaro) Done \o/ https://wm-lol.toolforge... [14:13:28] 06cloud-services-team, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Project: 14[wm-lol]Add a shortcut for tables to https://www.mediawiki.org/w/index.php?title=Manual:Database_layout/diagram&action=render - 14https://phabricator.wikimedia.org/T299232#9713523 (10dcaro) 05Open→03Resolved [14:14:01] 06cloud-services-team, 10VPS-project-devtools, 06collaboration-services, 13Patch-For-Review, and 2 others: Update devtools project puppetmaster - https://phabricator.wikimedia.org/T360470#9713527 (10Dzahn) That's new. It worked before and "no route to host" sounds like the machine is down or networking. [14:14:09] 06cloud-services-team, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Maintenance: 14[ceph] Investigate why cloudceph1021 has an abnormal disk usage (>20% vs <7%) - 14https://phabricator.wikimedia.org/T302982#9713528 (10dcaro) 05Open→03Resolved 14Will reopen if relevant [14:14:47] 10Tool-Pageviews: Outdated referral traffic info in FAQ - https://phabricator.wikimedia.org/T362535 (10mpopov) 03NEW [14:15:06] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api [14:15:15] 10Tool-Pageviews: Outdated referral traffic info in FAQ - https://phabricator.wikimedia.org/T362535#9713547 (10mpopov) Prompted by [[ https://wikimedia.slack.com/archives/C0153LQ5G82/p1711963832120499?thread_ts=1711961904.652889&cid=C0153LQ5G82 | confusion in Slack ]] [14:15:17] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api [14:20:27] 10Toolforge, 07Security: 14builds-api allows impersonating any user by bypassing local TLS termination - 14https://phabricator.wikimedia.org/T362525#9713565 (10taavi) [14:20:39] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Toolforge (Toolforge iteration 08): [builds-api] Add dashboards with the new statistics - https://phabricator.wikimedia.org/T352764#9713574 (10fnegri) 05Open→03In progress [14:38:05] !log taavi@cloudcumin1001 library-upgrader START - Cookbook wmcs.vps.remove_instance for instance upgrader-06 [14:38:17] !log taavi@cloudcumin1001 library-upgrader END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance upgrader-06 [14:42:41] (CloudVPSDesignateLeaks) firing: (2) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [14:47:41] (CloudVPSDesignateLeaks) firing: (3) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [14:47:51] 10PAWS: Remove prometheus migrate logic - https://phabricator.wikimedia.org/T362102#9713715 (10github-toolforge-bot) vivian-rook closed https://github.com/toolforge/paws/pull/401 [14:48:06] vivian-rook closed https://github.com/toolforge/paws/pull/401 [14:48:08] 10PAWS: 14Remove prometheus migrate logic - 14https://phabricator.wikimedia.org/T362102#9713717 (10rook) 05Open→03Resolved a:03rook [15:03:08] 10Tool-Global-user-contributions, 06Stewards-and-global-tools, 10Temporary accounts, 10XTools, and 2 others: [Design] Update wireframes with user testing learnings - https://phabricator.wikimedia.org/T359827#9713804 (10KColeman-WMF) [15:04:14] 10Data-Services, 06Data-Engineering: Expose more properties to the user_properties_anon table on Wiki Replicas - https://phabricator.wikimedia.org/T226162#9713805 (10odimitrijevic) a:05odimitrijevic→03None [15:06:21] 10Toolforge: [api-gateway] Explore using network policies to further secure traffic between toolforge api-gateway router and backends - https://phabricator.wikimedia.org/T362539 (10taavi) 03NEW [15:09:21] 10Toolforge: [api-gateway] Explore using network policies to further secure traffic between toolforge api-gateway router and backends - https://phabricator.wikimedia.org/T362539#9713857 (10aborrero) [15:17:28] 10Toolforge: [api-gateway] Explore using network policies to further secure traffic between toolforge api-gateway router and backends - https://phabricator.wikimedia.org/T362539#9713890 (10dcaro) p:05Triage→03Medium [15:35:41] 10Tool-bridgebot, 07Upstream: Bridgebot freaks out and sends double messages from IRC to Telegram - https://phabricator.wikimedia.org/T305487#9714053 (10bd808) I really need to work on this again. The bot breaks a lot and @LucasWerkmeister is amazing for [[https://wikitech.wikimedia.org/wiki/Nova_Resource:Tool... [15:47:19] 10Data-Services, 06Data-Engineering: 14Expose more properties to the user_properties_anon table on Wiki Replicas - 14https://phabricator.wikimedia.org/T226162#9714113 (10odimitrijevic) 05Open→03Declined 14With the migration to liftwing these settings are no longer applicable. cc @calbon  [15:52:41] (CloudVPSDesignateLeaks) firing: (3) Detected 5 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [15:52:49] 06cloud-services-team, 10Toolforge: [toolforge,storage] Provide per-tool access to cloud-vps object storage - https://phabricator.wikimedia.org/T358496#9714141 (10bd808) [15:54:14] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: [infra] Replace PodSecurityPolicy in Toolforge Kubernetes - https://phabricator.wikimedia.org/T279110#9714145 (10CodeReviewBot) aborrero opened https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/18 Draft: maintain_ku... [15:57:31] 06cloud-services-team, 10Toolforge: Find a modern hostname for tools-static.wmflabs.org - https://phabricator.wikimedia.org/T361435#9714148 (10bd808) >>! In T361435#9714121, @taavi wrote: > I've secretly been hoping that the current iteration of tools-static would go away at some point in favour of some object... [15:57:41] (CloudVPSDesignateLeaks) resolved: (3) Detected 5 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [16:03:15] 06cloud-services-team, 10VPS-project-devtools, 06collaboration-services, 13Patch-For-Review, and 2 others: Update devtools project puppetmaster - https://phabricator.wikimedia.org/T360470#9714191 (10Andrew) The latest puppetserver code is prone to gobbling RAM; I'd check for oom messages and see about usin... [16:13:25] 06cloud-services-team, 10Tool-toolviews, 10Toolforge, 03Wikimedia-Hackathon-2024: Provide basic page view metrics for individual tools on toolforge - https://phabricator.wikimedia.org/T87001#9714264 (10MusikAnimal) [16:14:45] 14Grid-Engine-to-K8s-Migration, 10Tools, 06All-and-every-Wikisource: Migrate phetools from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319965#9714273 (10Epigeneticist) Hi, is there any chance of MATCH&SPLIT being reintegrated into Wikisource soon? Many contributors (incl... [16:35:32] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-codfw, 06SRE: Q3:rack/setup/install cloudcontrol2006-dev.codfw.wmnet - https://phabricator.wikimedia.org/T354896#9714430 (10Jhancock.wm) [16:36:19] 06cloud-services-team, 10VPS-project-devtools, 06collaboration-services, 13Patch-For-Review, and 2 others: Update devtools project puppetmaster - https://phabricator.wikimedia.org/T360470#9714432 (10brennen) > Also, just so i'm clear... your new puppet7 server is still named 'puppetmaster' rathe... [16:41:40] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-codfw, 06SRE: Q3:rack/setup/install cloudcontrol2006-dev.codfw.wmnet - https://phabricator.wikimedia.org/T354896#9714481 (10Jhancock.wm) a:03Jhancock.wm @cmooney what is the vlan for this server? racked in : B1-U42 port: 41 [16:44:41] (CloudVPSDesignateLeaks) firing: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [16:48:02] 06cloud-services-team, 10VPS-project-devtools, 06collaboration-services, 13Patch-For-Review, and 2 others: Update devtools project puppetmaster - https://phabricator.wikimedia.org/T360470#9714532 (10Dzahn) Currently can't ssh to puppetmaster-1003 from external either - also with "no route to host". I'll... [16:49:06] 10wikitech.wikimedia.org, 10DiscussionTools, 10Editing-team (Kanban Board), 10MW-1.42-notes (1.42.0-wmf.25; 2024-04-02), 07Verified: 14Page state routing triggers DiscussionTools warning, e.g. #!/deploycal/current - 14https://phabricator.wikimedia.org/T361322#9714540 (10Esanders) 05Open→03Resolved [16:49:09] 06cloud-services-team, 10VPS-project-devtools, 06collaboration-services, 13Patch-For-Review, and 2 others: Update devtools project puppetmaster - https://phabricator.wikimedia.org/T360470#9714541 (10Dzahn) ` [?2004hroot@puppetmaster-1003:~# [ 899.382711] /dev/sdb: Can't open blockdev ` [16:51:59] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-codfw, 06SRE: Q3:rack/setup/install cloudcontrol2009-dev.codfw.wmnet - https://phabricator.wikimedia.org/T354896#9714556 (10Jhancock.wm) [16:52:12] 06cloud-services-team, 10VPS-project-devtools, 06collaboration-services, 13Patch-For-Review, and 2 others: Update devtools project puppetmaster - https://phabricator.wikimedia.org/T360470#9714557 (10Dzahn) "soft reboot" resurrected it, can ssh to it again. running puppet on the puppetmaster shows: ` Err... [16:54:41] (CloudVPSDesignateLeaks) resolved: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [16:55:16] 06cloud-services-team, 10VPS-project-devtools, 06collaboration-services, 13Patch-For-Review, and 2 others: Update devtools project puppetmaster - https://phabricator.wikimedia.org/T360470#9714583 (10Dzahn) Unfortunately this hasn't fixed the issue for agents yet, for example gitlab-runner-1002 now says:... [17:07:47] 10Toolforge (Toolforge iteration 08), 13Patch-For-Review: [jobs-api] Split the API, business, and k8s models - https://phabricator.wikimedia.org/T359808#9714677 (10CodeReviewBot) project_1317_bot_df3177307bed93c3f34e421e26c86e38 opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge... [17:08:51] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-codfw, 06SRE: Q3:rack/setup/install cloudcontrol2009-dev.codfw.wmnet - https://phabricator.wikimedia.org/T354896#9714676 (10Papaul) @Jhancock.wm cloud-hosts1-b1-codfw (2118) [17:10:07] 06cloud-services-team, 10VPS-project-devtools, 06collaboration-services, 13Patch-For-Review, and 2 others: Update devtools project puppetmaster - https://phabricator.wikimedia.org/T360470#9714686 (10Dzahn) ` Exception in thread "main" java.lang.IllegalStateException: Unable to borrow JRubyInstance from poo... [17:11:04] 06cloud-services-team, 10VPS-project-devtools, 06collaboration-services, 13Patch-For-Review, and 2 others: Update devtools project puppetmaster - https://phabricator.wikimedia.org/T360470#9714687 (10Dzahn) well, "parent directory /srv/puppet/server does not exist" sounds at least fixable, but wouldn't this... [17:14:07] 06cloud-services-team, 10VPS-project-devtools, 06collaboration-services, 13Patch-For-Review, and 2 others: Update devtools project puppetmaster - https://phabricator.wikimedia.org/T360470#9714700 (10Dzahn) ` # @param ssldir_on_srv used on cloud-vps; it allows storing certs on a detachable volume ` ` i... [17:30:12] !log raymond@ubuntu toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api [17:30:16] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [17:31:06] !log raymond@ubuntu toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api [17:31:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [17:40:26] 10VPS-project-Codesearch, 06serviceops: Add docker production images repo to codesearch - https://phabricator.wikimedia.org/T362567 (10akosiaris) 03NEW [17:44:41] (CloudVPSDesignateLeaks) firing: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:50:16] !log raymond@ubuntu toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api [17:50:19] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [17:51:10] !log raymond@ubuntu toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api [17:51:13] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [17:54:42] (CloudVPSDesignateLeaks) resolved: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [18:20:48] !log raymond@ubuntu toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api [18:20:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [18:21:36] !log raymond@ubuntu toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api [18:21:38] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [18:27:21] (03PS1) 10Scott French: write_config: index operations/docker-images/production-images [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1019863 (https://phabricator.wikimedia.org/T362567) [18:27:41] !log raymond@ubuntu tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api [18:27:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [18:28:26] !log raymond@ubuntu tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api [18:28:29] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [18:34:02] 10Toolforge (Toolforge iteration 08), 13Patch-For-Review: [jobs-api] Split the API, business, and k8s models - https://phabricator.wikimedia.org/T359808#9715029 (10CodeReviewBot) raymond-ndibe merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/259 jobs-api: bump to 0.0... [18:43:41] (CloudVPSDesignateLeaks) firing: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [18:51:56] (03CR) 10Scott French: "Adding Amir as well based on recent review activity. If you have a chance to review, that would be greatly appreciated :)" [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1019863 (https://phabricator.wikimedia.org/T362567) (owner: 10Scott French) [18:58:41] (CloudVPSDesignateLeaks) resolved: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [19:16:16] 10Toolforge (Toolforge iteration 08): [builds-api] replace all error message models with ResponseMessages - https://phabricator.wikimedia.org/T361901#9715161 (10CodeReviewBot) raymond-ndibe merged https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/86 [builds-api] update oapi-codegen [19:16:19] 10Toolforge (Toolforge iteration 08), 13Patch-For-Review: [builds-api, envvars-api] add oapi-codegen installation to makefile - https://phabricator.wikimedia.org/T362290#9715160 (10CodeReviewBot) raymond-ndibe merged https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/86 [builds-api... [19:35:07] (03PS3) 10Krinkle: app.py: remove DESCRIPTIONS from old UI, add "Hound" to title [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1017413 [19:45:44] (03PS1) 10Dzahn: delete graphite.discovery.wmnet dummy key [labs/private] - 10https://gerrit.wikimedia.org/r/1019889 (https://phabricator.wikimedia.org/T360414) [20:02:40] 06cloud-services-team, 10VPS-project-devtools, 06collaboration-services, 13Patch-For-Review, and 2 others: Update devtools project puppetmaster - https://phabricator.wikimedia.org/T360470#9715403 (10Dzahn) After the above created /srv/puppet the service could start again and puppet runs on clients work aga... [20:12:41] (CloudVPSDesignateLeaks) firing: (2) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [20:18:58] 10Toolforge (Toolforge iteration 08), 13Patch-For-Review: [builds-api, envvars-api] add oapi-codegen installation to makefile - https://phabricator.wikimedia.org/T362290#9715439 (10CodeReviewBot) raymond-ndibe opened https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/28 [envvars-a... [20:21:31] 10Toolforge (Toolforge iteration 08), 13Patch-For-Review: [builds-api, envvars-api] add oapi-codegen installation to makefile - https://phabricator.wikimedia.org/T362290#9715454 (10CodeReviewBot) project_1317_bot_df3177307bed93c3f34e421e26c86e38 opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolfor... [20:22:41] (CloudVPSDesignateLeaks) firing: (2) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [20:23:18] 10Toolforge (Toolforge iteration 08), 13Patch-For-Review: [builds-api] replace all error message models with ResponseMessages - https://phabricator.wikimedia.org/T361901#9715455 (10CodeReviewBot) project_1317_bot_df3177307bed93c3f34e421e26c86e38 opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolfor... [20:26:05] !log raymond@ubuntu toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api [20:26:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [20:26:56] !log raymond@ubuntu toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api [20:26:58] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [20:27:41] (CloudVPSDesignateLeaks) resolved: (2) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [20:33:30] !log raymond@ubuntu tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api [20:33:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [20:34:22] !log raymond@ubuntu tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api [20:34:24] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [20:40:03] 10Toolforge (Toolforge iteration 08), 13Patch-For-Review: [builds-api, envvars-api] add oapi-codegen installation to makefile - https://phabricator.wikimedia.org/T362290#9715526 (10CodeReviewBot) raymond-ndibe merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/260 buil... [20:40:04] 10Toolforge (Toolforge iteration 08), 13Patch-For-Review: [builds-api] replace all error message models with ResponseMessages - https://phabricator.wikimedia.org/T361901#9715527 (10CodeReviewBot) raymond-ndibe merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/260 buil... [20:45:36] 10Wikibugs: Hashar does not like grey foreground color for distinguishing closed status events - https://phabricator.wikimedia.org/T360353#9715532 (10bd808) 05Open→03In progress p:05Triage→03Medium a:03bd808 I'm going to switch the formatting from grey foreground to ~~strikethrough~~ text style. The ~~... [20:53:07] 10Wikibugs, 13Patch-For-Review: Hashar does not like grey foreground color for distinguishing closed status events - https://phabricator.wikimedia.org/T360353#9715551 (10CodeReviewBot) bd808 opened https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/31 irc: Use strikethrough style rather t... [20:56:37] 10Wikibugs: 14wikibugs test bug - 14https://phabricator.wikimedia.org/T1152#9715559 (10bd808) 14test comment on closed task [21:00:27] 10Wikibugs, 13Patch-For-Review: Hashar does not like grey foreground color for distinguishing closed status events - https://phabricator.wikimedia.org/T360353#9715565 (10bd808) Rendering of T1152#9715558 in my Textual client with some additional styling triggered by my irc nick being part of the the message: {... [21:01:09] 10Wikibugs: 14wikibugs test bug - 14https://phabricator.wikimedia.org/T1152#9715570 (10bd808) 14another closed task comment [21:05:22] 10Wikibugs, 13Patch-For-Review: Hashar does not like grey foreground color for distinguishing closed status events - https://phabricator.wikimedia.org/T360353#9715589 (10bd808) Rendering of T1152#9715566 via https://web.libera.chat/. Note that this client does not render the ~~strikethrough~~ style, but it als... [21:07:51] 10Wikibugs, 13Patch-For-Review: Hashar does not like grey foreground color for distinguishing closed status events - https://phabricator.wikimedia.org/T360353#9715590 (10CodeReviewBot) bd808 merged https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/31 irc: Use strikethrough style rather t... [21:12:41] (CloudVPSDesignateLeaks) firing: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [21:14:01] 10Wikibugs, 13Patch-For-Review: Hashar does not like grey foreground color for distinguishing closed status events - https://phabricator.wikimedia.org/T360353#9715597 (10bd808) 05In progress→03Resolved [21:17:41] (CloudVPSDesignateLeaks) firing: (2) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [21:22:41] (CloudVPSDesignateLeaks) resolved: (2) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [21:35:06] (03PS1) 10Alex Paskulin: ui: Fix typo [labs/striker] - 10https://gerrit.wikimedia.org/r/1019928 [21:37:30] (03CR) 10Ladsgroup: [C:03+2] write_config: index operations/docker-images/production-images [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1019863 (https://phabricator.wikimedia.org/T362567) (owner: 10Scott French) [21:38:25] (03Merged) 10jenkins-bot: write_config: index operations/docker-images/production-images [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1019863 (https://phabricator.wikimedia.org/T362567) (owner: 10Scott French) [21:40:14] 10VPS-project-Codesearch, 06serviceops, 13Patch-For-Review: Add docker production images repo to codesearch - https://phabricator.wikimedia.org/T362567#9715635 (10Ladsgroup) I merged the change, it'll be live in a day, if not, ping me to kick the VM violently like a True Persian. [21:49:18] 10VPS-project-Codesearch, 06serviceops, 13Patch-For-Review: Add docker production images repo to codesearch - https://phabricator.wikimedia.org/T362567#9715651 (10Scott_French) a:03Scott_French Many thanks for the review, Amir. Great, I'll check back tomorrow and close this out once I've confirmed its live. [22:17:51] (03CR) 10BryanDavis: [C:03+2] ui: Fix typo [labs/striker] - 10https://gerrit.wikimedia.org/r/1019928 (owner: 10Alex Paskulin) [22:19:17] (03Merged) 10jenkins-bot: ui: Fix typo [labs/striker] - 10https://gerrit.wikimedia.org/r/1019928 (owner: 10Alex Paskulin) [23:42:41] (CloudVPSDesignateLeaks) firing: (2) Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [23:47:41] (CloudVPSDesignateLeaks) firing: (3) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [23:52:41] (CloudVPSDesignateLeaks) firing: (3) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [23:57:41] (CloudVPSDesignateLeaks) resolved: (3) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks