[00:01:11] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api [00:04:32] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor [00:04:52] (03approved) 10raymond-ndibe: maintain-harbor: bump to 0.0.20-20250106225550-dc5e7ed4 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/649 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [00:04:56] (03update) 10raymond-ndibe: maintain-harbor: bump to 0.0.20-20250106225550-dc5e7ed4 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/649 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [00:05:14] (03merge) 10raymond-ndibe: maintain-harbor: bump to 0.0.20-20250106225550-dc5e7ed4 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/649 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [00:06:31] (03update) 10raymond-ndibe: components-api: bump to 0.0.74-20250106230052-21087bd0 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/653 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [00:08:08] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api [00:09:47] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component components-api [00:09:53] !log raymond-ndibe@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api [00:10:51] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component components-api [00:10:57] !log raymond-ndibe@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api [00:12:17] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component wmcs-metrics [00:12:21] !log raymond-ndibe@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component wmcs-metrics [00:13:42] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component wmcs-metrics [00:13:48] !log raymond-ndibe@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component wmcs-metrics [00:14:16] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component components-api [00:14:22] !log raymond-ndibe@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api [00:14:51] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor [00:15:07] !log raymond-ndibe@cloudcumin1001 toolsbeta END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component maintain-harbor [00:16:07] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics [00:22:35] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics [00:23:20] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics [00:29:16] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics [00:29:53] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component calico [00:31:26] !log raymond-ndibe@cloudcumin1001 toolsbeta END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component calico [00:33:28] (03update) 10raymond-ndibe: [jobs-api] replicas default to 1 in NewJob model [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/132 (https://phabricator.wikimedia.org/T364204) [00:33:38] (03update) 10raymond-ndibe: [toolforge-deploy] add more test cases to job loads [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/646 (https://phabricator.wikimedia.org/T364204) [00:37:18] (03update) 10raymond-ndibe: [maintain-harbor] get_example_config() return content of .env file [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/41 [00:37:48] (03update) 10raymond-ndibe: [toolforge-deploy] add maintain-harbor image retention tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/648 [01:05:23] 10VPS-Projects, 10fundraising-tech-ops, 10Puppet (Puppet 7.0): Update puppet civicrm-prototype puppetmaster - https://phabricator.wikimedia.org/T361595#10435861 (10Dwisehaupt) 05Open→03Resolved Closing this out since things have been ok with the new instances. [01:26:03] 06cloud-services-team, 10VPS-Projects, 13Patch-For-Review, 10Puppet (Puppet 7.0): Migrate per-project Puppet servers to Puppet 7 - https://phabricator.wikimedia.org/T351452#10435873 (10Dwisehaupt) [05:25:11] 06cloud-services-team, 10Data-Services: enwiki.analytics.db.svc.wikimedia.cloud still on replag for more than a week - https://phabricator.wikimedia.org/T372224#10436128 (10Marostegui) [05:31:27] 10Cloud Services Proposals, 06cloud-services-team, 06Data-Persistence, 06Data-Platform-SRE: Decision request - Who runs wikireplicas cookbooks - https://phabricator.wikimedia.org/T382607#10436138 (10Marostegui) >>! In T382607#10433585, @dcaro wrote: >> Those cookbooks are for the views, which is something... [08:57:53] 10Cloud Services Proposals, 06cloud-services-team, 06Data-Persistence, 06Data-Platform-SRE: Decision request - Who runs wikireplicas cookbooks - https://phabricator.wikimedia.org/T382607#10436521 (10dcaro) > They are involved because they own an-redacteddb hosts which have views and they are installed via... [08:58:49] 10Cloud Services Proposals, 06cloud-services-team, 06Data-Persistence, 06Data-Platform-SRE: Decision request - Who runs wikireplicas cookbooks - https://phabricator.wikimedia.org/T382607#10436523 (10Marostegui) I think I am also confused with all the renaming - I don't know anymore! [09:48:43] 10Cloud Services Proposals, 06cloud-services-team, 06Data-Persistence, 06Data-Platform-SRE: Decision request - Who runs wikireplicas cookbooks - https://phabricator.wikimedia.org/T382607#10436620 (10fnegri) My understanding: #data-engineering are responsible for the views definition, which columns should... [09:50:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [10:00:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [10:28:29] 06cloud-services-team, 10Cloud-VPS, 10Toolforge, 07Kubernetes: Allow Toolforge roots to use the cookbook to reboot k8s worker nodes (without wmcs-root) - https://phabricator.wikimedia.org/T382977#10436712 (10fnegri) I think having the toolforge cookbooks use the openstack API is the right way to go. We cou... [10:33:16] (03update) 10dcaro: scheduled job: add timeout parameter [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/76 (https://phabricator.wikimedia.org/T306391) [10:33:21] (03update) 10dcaro: scheduled jobs: add timeout option [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/129 (https://phabricator.wikimedia.org/T306391) [10:39:27] 06cloud-services-team, 10Toolforge, 06Design-Research, 07Design: Toolforge UI: Publish newcomer experience and recruitment survey - https://phabricator.wikimedia.org/T381266#10436737 (10Sarai-WMF) [10:42:37] 10Cloud Services Proposals, 06cloud-services-team, 06Data-Persistence, 06Data-Platform-SRE: Decision request - Who runs wikireplicas cookbooks - https://phabricator.wikimedia.org/T382607#10436742 (10dcaro) >>! In T382607#10436620, @fnegri wrote: ... > The last time we had some issues running the cookbook,... [10:43:58] 06cloud-services-team, 10Cloud-VPS, 10SRE Observability (FY2024/2025-Q2): Remove librenms -> graphite integration, replace with gnmi - https://phabricator.wikimedia.org/T372457#10436747 (10fgiunchedi) Thank you for the feedback @dcaro, appreciate it! We did resolve the gaps issue by extending the `rate()` wi... [10:47:07] 10Toolforge (Toolforge iteration 16), 13Patch-For-Review: [components-api] add basic prometheus instrumentation - https://phabricator.wikimedia.org/T381249#10436756 (10dcaro) p:05Triage→03Medium [10:48:09] 10Toolforge (Toolforge iteration 16): Persist maintain-harbor logs - https://phabricator.wikimedia.org/T383081#10436757 (10dcaro) p:05Triage→03High [11:13:20] 06cloud-services-team, 10Cloud-VPS, 10SRE Observability (FY2024/2025-Q2): Remove librenms -> graphite integration, replace with gnmi - https://phabricator.wikimedia.org/T372457#10436805 (10cmooney) Thanks for the work on this guys. @dcaro my concerns above were misplaced, the gaps were due to the queries I'... [13:29:53] 06cloud-services-team, 10Cloud-VPS, 10SRE Observability (FY2024/2025-Q2): Remove librenms -> graphite integration, replace with gnmi - https://phabricator.wikimedia.org/T372457#10437114 (10dcaro) @fgiunchedi just created a version of that graph with the new stats (gnmi), but I'm seeing some discrepancies. *... [13:32:33] 06cloud-services-team, 10Cloud-VPS, 10SRE Observability (FY2024/2025-Q2): Remove librenms -> graphite integration, replace with gnmi - https://phabricator.wikimedia.org/T372457#10437119 (10dcaro) > The traffic on the gnmi stats, does not seem to differentiate between in/out I'm the worst xd, `_in_octects` m... [13:47:01] 06cloud-services-team, 10Toolforge (Toolforge iteration 16): toolforge: Refresh certs that are not controlled by kubeadm (mid 2024 edition) - https://phabricator.wikimedia.org/T309782#10437164 (10dcaro) 05Stalled→03Resolved [13:47:40] 10Toolforge (Toolforge iteration 16): [toolforge-weld] read setting from envvars too - https://phabricator.wikimedia.org/T379893#10437166 (10dcaro) 05In progress→03Resolved [13:51:27] 10Striker, 13Patch-For-Review: toolsadmin.wikimedia.org is unavailable (2024-08-24) - https://phabricator.wikimedia.org/T373250#10437177 (10dcaro) The patch is merged, and should page only WMCS, will leave the task open for a bit to make sure. [13:51:31] 10Striker, 13Patch-For-Review: toolsadmin.wikimedia.org is unavailable (2024-08-24) - https://phabricator.wikimedia.org/T373250#10437178 (10dcaro) 05Open→03In progress [13:55:38] 06cloud-services-team, 10Toolforge: Add support for Python 3.13 - https://phabricator.wikimedia.org/T381899#10437180 (10dcaro) p:05Triage→03Medium [14:22:34] 06cloud-services-team, 10Toolforge (Toolforge iteration 16): [infra,k8s] remove deprecated kubelet flags before 1.28 upgrade (we might be able to remove all custom ones) - https://phabricator.wikimedia.org/T370245#10437294 (10dcaro) [14:24:54] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 16): [infra,k8s] Upgrade Toolforge Kubernetes to version 1.29 - https://phabricator.wikimedia.org/T362868#10437297 (10dcaro) [14:50:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [14:51:04] 06cloud-services-team, 10Toolforge, 07Epic: [Epic] Toolforge UI: Discovery - https://phabricator.wikimedia.org/T375914#10437372 (10Sarai-WMF) [14:51:06] 06cloud-services-team, 10Toolforge, 06Design-Research, 07Design: Toolforge UI: Publish newcomer experience and recruitment survey - https://phabricator.wikimedia.org/T381266#10437373 (10Sarai-WMF) [15:00:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [15:29:17] 10Striker: toolsadmin.wikimedia.org is unavailable (2024-08-24) - https://phabricator.wikimedia.org/T373250#10437481 (10dcaro) 05In progress→03Resolved [15:33:53] 10wikitech.wikimedia.org, 06serviceops-radar, 06SRE, 07SRE-Unowned: Redesign wikitech-static - https://phabricator.wikimedia.org/T376400#10437497 (10Andrew) https://wts.wmcloud.org/wiki/Main_Page.html now has a search bar (just on that one page) which is VERY ugly but which does provide useful (to my eyes,... [15:42:02] 06cloud-services-team, 10Cloud-VPS, 10Continuous-Integration-Infrastructure, 10ci-test-error (WMF-deployed Build Failure), and 2 others: Various CI jobs running in the integration Cloud VPS project failing due to transient DNS lookup failures, often for ou... - https://phabricator.wikimedia.org/T374830#10437519 [15:44:38] 06cloud-services-team, 10Cloud-VPS, 10Continuous-Integration-Infrastructure, 10ci-test-error (WMF-deployed Build Failure), and 2 others: Various CI jobs running in the integration Cloud VPS project failing due to transient DNS lookup failures, often for ou... - https://phabricator.wikimedia.org/T374830#10437522 [17:04:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance toolsbeta-puppetdb-03 on project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [17:06:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance tools-puppetdb-2 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [17:20:39] (03CR) 10Majavah: [C:03+2] Re-design home page with Codex [labs/striker] - 10https://gerrit.wikimedia.org/r/1106046 (https://phabricator.wikimedia.org/T380114) (owner: 10Majavah) [17:23:00] (03Merged) 10jenkins-bot: Re-design home page with Codex [labs/striker] - 10https://gerrit.wikimedia.org/r/1106046 (https://phabricator.wikimedia.org/T380114) (owner: 10Majavah) [17:33:38] (03PS1) 10Majavah: Fix robots.txt loading [labs/striker] - 10https://gerrit.wikimedia.org/r/1108796 [17:34:06] (03CR) 10Majavah: [C:03+2] Fix robots.txt loading [labs/striker] - 10https://gerrit.wikimedia.org/r/1108796 (owner: 10Majavah) [17:36:50] (03Merged) 10jenkins-bot: Fix robots.txt loading [labs/striker] - 10https://gerrit.wikimedia.org/r/1108796 (owner: 10Majavah) [17:42:10] 10Striker: striker-tools and striker-toolsbeta memcached backend is shared - https://phabricator.wikimedia.org/T383143 (10taavi) 03NEW p:05Triage→03High [17:42:49] (03PS1) 10Majavah: settings: Make cache prefix configurable [labs/striker] - 10https://gerrit.wikimedia.org/r/1108800 [17:43:17] (03PS2) 10Majavah: settings: Make cache prefix configurable [labs/striker] - 10https://gerrit.wikimedia.org/r/1108800 (https://phabricator.wikimedia.org/T383143) [17:44:26] (03open) 10dcaro: add admin console to splash [toolforge-repos/admin-web] (add_dev_docs) - 10https://gitlab.wikimedia.org/toolforge-repos/admin-web/-/merge_requests/2 [17:45:05] (03update) 10dcaro: add admin console to splash [toolforge-repos/admin-web] (add_dev_docs) - 10https://gitlab.wikimedia.org/toolforge-repos/admin-web/-/merge_requests/2 [17:46:33] (03update) 10dcaro: add admin console to splash [toolforge-repos/admin-web] (add_dev_docs) - 10https://gitlab.wikimedia.org/toolforge-repos/admin-web/-/merge_requests/2 [17:49:28] RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance toolsbeta-puppetdb-03 on project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [17:51:28] RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance tools-puppetdb-2 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [17:53:20] (03CR) 10David Caro: [C:03+1] "LGTM" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1108454 (owner: 10Raymond Ndibe) [18:01:08] 06cloud-services-team, 10Toolforge, 07Epic: [WIP] Toolforge UI: Investigate integration of Striker functionality - https://phabricator.wikimedia.org/T383146 (10Sarai-WMF) 03NEW [18:15:57] 06cloud-services-team, 10Toolforge, 03Wikimedia-Hackathon-2025: [WIP, placeholder] Introducing and evaluating Toolforge UI - https://phabricator.wikimedia.org/T383149 (10Sarai-WMF) 03NEW [18:18:17] 06cloud-services-team, 10Toolforge, 07Epic: [WIP] Toolforge UI: Investigate integration of Striker functionality - https://phabricator.wikimedia.org/T383146#10438208 (10bd808) The description given for option 1 is full of biasing statements without evidence. Many of them sound objectively incorrect to me, bu... [18:28:23] 06cloud-services-team, 10Toolforge, 07Epic: [WIP] Toolforge UI: Investigate integration of Striker functionality - https://phabricator.wikimedia.org/T383146#10438225 (10taavi) > Notes from previous discussion: The current implementation would present many blockers and require a lot of re-architecting to allo... [18:30:18] !log bd808@cloudcumin1001 deployment-prep START - Cookbook wmcs.vps.remove_instance for instance deployment-etcd04 [18:30:24] !log bd808@cloudcumin1001 deployment-prep END (FAIL) - Cookbook wmcs.vps.remove_instance (exit_code=1) for instance deployment-etcd04 [18:40:13] (03CR) 10Majavah: [C:03+2] settings: Make cache prefix configurable [labs/striker] - 10https://gerrit.wikimedia.org/r/1108800 (https://phabricator.wikimedia.org/T383143) (owner: 10Majavah) [18:41:31] (03Merged) 10jenkins-bot: settings: Make cache prefix configurable [labs/striker] - 10https://gerrit.wikimedia.org/r/1108800 (https://phabricator.wikimedia.org/T383143) (owner: 10Majavah) [18:42:03] (03PS1) 10Majavah: fixtures: software_license: Format JSON with eslint [labs/striker] - 10https://gerrit.wikimedia.org/r/1108807 [18:43:54] (03CR) 10Majavah: [C:03+2] fixtures: software_license: Format JSON with eslint [labs/striker] - 10https://gerrit.wikimedia.org/r/1108807 (owner: 10Majavah) [18:45:10] (03Merged) 10jenkins-bot: fixtures: software_license: Format JSON with eslint [labs/striker] - 10https://gerrit.wikimedia.org/r/1108807 (owner: 10Majavah) [18:49:24] 06cloud-services-team, 10Cloud-VPS, 10Beta-Cluster-Infrastructure: prometheus-openstack-stale-puppet-certs crashing on deployment-puppetserver-1.deployment-prep.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T383153#10438323 (10bd808) [19:00:19] 06cloud-services-team, 10Cloud-VPS, 10Beta-Cluster-Infrastructure: prometheus-openstack-stale-puppet-certs crashing on deployment-puppetserver-1.deployment-prep.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T383153#10438358 (10bd808) a:03taavi After @taavi cleaned the certs causing the script... [19:02:13] 10Striker, 13Patch-For-Review: striker-tools and striker-toolsbeta memcached backend is shared - https://phabricator.wikimedia.org/T383143#10438363 (10taavi) 05Open→03Resolved a:03taavi [19:23:12] 10cloud-services-team (Hardware), 10Cloud-VPS, 06DC-Ops, 10ops-eqiad, 06SRE: Relocate cloudnet1007-dev and cloudnet1008-dev to new racks and rename - https://phabricator.wikimedia.org/T382412#10438432 (10Andrew) Note that these servers are not currently in service, so this move can happen anytime w/out W... [19:24:58] 06cloud-services-team, 10Cloud-VPS, 10Beta-Cluster-Infrastructure: Future growth of deployment-prep? - https://phabricator.wikimedia.org/T381420#10438434 (10Andrew) 05Open→03Resolved a:03Andrew We've now placed the order for new cloudvirts so I'm going to close this task for now. [19:25:26] 06cloud-services-team, 10Cloud-VPS, 06collaboration-services, 10Continuous-Integration-Infrastructure, and 2 others: Future testing-infra growth on cloud-vps - https://phabricator.wikimedia.org/T381419#10438458 (10Andrew) 05Open→03Resolved a:03Andrew We've now placed the order for new cloudvirts... [19:50:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [20:30:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [20:36:10] 06cloud-services-team, 10Horizon: Clean up horizon/deploy branches - https://phabricator.wikimedia.org/T382957#10438728 (10Andrew) In the top repo I have deleted most branches (leaving only the two most recent ones because I'm scared to delete everything). I have also force-pushed things to 'main' so that 'mai... [21:20:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [21:45:41] 06cloud-services-team, 10Horizon: Clean up horizon/deploy branches - https://phabricator.wikimedia.org/T382957#10438897 (10Andrew) Oh, and I also made 'main' the default branch. [22:30:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [22:52:18] (03open) 10raymond-ndibe: [maintain-harbor] persist log [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/42 (https://phabricator.wikimedia.org/T383081) [22:52:23] (03update) 10raymond-ndibe: [maintain-harbor] persist log [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/42 (https://phabricator.wikimedia.org/T383081) [22:53:14] (03update) 10raymond-ndibe: [maintain-harbor] persist log [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/42 (https://phabricator.wikimedia.org/T383081) [22:53:32] (03update) 10raymond-ndibe: [maintain-harbor] persist log [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/42 (https://phabricator.wikimedia.org/T383081) [23:24:51] FIRING: ProbeDown: Service tools-static-15:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-15:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [23:29:50] RESOLVED: ProbeDown: Service tools-static-15:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-15:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown