[00:07:12] <wikibugs>	 10VPS-Projects, 10fundraising-tech-ops, 10Puppet (Puppet 7.0): Update puppet civicrm-prototype puppetmaster - https://phabricator.wikimedia.org/T361595#10163137 (10Dwisehaupt) After some testing (and much distraction by other projects) we're going to upgrade to puppetserver 7 instead of trying to run off the...
[00:07:33] <wikibugs>	 10VPS-Projects, 10fundraising-tech-ops, 10Puppet (Puppet 7.0): Update puppet civicrm-prototype puppetmaster - https://phabricator.wikimedia.org/T361595#10163138 (10Dwisehaupt) a:03Dwisehaupt
[00:09:32] <wikibugs>	 (03update) 10raymond-ndibe: [toolforge-deploy] upgrade wmcs-k8s-metrics [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/520 (https://phabricator.wikimedia.org/T359641)
[00:10:02] <wikibugs>	 (03update) 10raymond-ndibe: [toolforge-deploy] upgrade wmcs-k8s-metrics [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/520 (https://phabricator.wikimedia.org/T359641)
[00:12:08] <wikibugs>	 (03update) 10raymond-ndibe: [toolforge-deploy] upgrade wmcs-k8s-metrics [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/520 (https://phabricator.wikimedia.org/T359641)
[00:14:39] <wikibugs>	 (03update) 10raymond-ndibe: [toolforge-deploy] upgrade wmcs-k8s-metrics [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/520 (https://phabricator.wikimedia.org/T359641)
[00:25:14] <wm-bot2>	 !log raymondndibe@wmf3402 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics (T359641)
[00:25:18] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[00:25:19] <stashbot>	 T359641: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641
[00:30:52] <wm-bot2>	 !log raymondndibe@wmf3402 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics (T359641)
[00:30:55] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[00:30:56] <stashbot>	 T359641: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641
[00:32:29] <wm-bot2>	 !log raymondndibe@wmf3402 tools START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics (T359641)
[00:32:32] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[00:35:56] <wikibugs>	 (03update) 10raymond-ndibe: [toolforge-deploy] upgrade wmcs-k8s-metrics [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/520 (https://phabricator.wikimedia.org/T359641)
[00:39:29] <wm-bot2>	 !log raymondndibe@wmf3402 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics (T359641)
[00:39:33] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[00:39:33] <stashbot>	 T359641: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641
[00:40:03] <wikibugs>	 (03approved) 10raymond-ndibe: [toolforge-deploy] upgrade wmcs-k8s-metrics [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/520 (https://phabricator.wikimedia.org/T359641)
[00:42:59] <wikibugs>	 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 14): [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641#10163180 (10Raymond_Ndibe)
[00:44:53] <wmcs-alerts>	 FIRING: ToolforgeKubernetesNodeNotReady: Kubernetes node tools-k8s-worker-nfs-24 is not ready - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesNodeNotReady - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesNodeNotReady
[00:49:53] <wmcs-alerts>	 FIRING: [2x] ToolforgeKubernetesNodeNotReady: Kubernetes node tools-k8s-worker-nfs-24 is not ready - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesNodeNotReady - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesNodeNotReady
[04:20:41] <jinxer-wm>	 FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[04:30:41] <jinxer-wm>	 RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[05:04:14] <wikibugs>	 (03open) 10samwilson: Update all extensions, skins, and composer dependencies [toolforge-repos/wishlist-test] - 10https://gitlab.wikimedia.org/toolforge-repos/wishlist-test/-/merge_requests/6
[05:07:14] <wikibugs>	 (03update) 10samwilson: Update all extensions, skins, and composer dependencies [toolforge-repos/wishlist-test] - 10https://gitlab.wikimedia.org/toolforge-repos/wishlist-test/-/merge_requests/6
[05:10:27] <wikibugs>	 (03merge) 10samwilson: Update all extensions, skins, and composer dependencies [toolforge-repos/wishlist-test] - 10https://gitlab.wikimedia.org/toolforge-repos/wishlist-test/-/merge_requests/6
[06:29:47] <wm-bot2>	 !log dcaro@urcuchillay tools END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=1)
[06:29:50] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[06:34:53] <wmcs-alerts>	 RESOLVED: ToolforgeKubernetesNodeNotReady: Kubernetes node tools-k8s-worker-nfs-24 is not ready - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesNodeNotReady - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesNodeNotReady
[06:50:58] <wmcs-alerts>	 RESOLVED: InstanceDown: Project tools instance tools-k8s-worker-nfs-24 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[08:18:25] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: openstack: clarify default security group semantics - https://phabricator.wikimedia.org/T375111#10163387 (10aborrero) Obviously I was missing this `data` tofu functionality when considering the options earlier. I like this approach, I believe it covers...
[08:25:50] <wikibugs>	 10cloud-services-team (FY2024/2025-Q1-Q2): [cloudceph] Improve downtime when a switch goes down - https://phabricator.wikimedia.org/T375204#10163423 (10aborrero)
[08:27:03] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (T373740)
[08:27:08] <stashbot>	 T373740: 2024-08-31 cloudvirt1048 NodeDown because memory hardware error - https://phabricator.wikimedia.org/T373740
[08:27:12] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0) (T373740)
[08:27:53] <wikibugs>	 06cloud-services-team, 06DC-Ops, 10ops-eqiad, 06SRE: 2024-08-31 cloudvirt1048 NodeDown because memory hardware error - https://phabricator.wikimedia.org/T373740#10163445 (10aborrero) thanks!
[08:29:11] <wikibugs>	 06cloud-services-team, 06DC-Ops, 10ops-eqiad, 06SRE: 2024-08-31 cloudvirt1048 NodeDown because memory hardware error - https://phabricator.wikimedia.org/T373740#10163458 (10aborrero)
[08:49:17] <wikibugs>	 10cloud-services-team (FY2024/2025-Q1-Q2): cloud: edge network suffers downtime if one cloudsw is down - https://phabricator.wikimedia.org/T375259 (10aborrero) 03NEW
[08:49:33] <wikibugs>	 10cloud-services-team (FY2024/2025-Q1-Q2), 06Infrastructure-Foundations, 10netops: cloud: edge network suffers downtime if one cloudsw is down - https://phabricator.wikimedia.org/T375259#10163529 (10aborrero)
[08:50:25] <wikibugs>	 10cloud-services-team (FY2024/2025-Q1-Q2), 06Infrastructure-Foundations, 10netops: cloud: edge network suffers downtime if one cloudsw is down - https://phabricator.wikimedia.org/T375259#10163533 (10aborrero) p:05Triage→03Medium I'm tagging #netops in the hope they can help us figure out what happens her...
[10:50:41] <jinxer-wm>	 FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[10:59:22] <wikibugs>	 (03update) 10aborrero: secgroups: add optional default security group [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50 (https://phabricator.wikimedia.org/T375111)
[10:59:25] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50
[10:59:32] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50
[11:00:06] <wikibugs>	 (03update) 10aborrero: secgroups: add optional default security group [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50 (https://phabricator.wikimedia.org/T375111)
[11:00:33] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50
[11:00:41] <jinxer-wm>	 RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[11:00:42] <logmsgbot_cloud>	 !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50
[11:01:24] <wikibugs>	 (03update) 10aborrero: secgroups: add optional default security group [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50 (https://phabricator.wikimedia.org/T375111)
[11:09:56] <wikibugs>	 (03update) 10aborrero: secgroups: add optional default security group [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50 (https://phabricator.wikimedia.org/T375111)
[11:11:28] <wikibugs>	 (03update) 10aborrero: secgroups: add optional default security group [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50 (https://phabricator.wikimedia.org/T375111)
[11:18:25] <wikibugs>	 (03update) 10aborrero: secgroups: add optional default security group [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50 (https://phabricator.wikimedia.org/T375111)
[11:20:48] <wikibugs>	 (03update) 10aborrero: secgroups: add optional default security group [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50 (https://phabricator.wikimedia.org/T375111)
[11:28:28] <wikibugs>	 (03update) 10aborrero: secgroups: add optional default security group [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50 (https://phabricator.wikimedia.org/T375111)
[11:30:56] <wikibugs>	 (03update) 10aborrero: secgroups: add optional default security group [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50 (https://phabricator.wikimedia.org/T375111)
[12:38:48] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: openstack: clarify default security group semantics - https://phabricator.wikimedia.org/T375111#10163933 (10aborrero) I have integrated the idea by @fnegri in the patch at https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50...
[12:41:28] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: openstack: clarify default security group semantics - https://phabricator.wikimedia.org/T375111#10163949 (10aborrero) We can extend this workflow in the following way:  * delete neutron native default security group rules * when a new project is created...
[13:50:41] <jinxer-wm>	 FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[14:05:41] <jinxer-wm>	 RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[15:43:37] <wikibugs>	 (03CR) 10D3r1ck01: [C:03+2] Add mw-node-qunit project [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1074107 (https://phabricator.wikimedia.org/T375079) (owner: 10Ebrahim)
[15:44:55] <wikibugs>	 (03Merged) 10jenkins-bot: Add mw-node-qunit project [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1074107 (https://phabricator.wikimedia.org/T375079) (owner: 10Ebrahim)
[15:49:50] <wikibugs>	 10Toolforge: unexpected root-owned files in /data/project/pywikibot/public_html - https://phabricator.wikimedia.org/T375279 (10valhallasw) 03NEW
[15:52:47] <wikibugs>	 06cloud-services-team, 10Toolforge, 10Pywikibot: unexpected root-owned files in /data/project/pywikibot/public_html - https://phabricator.wikimedia.org/T375279#10164326 (10fnegri)
[15:52:55] <wikibugs>	 06cloud-services-team, 10Toolforge, 10Pywikibot: unexpected root-owned files in /data/project/pywikibot/public_html - https://phabricator.wikimedia.org/T375279#10164328 (10JJMC89)
[16:11:59] <wikibugs>	 (03merge) 10raymond-ndibe: [toolforge-deploy] upgrade wmcs-k8s-metrics [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/520 (https://phabricator.wikimedia.org/T359641)
[16:33:49] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: tofu-infra: refactor repo structure - https://phabricator.wikimedia.org/T375283 (10fnegri) 03NEW
[16:47:56] <wikibugs>	 (03approved) 10fnegri: secgroups: add optional default security group [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50 (https://phabricator.wikimedia.org/T375111) (owner: 10aborrero)
[16:48:10] <wikibugs>	 (03update) 10fnegri: secgroups: add optional default security group [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50 (https://phabricator.wikimedia.org/T375111) (owner: 10aborrero)
[16:49:37] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: tofu-infra: refactor repo structure - https://phabricator.wikimedia.org/T375283#10164509 (10fnegri)
[16:50:38] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: tofu-infra: refactor repo structure - https://phabricator.wikimedia.org/T375283#10164512 (10fnegri)
[16:51:25] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: tofu-infra: refactor repo structure - https://phabricator.wikimedia.org/T375283#10164515 (10fnegri)
[16:54:38] <wm-bot2>	 !log raymondndibe@wmf3402 tools START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (T359641)
[16:54:38] <wm-bot2>	 !log raymondndibe@wmf3402 tools Updating container image docker-registry.tools.wmflabs.org/calico/cni:v3.28.2 (T359641)
[16:54:41] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[16:54:42] <stashbot>	 T359641: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641
[16:54:44] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[16:56:37] <logmsgbot_cloud>	 !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (T359641)
[16:56:37] <logmsgbot_cloud>	 !log raymond-ndibe@cloudcumin1001 tools Updating container image docker-registry.tools.wmflabs.org/calico/cni:v3.28.2 (T359641)
[16:57:03] <logmsgbot_cloud>	 !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) (T359641)
[16:59:34] <logmsgbot_cloud>	 !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (T359641)
[16:59:34] <logmsgbot_cloud>	 !log raymond-ndibe@cloudcumin1001 tools Updating container image docker-registry.tools.wmflabs.org/calico/ctl:v3.28.2 (T359641)
[16:59:46] <logmsgbot_cloud>	 !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) (T359641)
[16:59:50] <stashbot>	 T359641: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641
[17:02:18] <logmsgbot_cloud>	 !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (T359641)
[17:02:18] <logmsgbot_cloud>	 !log raymond-ndibe@cloudcumin1001 tools Updating container image docker-registry.tools.wmflabs.org/calico/kube-controllers:v3.28.2 (T359641)
[17:02:32] <logmsgbot_cloud>	 !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) (T359641)
[17:03:08] <logmsgbot_cloud>	 !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (T359641)
[17:03:08] <logmsgbot_cloud>	 !log raymond-ndibe@cloudcumin1001 tools Updating container image docker-registry.tools.wmflabs.org/calico/node:v3.28.2 (T359641)
[17:04:02] <logmsgbot_cloud>	 !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) (T359641)
[17:04:38] <logmsgbot_cloud>	 !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (T359641)
[17:04:38] <logmsgbot_cloud>	 !log raymond-ndibe@cloudcumin1001 tools Updating container image docker-registry.tools.wmflabs.org/calico/typha:v3.28.2 (T359641)
[17:04:51] <logmsgbot_cloud>	 !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) (T359641)
[17:04:55] <stashbot>	 T359641: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641
[17:05:57] <logmsgbot_cloud>	 !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (T359641)
[17:05:57] <logmsgbot_cloud>	 !log raymond-ndibe@cloudcumin1001 tools Updating container image docker-registry.tools.wmflabs.org/calico/pod2daemon-flexvol:v3.28.2 (T359641)
[17:06:02] <logmsgbot_cloud>	 !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) (T359641)
[17:16:54] <wikibugs>	 (03open) 10raymond-ndibe: [calico] upgrade to 3.28.2 [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/10 (https://phabricator.wikimedia.org/T359641)
[17:17:53] <wikibugs>	 (03approved) 10raymond-ndibe: [calico] upgrade to 3.28.2 [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/10 (https://phabricator.wikimedia.org/T359641)
[17:18:00] <wikibugs>	 (03merge) 10raymond-ndibe: [calico] upgrade to 3.28.2 [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/10 (https://phabricator.wikimedia.org/T359641)
[17:19:23] <wikibugs>	 (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: calico: bump to 0.0.10-20240920171811-afd7f481 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/524 (https://phabricator.wikimedia.org/T359641)
[17:25:18] <wikibugs>	 (03open) 10raymond-ndibe: Revert "[calico] upgrade to 3.28.2" [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/11
[17:25:47] <wikibugs>	 (03close) 10raymond-ndibe: calico: bump to 0.0.10-20240920171811-afd7f481 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/524 (https://phabricator.wikimedia.org/T359641) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38)
[17:26:08] <wikibugs>	 (03approved) 10raymond-ndibe: Revert "[calico] upgrade to 3.28.2" [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/11
[17:26:12] <wikibugs>	 (03merge) 10raymond-ndibe: Revert "[calico] upgrade to 3.28.2" [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/11
[17:27:29] <wikibugs>	 (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: calico: bump to 0.0.11-20240920172624-a95e9e60 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/525 (https://phabricator.wikimedia.org/T359641)
[17:27:53] <wikibugs>	 (03close) 10raymond-ndibe: calico: bump to 0.0.11-20240920172624-a95e9e60 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/525 (https://phabricator.wikimedia.org/T359641) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38)
[17:30:09] <wikibugs>	 (03open) 10raymond-ndibe: [calico] upgrade to 3.28.2 [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/12 (https://phabricator.wikimedia.org/T359641)
[17:30:31] <wikibugs>	 (03update) 10raymond-ndibe: [calico] upgrade to 3.28.2 [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/12 (https://phabricator.wikimedia.org/T359641)
[17:31:52] <wikibugs>	 (03update) 10raymond-ndibe: [calico] upgrade to 3.28.2 [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/12 (https://phabricator.wikimedia.org/T359641)
[17:32:36] <wikibugs>	 (03update) 10raymond-ndibe: [calico] upgrade to 3.28.2 [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/12 (https://phabricator.wikimedia.org/T359641)
[17:33:11] <wikibugs>	 (03update) 10raymond-ndibe: [calico] upgrade to 3.28.2 [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/12 (https://phabricator.wikimedia.org/T359641)
[17:34:39] <wikibugs>	 (03update) 10raymond-ndibe: [calico] upgrade to 3.28.2 [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/12 (https://phabricator.wikimedia.org/T359641)
[17:35:00] <wikibugs>	 (03update) 10raymond-ndibe: [calico] upgrade to 3.28.2 [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/12 (https://phabricator.wikimedia.org/T359641)
[17:41:17] <wikibugs>	 10Tool-lexeme-forms: Lexeme-forms on Toolforge returns error - https://phabricator.wikimedia.org/T374344#10164712 (10LucasWerkmeister)
[17:41:55] <wikibugs>	 10Tool-lexeme-forms, 10toolforge_i18n, 07I18n, 03Wikimania-Hackathon-2024: Extract Python library for Wikimedia tool i18n from Wikidata Lexeme Forms tool - https://phabricator.wikimedia.org/T283376#10164716 (10LucasWerkmeister)
[17:42:32] <wikibugs>	 10Tool-lexeme-forms, 06translatewiki.net, 10Wikidata, 10Wikidata Lexicographical data, 13Patch-For-Review: Add Wikidata Lexeme Forms to translatewiki.net - https://phabricator.wikimedia.org/T272243#10164717 (10LucasWerkmeister)
[17:42:45] <wikibugs>	 10Tool-lexeme-forms, 06translatewiki.net, 07Unplanned-Sprint-Work: Wikidata Lexeme Forms uses inappropriate language names for translations - https://phabricator.wikimedia.org/T336606#10164718 (10LucasWerkmeister)
[17:42:47] <wikibugs>	 10Tool-lexeme-forms, 10Toolforge: Wikidata Lexeme Forms tool configuration was public - https://phabricator.wikimedia.org/T286414#10164719 (10LucasWerkmeister)
[17:43:00] <wikibugs>	 10Tool-lexeme-forms, 10Wikidata, 10Wikidata Lexicographical data: Support editing unmatched forms in Wikidata Lexeme Forms - https://phabricator.wikimedia.org/T252214#10164720 (10LucasWerkmeister)
[17:43:21] <wikibugs>	 10Tool-lexeme-forms, 10Wikidata, 10Wikidata Lexicographical data: Make form fields in Wikidata Lexeme Forms longer - https://phabricator.wikimedia.org/T238040#10164721 (10LucasWerkmeister)
[17:44:45] <wikibugs>	 10Tool-lexeme-forms: Lexeme-forms on Toolforge returns error - https://phabricator.wikimedia.org/T374344#10164725 (10LucasWerkmeister) 05Open→03Resolved a:03LucasWerkmeister Let’s close this for now, feel free to reopen if it comes back.
[17:50:43] <wikibugs>	 10Quarry: Emtpy Quarry query names are not linked/should not be allowed - https://phabricator.wikimedia.org/T375292 (101234qwer1234qwer4) 03NEW
[19:05:27] <wikibugs>	 (03approved) 10raymond-ndibe: [calico] upgrade to 3.28.2 [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/12 (https://phabricator.wikimedia.org/T359641)
[19:05:33] <wikibugs>	 (03merge) 10raymond-ndibe: [calico] upgrade to 3.28.2 [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/12 (https://phabricator.wikimedia.org/T359641)
[19:07:18] <wikibugs>	 (03update) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: calico: bump to 0.0.12-20240920190545-a342cb41 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/526 (https://phabricator.wikimedia.org/T359641)
[19:07:20] <wikibugs>	 (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: calico: bump to 0.0.12-20240920190545-a342cb41 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/526 (https://phabricator.wikimedia.org/T359641)
[19:08:05] <wikibugs>	 (03open) 10raymond-ndibe: Revert "[calico] upgrade to 3.28.2" [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/13
[19:08:11] <wikibugs>	 (03approved) 10raymond-ndibe: Revert "[calico] upgrade to 3.28.2" [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/13
[19:08:54] <wikibugs>	 (03merge) 10raymond-ndibe: Revert "[calico] upgrade to 3.28.2" [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/13
[19:10:13] <wikibugs>	 (03update) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: calico: bump to 0.0.12-20240920190545-a342cb41 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/526 (https://phabricator.wikimedia.org/T359641)
[19:10:43] <wikibugs>	 (03update) 10raymond-ndibe: Revert "[calico] upgrade to 3.28.2" [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/13
[19:10:51] <wikibugs>	 (03update) 10raymond-ndibe: [calico] upgrade to 3.28.2 [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/12 (https://phabricator.wikimedia.org/T359641)
[19:10:58] <wikibugs>	 (03update) 10raymond-ndibe: Revert "[calico] upgrade to 3.28.2" [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/11
[19:11:07] <wikibugs>	 (03update) 10raymond-ndibe: [calico] upgrade to 3.28.2 [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/10 (https://phabricator.wikimedia.org/T359641)
[19:12:20] <wikibugs>	 (03update) 10raymond-ndibe: calico: bump to 0.0.12-20240920190545-a342cb41 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/526 (https://phabricator.wikimedia.org/T359641) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38)
[19:12:24] <wikibugs>	 (03close) 10raymond-ndibe: calico: bump to 0.0.12-20240920190545-a342cb41 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/526 (https://phabricator.wikimedia.org/T359641) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38)
[19:12:35] <wikibugs>	 (03update) 10raymond-ndibe: calico: bump to 0.0.11-20240920172624-a95e9e60 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/525 (https://phabricator.wikimedia.org/T359641) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38)
[19:12:41] <wikibugs>	 (03update) 10raymond-ndibe: calico: bump to 0.0.10-20240920171811-afd7f481 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/524 (https://phabricator.wikimedia.org/T359641) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38)
[19:13:46] <wikibugs>	 (03open) 10raymond-ndibe: [calico] upgrade to 3.28.2 [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/14 (https://phabricator.wikimedia.org/T359641)
[19:14:02] <wikibugs>	 (03update) 10raymond-ndibe: [calico] upgrade to 3.28.2 [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/14 (https://phabricator.wikimedia.org/T359641)
[19:14:11] <wikibugs>	 (03approved) 10raymond-ndibe: [calico] upgrade to 3.28.2 [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/14 (https://phabricator.wikimedia.org/T359641)
[19:15:22] <wikibugs>	 (03merge) 10raymond-ndibe: [calico] upgrade to 3.28.2 [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/14 (https://phabricator.wikimedia.org/T359641)
[19:16:38] <wikibugs>	 (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: calico: bump to 0.0.14-20240920191533-f94f2f8d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/527 (https://phabricator.wikimedia.org/T359641)
[19:19:34] <logmsgbot_cloud>	 !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component calico (T341066)
[19:19:39] <stashbot>	 T341066: [jobs-api,jobs-cli] Support multiple replicas of continuous jobs - https://phabricator.wikimedia.org/T341066
[19:24:49] <logmsgbot_cloud>	 !log raymond-ndibe@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component calico (T341066)
[19:24:53] <stashbot>	 T341066: [jobs-api,jobs-cli] Support multiple replicas of continuous jobs - https://phabricator.wikimedia.org/T341066
[19:26:29] <logmsgbot_cloud>	 !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component calico (T341066)
[19:30:29] <logmsgbot_cloud>	 !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico (T341066)
[19:30:35] <stashbot>	 T341066: [jobs-api,jobs-cli] Support multiple replicas of continuous jobs - https://phabricator.wikimedia.org/T341066
[19:31:08] <logmsgbot_cloud>	 !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component calico (T341066)
[19:36:23] <logmsgbot_cloud>	 !log raymond-ndibe@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component calico (T341066)
[19:36:27] <stashbot>	 T341066: [jobs-api,jobs-cli] Support multiple replicas of continuous jobs - https://phabricator.wikimedia.org/T341066
[19:50:41] <jinxer-wm>	 FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[20:05:41] <jinxer-wm>	 RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks
[20:06:47] <logmsgbot_cloud>	 !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component calico (T341066)
[20:06:50] <stashbot>	 T341066: [jobs-api,jobs-cli] Support multiple replicas of continuous jobs - https://phabricator.wikimedia.org/T341066
[20:08:23] <logmsgbot_cloud>	 !log raymond-ndibe@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component calico (T341066)
[20:08:29] <logmsgbot_cloud>	 !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component calico (T341066)
[20:12:51] <logmsgbot_cloud>	 !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico (T341066)
[20:12:55] <stashbot>	 T341066: [jobs-api,jobs-cli] Support multiple replicas of continuous jobs - https://phabricator.wikimedia.org/T341066
[20:13:45] <wikibugs>	 (03approved) 10raymond-ndibe: calico: bump to 0.0.14-20240920191533-f94f2f8d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/527 (https://phabricator.wikimedia.org/T359641) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38)
[20:13:49] <wikibugs>	 (03merge) 10raymond-ndibe: calico: bump to 0.0.14-20240920191533-f94f2f8d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/527 (https://phabricator.wikimedia.org/T359641) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38)
[20:14:36] <wikibugs>	 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 14), 13Patch-For-Review: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641#10165048 (10Raymond_Ndibe)
[21:05:07] <wikibugs>	 10Tool-video-answer-tool, 06Future-Audiences, 07Spike: Investigate different options for animation of images - https://phabricator.wikimedia.org/T374367#10165116 (10etz) @Maryana I think I need access to the linked google doc on this ticket
[23:29:54] <wikibugs>	 06cloud-services-team, 06DC-Ops, 10ops-eqiad, 06SRE, 13Patch-For-Review: Put cloudcephosd10[39-41] into service - https://phabricator.wikimedia.org/T372814#10165304 (10Jclark-ctr) @Andrew  i see this ticket is in my name. is there something i need to do for this?
[23:30:57] <wikibugs>	 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS, 06DC-Ops, 10ops-eqiad, 06SRE: cloudcephosd1021-1034: hard drive sector errors increasing - https://phabricator.wikimedia.org/T348643#10165306 (10Jclark-ctr) @dcaro  did you have an update with what servers and drives I can send?   I will reach out o...