[01:18:57] FIRING: CloudVPSDesignateLeaks: Detected 3 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [16:30:44] !log andrew@cloudcumin1001 cloudvirt-canary START - Cookbook wmcs.openstack.cloudvirt.lib.ensure_canary on eqiad1, with recreate False, for hosts list: ['cloudvirt1052'] [16:30:50] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Cloud-VPS, 13Patch-For-Review: Migrate eqiad1 hypervisors to Neutron OVS agent - https://phabricator.wikimedia.org/T364457#9910766 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1002 for host cloudvirt1052.eqiad.wmnet with OS bo... [16:31:06] !log andrew@cloudcumin1001 cloudvirt-canary END (PASS) - Cookbook wmcs.openstack.cloudvirt.lib.ensure_canary (exit_code=0) on eqiad1, with recreate False, for hosts list: ['cloudvirt1052'] [16:33:46] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Cloud-VPS: Migrate eqiad1 hypervisors to Neutron OVS agent - https://phabricator.wikimedia.org/T364457#9910789 (10Andrew) [16:34:34] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Cloud-VPS: Migrate eqiad1 hypervisors to Neutron OVS agent - https://phabricator.wikimedia.org/T364457#9910807 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1002 for host cloudvirt1063.eqiad.wmnet with OS bookworm [16:35:21] !log andrew@cloudcumin1001 wikiwho START - Cookbook wmcs.openstack.migrate_project_to_ovs [16:35:29] !log andrew@cloudcumin1001 wikiwho END (FAIL) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=1) [16:35:48] !log andrew@cloudcumin1001 wildcat START - Cookbook wmcs.openstack.migrate_project_to_ovs [16:37:10] !log andrew@cloudcumin1001 wildcat END (PASS) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=0) [16:37:14] !log andrew@cloudcumin1001 wlm-it-visual START - Cookbook wmcs.openstack.migrate_project_to_ovs [16:38:37] !log andrew@cloudcumin1001 wlm-it-visual END (PASS) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=0) [16:40:10] !log andrew@cloudcumin1001 wm-bot START - Cookbook wmcs.openstack.migrate_project_to_ovs [16:40:20] !log andrew@cloudcumin1001 wm-bot END (FAIL) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=1) [16:40:29] !log andrew@cloudcumin1001 wmcs-uptime START - Cookbook wmcs.openstack.migrate_project_to_ovs [16:40:32] !log andrew@cloudcumin1001 wmcs-uptime END (PASS) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=0) [16:40:40] !log andrew@cloudcumin1001 wmcz-stats START - Cookbook wmcs.openstack.migrate_project_to_ovs [16:42:59] !log andrew@cloudcumin1001 wmcz-stats END (PASS) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=0) [16:43:06] !log andrew@cloudcumin1001 wmdeanalytics START - Cookbook wmcs.openstack.migrate_project_to_ovs [16:48:16] !log andrew@cloudcumin1001 wmdeanalytics END (ERROR) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=97) [16:48:18] !log andrew@cloudcumin1001 wmdeanalytics START - Cookbook wmcs.openstack.migrate_project_to_ovs [16:49:40] !log andrew@cloudcumin1001 wmdeanalytics END (PASS) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=0) [16:49:45] !log andrew@cloudcumin1001 wmf-dumps-playground START - Cookbook wmcs.openstack.migrate_project_to_ovs [16:49:48] !log andrew@cloudcumin1001 wmf-dumps-playground END (PASS) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=0) [16:50:02] !log andrew@cloudcumin1001 wmf-research-tools START - Cookbook wmcs.openstack.migrate_project_to_ovs [16:56:55] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Data-Services: [wikireplicas] frequent replag spikes in clouddb1017 (s1) - https://phabricator.wikimedia.org/T367778#9910881 (10fnegri) 05In progress→03Resolved > it was agreed that it was a best effort and it was never guaranteed the hosts would have 0 lag.... [17:05:41] 10Quarry: [bug] Quarry queries not completing - https://phabricator.wikimedia.org/T367464#9910928 (10fnegri) @Liz I'm sorry that you're still having issues, I suspect that sometimes your queries take a bit longer to complete, and when that happens you run into the `ConnectionResetError` described above. The tim... [17:07:23] !log andrew@cloudcumin1001 wmf-research-tools END (FAIL) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=1) [17:07:54] !log andrew@cloudcumin1001 wmflabsdotorg START - Cookbook wmcs.openstack.migrate_project_to_ovs [17:07:57] !log andrew@cloudcumin1001 wmflabsdotorg END (PASS) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=0) [17:08:01] !log andrew@cloudcumin1001 xtools START - Cookbook wmcs.openstack.migrate_project_to_ovs [17:08:04] !log andrew@cloudcumin1001 xtools END (PASS) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=0) [17:11:27] !log andrew@cloudcumin1001 cloudvirt-canary START - Cookbook wmcs.openstack.cloudvirt.lib.ensure_canary on eqiad1, with recreate False, for hosts list: ['cloudvirt1063'] [17:11:49] !log andrew@cloudcumin1001 cloudvirt-canary END (PASS) - Cookbook wmcs.openstack.cloudvirt.lib.ensure_canary (exit_code=0) on eqiad1, with recreate False, for hosts list: ['cloudvirt1063'] [17:13:54] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1053.eqiad.wmnet' [17:15:11] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Cloud-VPS: Migrate eqiad1 hypervisors to Neutron OVS agent - https://phabricator.wikimedia.org/T364457#9910977 (10Andrew) [17:15:35] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Cloud-VPS: Migrate eqiad1 hypervisors to Neutron OVS agent - https://phabricator.wikimedia.org/T364457#9910979 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1002 for host cloudvirt1063.eqiad.wmnet with OS bookworm completed: - cl... [17:18:57] FIRING: CloudVPSDesignateLeaks: Detected 3 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:24:57] (03CR) 10Andrew Bogott: [C:04-2] "I would expect this to be a separate project rather than co-mingled with striker -- among other things, I think we only want to deploy th" [labs/striker] - 10https://gerrit.wikimedia.org/r/1035718 (https://phabricator.wikimedia.org/T362318) (owner: 10Slyngshede) [17:27:38] !log andrew@cloudcumin1001 cloudvirt-canary START - Cookbook wmcs.openstack.cloudvirt.lib.ensure_canary on eqiad1, with recreate False, for hosts list: ['cloudvirt1053'] [17:27:57] !log andrew@cloudcumin1001 cloudvirt-canary END (PASS) - Cookbook wmcs.openstack.cloudvirt.lib.ensure_canary (exit_code=0) on eqiad1, with recreate False, for hosts list: ['cloudvirt1053'] [17:28:49] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1053.eqiad.wmnet' [17:30:29] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Cloud-VPS, 13Patch-For-Review: Migrate eqiad1 hypervisors to Neutron OVS agent - https://phabricator.wikimedia.org/T364457#9911033 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1002 for host cloudvirt1053.eqiad.wmnet with O... [17:38:26] 06cloud-services-team, 06DC-Ops, 10ops-eqiad: reapply thermal paste to processors in cloudvirt1063 - https://phabricator.wikimedia.org/T368093 (10Andrew) 03NEW [17:57:59] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Cloud-VPS, 06DC-Ops, 10ops-eqiad, 06SRE: cloudcephosd1021-1034: hard drive sector errors increasing - https://phabricator.wikimedia.org/T348643#9911180 (10wiki_willy) During my call with the Dell Account team today, I asked them to push on this a bit more. Th... [18:21:13] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Cloud-VPS: Migrate eqiad1 hypervisors to Neutron OVS agent - https://phabricator.wikimedia.org/T364457#9911240 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1002 for host cloudvirt1053.eqiad.wmnet with OS bookworm completed: - cl... [18:23:38] !log andrew@cloudcumin1001 cloudvirt-canary START - Cookbook wmcs.openstack.cloudvirt.lib.ensure_canary on eqiad1, with recreate False, for hosts list: ['cloudvirt1053'] [18:23:45] !log andrew@cloudcumin1001 cloudvirt-canary END (PASS) - Cookbook wmcs.openstack.cloudvirt.lib.ensure_canary (exit_code=0) on eqiad1, with recreate False, for hosts list: ['cloudvirt1053'] [18:50:25] !log andrew@cloudcumin1001 testlabs START - Cookbook wmcs.openstack.migrate_database_instance_to_ovs [18:50:25] !log andrew@cloudcumin1001 testlabs END (FAIL) - Cookbook wmcs.openstack.migrate_database_instance_to_ovs (exit_code=99) [18:51:47] !log andrew@cloudcumin1001 testlabs START - Cookbook wmcs.openstack.migrate_database_instance_to_ovs [18:51:48] !log andrew@cloudcumin1001 testlabs END (FAIL) - Cookbook wmcs.openstack.migrate_database_instance_to_ovs (exit_code=99) [18:51:59] !log andrew@cloudcumin1001 testlabs START - Cookbook wmcs.openstack.migrate_database_instance_to_ovs [18:52:05] !log andrew@cloudcumin1001 testlabs END (FAIL) - Cookbook wmcs.openstack.migrate_database_instance_to_ovs (exit_code=99) [18:52:43] !log andrew@cloudcumin1001 testlabs START - Cookbook wmcs.openstack.migrate_database_instance_to_ovs [18:52:50] !log andrew@cloudcumin1001 testlabs END (FAIL) - Cookbook wmcs.openstack.migrate_database_instance_to_ovs (exit_code=99) [18:53:07] !log andrew@cloudcumin1001 testlabs START - Cookbook wmcs.openstack.migrate_database_instance_to_ovs [18:53:15] !log andrew@cloudcumin1001 testlabs END (FAIL) - Cookbook wmcs.openstack.migrate_database_instance_to_ovs (exit_code=99) [18:53:59] !log andrew@cloudcumin1001 testlabs START - Cookbook wmcs.openstack.migrate_database_instance_to_ovs [18:54:05] !log andrew@cloudcumin1001 testlabs END (FAIL) - Cookbook wmcs.openstack.migrate_database_instance_to_ovs (exit_code=99) [18:54:50] FIRING: NeutronAgentDown: Neutron neutron-linuxbridge-agent on cloudvirt1063 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [18:55:30] !log andrew@cloudcumin1001 testlabs START - Cookbook wmcs.openstack.migrate_database_instance_to_ovs [18:55:50] FIRING: NeutronAgentDownForLong: Neutron neutron-linuxbridge-agent on cloudvirt1063 has been down for more than 2h - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDownForLong [18:55:56] 06cloud-services-team: NeutronAgentDownForLong A Neutron agent has been down for more than 2h, VMs will have connectivity issues - https://phabricator.wikimedia.org/T365461#9911407 (10phaultfinder) [18:56:46] !log andrew@cloudcumin1001 testlabs END (FAIL) - Cookbook wmcs.openstack.migrate_database_instance_to_ovs (exit_code=99) [18:57:08] !log andrew@cloudcumin1001 testlabs START - Cookbook wmcs.openstack.migrate_database_instance_to_ovs [18:57:17] !log andrew@cloudcumin1001 testlabs END (FAIL) - Cookbook wmcs.openstack.migrate_database_instance_to_ovs (exit_code=99) [19:01:17] !log andrew@cloudcumin1001 testlabs START - Cookbook wmcs.openstack.migrate_database_instance_to_ovs [19:01:23] !log andrew@cloudcumin1001 testlabs END (FAIL) - Cookbook wmcs.openstack.migrate_database_instance_to_ovs (exit_code=99) [19:01:43] !log andrew@cloudcumin1001 testlabs START - Cookbook wmcs.openstack.migrate_database_instance_to_ovs [19:01:49] !log andrew@cloudcumin1001 testlabs END (PASS) - Cookbook wmcs.openstack.migrate_database_instance_to_ovs (exit_code=0) [19:05:19] 10VPS-project-Wikistats: Add btmwiki to wikistats - https://phabricator.wikimedia.org/T368071#9911451 (10Dzahn) 05Open→03Stalled stalled until T368038 is resolved. ideally these would only be created once the actual parent ticket is resolved. [19:10:51] !log andrew@cloudcumin1001 testlabs START - Cookbook wmcs.openstack.migrate_database_instance_to_ovs [19:12:12] !log andrew@cloudcumin1001 testlabs END (FAIL) - Cookbook wmcs.openstack.migrate_database_instance_to_ovs (exit_code=99) [19:15:22] !log andrew@cloudcumin1001 testlabs START - Cookbook wmcs.openstack.migrate_database_instance_to_ovs [19:15:31] !log andrew@cloudcumin1001 testlabs END (PASS) - Cookbook wmcs.openstack.migrate_database_instance_to_ovs (exit_code=0) [19:34:40] !log andrew@cloudcumin1001 testlabs START - Cookbook wmcs.openstack.migrate_database_instance_to_ovs [19:36:01] !log andrew@cloudcumin1001 testlabs END (FAIL) - Cookbook wmcs.openstack.migrate_database_instance_to_ovs (exit_code=99) [19:39:12] !log andrew@cloudcumin1001 testlabs START - Cookbook wmcs.openstack.migrate_database_instance_to_ovs for server None [19:39:12] !log andrew@cloudcumin1001 testlabs END (FAIL) - Cookbook wmcs.openstack.migrate_database_instance_to_ovs (exit_code=99) for server None [19:39:54] !log andrew@cloudcumin1001 testlabs START - Cookbook wmcs.openstack.migrate_database_instance_to_ovs for server None [19:40:00] !log andrew@cloudcumin1001 testlabs END (ERROR) - Cookbook wmcs.openstack.migrate_database_instance_to_ovs (exit_code=97) for server None [19:40:15] !log andrew@cloudcumin1001 testlabs START - Cookbook wmcs.openstack.migrate_database_instance_to_ovs for server None [19:40:22] !log andrew@cloudcumin1001 testlabs END (FAIL) - Cookbook wmcs.openstack.migrate_database_instance_to_ovs (exit_code=99) for server None [19:41:11] !log andrew@cloudcumin1001 testlabs START - Cookbook wmcs.openstack.migrate_database_instance_to_ovs for server None [19:41:17] !log andrew@cloudcumin1001 testlabs END (FAIL) - Cookbook wmcs.openstack.migrate_database_instance_to_ovs (exit_code=99) for server None [19:41:55] !log andrew@cloudcumin1001 testlabs START - Cookbook wmcs.openstack.migrate_database_instance_to_ovs for server None [19:41:59] !log andrew@cloudcumin1001 testlabs END (FAIL) - Cookbook wmcs.openstack.migrate_database_instance_to_ovs (exit_code=99) for server None [19:42:44] !log andrew@cloudcumin1001 testlabs START - Cookbook wmcs.openstack.migrate_database_instance_to_ovs for server None [19:43:03] !log andrew@cloudcumin1001 testlabs END (FAIL) - Cookbook wmcs.openstack.migrate_database_instance_to_ovs (exit_code=99) for server None [19:43:47] !log andrew@cloudcumin1001 testlabs START - Cookbook wmcs.openstack.migrate_database_instance_to_ovs for server None [19:44:07] !log andrew@cloudcumin1001 testlabs END (FAIL) - Cookbook wmcs.openstack.migrate_database_instance_to_ovs (exit_code=99) for server None [19:45:12] !log andrew@cloudcumin1001 testlabs START - Cookbook wmcs.openstack.migrate_database_instance_to_ovs for server None [19:45:35] !log andrew@cloudcumin1001 testlabs END (FAIL) - Cookbook wmcs.openstack.migrate_database_instance_to_ovs (exit_code=99) for server None [19:47:27] !log andrew@cloudcumin1001 testlabs START - Cookbook wmcs.openstack.migrate_database_instance_to_ovs for server None [19:47:51] !log andrew@cloudcumin1001 testlabs END (FAIL) - Cookbook wmcs.openstack.migrate_database_instance_to_ovs (exit_code=99) for server None [19:50:44] !log andrew@cloudcumin1001 testlabs START - Cookbook wmcs.openstack.migrate_database_instance_to_ovs for server None [19:54:44] FIRING: InterfaceSpeedError: brq7425e328-56 on cloudvirt1053:9100 has the wrong speed: 1.25e+06. - https://wikitech.wikimedia.org/wiki/Monitoring/check_eth - https://grafana.wikimedia.org/d/000000562 - https://alerts.wikimedia.org/?q=alertname%3DInterfaceSpeedError [19:54:49] 06cloud-services-team: InterfaceSpeedError brq7425e328-56 on cloudvirt1053:9100 has the wrong speed: 1.25e+06. - https://phabricator.wikimedia.org/T368105 (10phaultfinder) 03NEW [19:56:18] !log andrew@cloudcumin1001 testlabs END (ERROR) - Cookbook wmcs.openstack.migrate_database_instance_to_ovs (exit_code=97) for server None [19:58:07] !log andrew@cloudcumin1001 cloudvirt-canary START - Cookbook wmcs.openstack.cloudvirt.lib.ensure_canary on eqiad1, with recreate False, for hosts list: ['cloudvirt1053'] [19:58:11] !log andrew@cloudcumin1001 cloudvirt-canary END (PASS) - Cookbook wmcs.openstack.cloudvirt.lib.ensure_canary (exit_code=0) on eqiad1, with recreate False, for hosts list: ['cloudvirt1053'] [20:01:26] 10MediaWiki-extensions-OpenStackManager, 06Diffusion-Repository-Administrators, 10Projects-Cleanup, 06translatewiki.net, 10Wikimedia-GitHub: Archive the OpenStackManager extension - https://phabricator.wikimedia.org/T367220#9911664 (10Jdforrester-WMF) Time to proceed with this? [20:02:05] 10MediaWiki-extensions-OpenStackManager, 06Diffusion-Repository-Administrators, 10Projects-Cleanup, 06translatewiki.net, 10Wikimedia-GitHub: Archive the OpenStackManager extension - https://phabricator.wikimedia.org/T367220#9911668 (10taavi) Yes. [20:11:30] !log andrew@cloudcumin1001 testlabs START - Cookbook wmcs.openstack.migrate_database_instance_to_ovs for server None [20:15:10] !log andrew@cloudcumin1001 testlabs END (ERROR) - Cookbook wmcs.openstack.migrate_database_instance_to_ovs (exit_code=97) for server None [20:15:28] !log andrew@cloudcumin1001 testlabs START - Cookbook wmcs.openstack.migrate_database_instance_to_ovs for server None [20:19:46] !log andrew@cloudcumin1001 testlabs END (FAIL) - Cookbook wmcs.openstack.migrate_database_instance_to_ovs (exit_code=99) for server None [20:19:50] RESOLVED: NeutronAgentDown: Neutron neutron-linuxbridge-agent on cloudvirt1063 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [20:20:50] RESOLVED: NeutronAgentDownForLong: Neutron neutron-linuxbridge-agent on cloudvirt1063 has been down for more than 2h - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDownForLong [20:25:11] !log andrew@cloudcumin1001 testlabs START - Cookbook wmcs.openstack.migrate_database_instance_to_ovs for server tbd [20:29:41] !log andrew@cloudcumin1001 testlabs END (FAIL) - Cookbook wmcs.openstack.migrate_database_instance_to_ovs (exit_code=99) for server tbd [20:41:59] !log andrew@cloudcumin1001 testlabs START - Cookbook wmcs.openstack.migrate_database_instance_to_ovs for server tbd [20:46:25] !log andrew@cloudcumin1001 testlabs END (FAIL) - Cookbook wmcs.openstack.migrate_database_instance_to_ovs (exit_code=99) for server tbd [20:53:39] !log andrew@cloudcumin1001 testlabs START - Cookbook wmcs.openstack.migrate_database_instance_to_ovs for server tbd [20:57:59] !log andrew@cloudcumin1001 testlabs END (PASS) - Cookbook wmcs.openstack.migrate_database_instance_to_ovs (exit_code=0) for server tbd [20:59:22] !log andrew@cloudcumin1001 testlabs START - Cookbook wmcs.openstack.migrate_database_instance_to_ovs for server tbd [21:03:45] !log andrew@cloudcumin1001 testlabs END (PASS) - Cookbook wmcs.openstack.migrate_database_instance_to_ovs (exit_code=0) for server tbd [21:03:57] (03PS1) 10Andrew Bogott: openstack api: clarify that server_show takes a name or an ID [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1048084 [21:03:57] (03PS1) 10Andrew Bogott: Add cookbook to migrate a database instance [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1048085 [21:06:52] (03CR) 10CI reject: [V:04-1] openstack api: clarify that server_show takes a name or an ID [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1048084 (owner: 10Andrew Bogott) [21:07:04] (03CR) 10CI reject: [V:04-1] Add cookbook to migrate a database instance [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1048085 (owner: 10Andrew Bogott) [21:08:23] FIRING: OOM: OOM killer active on cloudcontrol2006-dev:9100 - TODO - https://grafana.wikimedia.org/d/-OcleDKIz/oom-kill - https://alerts.wikimedia.org/?q=alertname%3DOOM [21:13:23] RESOLVED: OOM: OOM killer active on cloudcontrol2006-dev:9100 - TODO - https://grafana.wikimedia.org/d/-OcleDKIz/oom-kill - https://alerts.wikimedia.org/?q=alertname%3DOOM [21:14:32] (03PS2) 10Andrew Bogott: openstack api: clarify that server_show takes a name or an ID [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1048084 [21:14:32] (03PS2) 10Andrew Bogott: Add cookbook to migrate a database instance [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1048085 [21:17:26] (03CR) 10CI reject: [V:04-1] Add cookbook to migrate a database instance [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1048085 (owner: 10Andrew Bogott) [21:18:57] FIRING: CloudVPSDesignateLeaks: Detected 3 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [21:21:16] (03PS3) 10Andrew Bogott: Add cookbook to migrate a database instance [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1048085 [21:23:22] !log andrew@cloudcumin1001 testlabs START - Cookbook wmcs.openstack.migrate_dbinstance_to_ovs for server tbd [21:23:26] !log andrew@cloudcumin1001 testlabs END (FAIL) - Cookbook wmcs.openstack.migrate_dbinstance_to_ovs (exit_code=99) for server tbd [21:23:41] !log andrew@cloudcumin1001 testlabs START - Cookbook wmcs.openstack.migrate_dbinstance_to_ovs for server tbd [21:24:19] (03CR) 10CI reject: [V:04-1] Add cookbook to migrate a database instance [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1048085 (owner: 10Andrew Bogott) [21:26:26] (03PS4) 10Andrew Bogott: Add cookbook to migrate a database instance [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1048085 [21:28:00] !log andrew@cloudcumin1001 testlabs END (PASS) - Cookbook wmcs.openstack.migrate_dbinstance_to_ovs (exit_code=0) for server tbd [21:29:20] (03CR) 10CI reject: [V:04-1] Add cookbook to migrate a database instance [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1048085 (owner: 10Andrew Bogott) [21:30:48] (03PS5) 10Andrew Bogott: Add cookbook to migrate a database instance [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1048085 [21:38:29] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 14789ac1-bc06-4677-9bb0-66c16c887427 [21:38:33] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server 14789ac1-bc06-4677-9bb0-66c16c887427 [21:46:28] (03open) 10andrew: Add another integration-specific flavor [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/7 [23:03:34] (03open) 10raymond-ndibe: d/changelog: bump to 0.0.8 [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/47 [23:03:39] (03approved) 10raymond-ndibe: d/changelog: bump to 0.0.8 [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/47 [23:15:20] (03merge) 10raymond-ndibe: d/changelog: bump to 0.0.8 [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/47 [23:21:19] (03update) 10raymond-ndibe: d/changelog: bump to 0.0.8 [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/47 [23:29:17] 10Data-Services, 06DBA: Prepare and check storage layer for btmwiki - https://phabricator.wikimedia.org/T368066#9912131 (10Zabe) Wiki has been created [23:45:42] 10Tool-replag: Create embeddable version of replag tool for other tools - https://phabricator.wikimedia.org/T321640#9912175 (10Legoktm) I finally got around to doing this after the high s1 replag earlier this week. Unfortunately the problem with iframes is that they don't dynamically size based on the contents (... [23:46:13] 10VPS-project-Wikistats: Add btmwiki to wikistats - https://phabricator.wikimedia.org/T368071#9912169 (10Dzahn) 05Stalled→03Resolved a:03Dzahn ` MariaDB [wikistats]> insert into wikipedias (prefix, lang, loclang, method) values ("btm", "Mandailing", "Saro Mandailing", 8); ` ` dzahn@wikistats-bookwor... [23:48:33] (03close) 10raymond-ndibe: Revert "envvars-api: bump to 0.0.50-20240619035607-42829b67" [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/340 (owner: 10taavi) [23:49:06] 10Tool-replag: Create embeddable version of replag tool for other tools - https://phabricator.wikimedia.org/T321640#9912177 (10Legoktm) a:03Legoktm [23:50:00] 10Toolforge (Toolforge iteration 11): envvars-api 0.0.50 depends on unreleased envvars-cli changes - https://phabricator.wikimedia.org/T367961#9912179 (10Raymond_Ndibe) 05Open→03In progress [23:50:03] 10Toolforge (Toolforge iteration 11): envvars-api 0.0.50 depends on unreleased envvars-cli changes - https://phabricator.wikimedia.org/T367961#9912181 (10Raymond_Ndibe) 05In progress→03Resolved [23:54:59] FIRING: InterfaceSpeedError: brq7425e328-56 on cloudvirt1053:9100 has the wrong speed: 1.25e+06. - https://wikitech.wikimedia.org/wiki/Monitoring/check_eth - https://grafana.wikimedia.org/d/000000562 - https://alerts.wikimedia.org/?q=alertname%3DInterfaceSpeedError