[01:01:42] 10Striker, 10Bitu, 06Infrastructure-Foundations, 07Security: Forms for adding cloud ssh keys should have a message not to add production keys - https://phabricator.wikimedia.org/T276761#10128075 (10Pppery) [01:11:56] FIRING: SystemdUnitDown: The systemd unit opentofu-infra-diff.service on node cloudcontrol1007 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [03:11:56] RESOLVED: SystemdUnitDown: The systemd unit opentofu-infra-diff.service on node cloudcontrol1007 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [03:34:47] 10Toolforge: Help needed to deploy a react tool in Toolforge - https://phabricator.wikimedia.org/T374304#10128100 (10Ederporto) Thank you for your answer @dcaro . Unfortunatelly, I was unable to make it work: I get that the image does not exists, even though I can see it something existing [[ https://tools-harbo... [03:50:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [04:00:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [04:06:56] FIRING: SystemdUnitDown: The service unit purge_vm_rbd_images.service is in failed status on host cloudcontrol1005. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1005 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [06:01:56] FIRING: SystemdUnitDown: The systemd unit purge_vm_rbd_images.service on node cloudcontrol1005 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1005 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [06:02:02] 06cloud-services-team: SystemdUnitDown Unit purge_vm_rbd_images.service on node cloudcontrol1005 has been down for long. - https://phabricator.wikimedia.org/T374313 (10phaultfinder) 03NEW [06:20:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:30:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:55:05] 06cloud-services-team, 10Cloud-VPS: openstack: fix missing prometheus metrics - https://phabricator.wikimedia.org/T373878#10128134 (10taavi) [06:56:41] 06cloud-services-team, 10Cloud-VPS, 07Epic: tofu-infra: investigate S3 spurious endpoint errors - https://phabricator.wikimedia.org/T370660#10128136 (10taavi) →14Duplicate dup:03T360626 [06:57:18] 10Cloud-VPS: Frequent radosgw 500 errors with OpenTofu - https://phabricator.wikimedia.org/T360626#10128138 (10taavi) [08:45:55] 10Cloud-VPS (Debian Buster Deprecation), 10Wikispore: Rebuild Wikispore Vagrant boxes on Bullseye or Bookworm - https://phabricator.wikimedia.org/T365934#10128216 (10Samwilson) It seems like the lack of Vagrant skills in the Wikispore technical community was part of the issue here. I'm going to boot up `wikis... [09:20:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [10:01:56] FIRING: SystemdUnitDown: The systemd unit purge_vm_rbd_images.service on node cloudcontrol1005 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1005 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [12:35:23] (03open) 10hunsvotti: Add more documentation [toolforge-repos/imgs-for-wikt] - 10https://gitlab.wikimedia.org/toolforge-repos/imgs-for-wikt/-/merge_requests/4 [12:35:29] (03merge) 10hunsvotti: Add more documentation [toolforge-repos/imgs-for-wikt] - 10https://gitlab.wikimedia.org/toolforge-repos/imgs-for-wikt/-/merge_requests/4 [12:47:54] (03open) 10hunsvotti: Separate English and default locales, to make the edit link work on English witktionary too [toolforge-repos/imgs-for-wikt] - 10https://gitlab.wikimedia.org/toolforge-repos/imgs-for-wikt/-/merge_requests/5 [12:48:02] (03merge) 10hunsvotti: Separate English and default locales, to make the edit link work on English witktionary too [toolforge-repos/imgs-for-wikt] - 10https://gitlab.wikimedia.org/toolforge-repos/imgs-for-wikt/-/merge_requests/5 [13:05:28] 10Cloud-VPS (Debian Buster Deprecation), 10Wikispore: Rebuild Wikispore Vagrant boxes on Bullseye or Bookworm - https://phabricator.wikimedia.org/T365934#10128305 (10Samwilson) a:05Tgr→03Samwilson I started a new repo for the new server's config: https://gitlab.wikimedia.org/toolforge-repos/wikispore-confi... [13:16:25] 10Toolforge: Help needed to deploy a react tool in Toolforge - https://phabricator.wikimedia.org/T374304#10128311 (10dcaro) Oh, yes sorry, the right toolforge jobs command to run is: ` tools.infographics@tools-bastion-13:~$ toolforge jobs run --command "backend" --image tool-infographics/backend:latest... [13:20:41] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [14:01:57] FIRING: SystemdUnitDown: The systemd unit purge_vm_rbd_images.service on node cloudcontrol1005 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1005 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [15:32:19] 10Cloud-VPS (Debian Buster Deprecation), 10linkwatcher: Cloud VPS "linkwatcher" project Buster deprecation - https://phabricator.wikimedia.org/T367536#10128340 (10Beetstra) I have moved the bots to two new instances, using bookworm, and re-mounted volumes for their data. Old volumes and old instances are dele... [15:39:29] 10Cloud-VPS (Debian Buster Deprecation), 10Wikispore: Rebuild Wikispore Vagrant boxes on Bullseye or Bookworm - https://phabricator.wikimedia.org/T365934#10128342 (10Pppery) (that would be T322991) [16:36:12] (03open) 10hunsvotti: Localization instructions to readme [toolforge-repos/imgs-for-wikt] - 10https://gitlab.wikimedia.org/toolforge-repos/imgs-for-wikt/-/merge_requests/6 [16:36:27] (03merge) 10hunsvotti: Localization instructions to readme [toolforge-repos/imgs-for-wikt] - 10https://gitlab.wikimedia.org/toolforge-repos/imgs-for-wikt/-/merge_requests/6 [16:51:05] (03Abandoned) 10GauriGuptaa: Fix the typo error from one to on [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1059930 (owner: 10GauriGuptaa) [17:20:41] FIRING: CloudVPSDesignateLeaks: Detected 5 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:56:42] (03open) 10hunsvotti: Add webhook to download code from main branch [toolforge-repos/imgs-for-wikt] - 10https://gitlab.wikimedia.org/toolforge-repos/imgs-for-wikt/-/merge_requests/7 [17:56:48] (03merge) 10hunsvotti: Add webhook to download code from main branch [toolforge-repos/imgs-for-wikt] - 10https://gitlab.wikimedia.org/toolforge-repos/imgs-for-wikt/-/merge_requests/7 [18:01:56] FIRING: SystemdUnitDown: The systemd unit purge_vm_rbd_images.service on node cloudcontrol1005 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1005 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [18:04:49] (03open) 10hunsvotti: Fix error on previous commit [toolforge-repos/imgs-for-wikt] - 10https://gitlab.wikimedia.org/toolforge-repos/imgs-for-wikt/-/merge_requests/8 [18:04:54] (03merge) 10hunsvotti: Fix error on previous commit [toolforge-repos/imgs-for-wikt] - 10https://gitlab.wikimedia.org/toolforge-repos/imgs-for-wikt/-/merge_requests/8 [21:20:41] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [22:01:57] FIRING: SystemdUnitDown: The systemd unit purge_vm_rbd_images.service on node cloudcontrol1005 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1005 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown