[01:21:30] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [03:18:58] FIRING: SystemdUnitDown: The service unit opentofu-infra-diff.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [05:13:58] FIRING: SystemdUnitDown: The systemd unit opentofu-infra-diff.service on node cloudcontrol1007 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [05:21:30] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [07:34:18] 06cloud-services-team, 10Data-Services: Add 'centralauth' to meta_p.wiki so that apps can re-use the appropriate slice - https://phabricator.wikimedia.org/T186675#10264520 (10taavi) [07:34:22] 06cloud-services-team, 10Toolforge: [toolforge] Automate addition/removal of control node - https://phabricator.wikimedia.org/T274499#10264523 (10taavi) 05Open→03Resolved a:03taavi [07:34:26] 06cloud-services-team, 10Cloud-VPS, 10MediaWiki-Vagrant, 07Documentation: Improve content design of: https://wikitech.wikimedia.org/wiki/Help:MediaWiki-Vagrant_in_Cloud_VPS - https://phabricator.wikimedia.org/T245083#10264521 (10taavi) [09:13:58] FIRING: SystemdUnitDown: The systemd unit opentofu-infra-diff.service on node cloudcontrol1007 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [09:21:30] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [10:52:20] 10wikitech.wikimedia.org, 06SRE: Redesign wikitech-static - https://phabricator.wikimedia.org/T376400#10264635 (10Peachey88) [11:07:47] (03PS1) 10Majavah: tools: membership: Post note to SUL user talk page [labs/striker] - 10https://gerrit.wikimedia.org/r/1083302 [11:07:47] (03PS1) 10Majavah: tools: membership: Link to ldap tool instead of wikitech [labs/striker] - 10https://gerrit.wikimedia.org/r/1083303 [11:32:35] 10wikitech.wikimedia.org: New e-mail-created wikitechwiki user "Per Magnus" can't set their password - https://phabricator.wikimedia.org/T178417#10264667 (10taavi) 05Open→03Invalid Assuming this is no longer a problem after the SUL migration. Please re-open if it is. [11:38:10] 10wikitech.wikimedia.org: Retire labtestwiki - https://phabricator.wikimedia.org/T378260 (10taavi) 03NEW [11:38:26] 10wikitech.wikimedia.org: Retire labtestwiki - https://phabricator.wikimedia.org/T378260#10264685 (10taavi) [11:38:30] 06cloud-services-team, 10Cloud-VPS, 06Data-Persistence: Decommission clouddb2002-dev.codfw.wmnet - https://phabricator.wikimedia.org/T369308#10264687 (10taavi) [13:06:49] (03update) 10raymond-ndibe: [maintain-harbor] Move to become a toolforge component [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/34 (https://phabricator.wikimedia.org/T358225) [13:13:58] FIRING: SystemdUnitDown: The systemd unit opentofu-infra-diff.service on node cloudcontrol1007 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [13:21:31] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:41:18] 10wikitech.wikimedia.org, 13Patch-For-Review, 10Wiki-Setup (Delete / Redirect): Retire labtestwiki - https://phabricator.wikimedia.org/T378260#10264778 (10Bugreporter) [15:13:49] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10264842 (10Robertsky) @Ladsgroup I can't log into the wikitech account which makes it a nonstarter (as somehow the password for wikitech wiki was lost in my password manager) to edit in wiki... [17:13:58] FIRING: SystemdUnitDown: The systemd unit opentofu-infra-diff.service on node cloudcontrol1007 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [17:21:31] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [20:27:28] 10Tool-wlh: Expose pagination in URL params - https://phabricator.wikimedia.org/T336317#10265162 (10egardner) a:05egardner→03None [20:28:14] 10Tool-wlh: Multi-word titles fail - https://phabricator.wikimedia.org/T325983#10265167 (10egardner) a:05egardner→03None [21:13:59] FIRING: SystemdUnitDown: The systemd unit opentofu-infra-diff.service on node cloudcontrol1007 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [21:21:31] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [22:47:37] FIRING: PuppetCertificateAboutToExpire: Puppet CA certificate mwv-builder-03.mediawiki-vagrant.eqiad.wmflabs is about to expire in -1d 0h 1m 25s - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetCertificateAboutToExpire - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire