[00:11:25] FIRING: TfInfraTestApplyFailed: Terraform failed to apply/create the resources on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [01:12:57] FIRING: [3x] CloudVPSDesignateLeaks: Detected 19 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [01:33:06] 10Cloud-VPS (Debian Buster Deprecation), 06Community-Tech, 10IA Upload, 10Wikimedia OCR: Cloud VPS "wikisource" project Buster deprecation - https://phabricator.wikimedia.org/T367564#9895125 (10Samwilson) [02:32:38] 10Cloud-VPS (Debian Buster Deprecation), 06Community-Tech, 10IA Upload, 10Wikimedia OCR: Cloud VPS "wikisource" project Buster deprecation - https://phabricator.wikimedia.org/T367564#9895143 (10Samwilson) The OCR upgrade is tracked in T332611. IA Upload can be done, but there's also the possibility of rep... [03:47:25] (03open) 10legoktm: Bridge #wikimedia-rust IRC and Matrix [toolforge-repos/bridgebot] - 10https://gitlab.wikimedia.org/toolforge-repos/bridgebot/-/merge_requests/5 (https://phabricator.wikimedia.org/T366767) [05:12:57] FIRING: [3x] CloudVPSDesignateLeaks: Detected 19 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:35:24] 10Quarry: [bug] Quarry queries not completing - https://phabricator.wikimedia.org/T367464#9895245 (10Wurgl) I tried https://quarry.wmcloud.org/query/83617 multiple times since T366909 was closed, and it never finished. What can a simple user do to prevent running into this problem? Starting again and again does... [06:51:29] 10Tool-Pageviews, 10PageViewInfo: Time period cannot be selected on pageviews.wmcloud.org on mobile - https://phabricator.wikimedia.org/T367626 (10MBH) 03NEW [09:12:57] FIRING: [3x] CloudVPSDesignateLeaks: Detected 22 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [10:47:40] 10Quarry: github action building main rather than branch - https://phabricator.wikimedia.org/T367630 (10rook) 03NEW [10:48:51] vivian-rook closed https://github.com/toolforge/quarry/pull/50 [11:28:38] 10tool-wscontest, 06All-and-every-Wikisource, 07Accessibility, 07I18n: [[Wikimedia:Wscontest-click-here-link/en]] accessibility issue - https://phabricator.wikimedia.org/T367634#9895432 (10Peachey88) [11:36:32] 10tool-wscontest, 07Accessibility, 07Voice & Tone: [[Wikimedia:Wscontest-click-here-link/en]] accessibility issue - https://phabricator.wikimedia.org/T367634#9895436 (10Aklapper) [12:56:48] FIRING: PuppetConstantChange: Puppet performing a change on every puppet run on cloudcontrol2005-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [13:11:48] FIRING: [2x] PuppetConstantChange: Puppet performing a change on every puppet run on cloudcontrol2005-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [13:12:57] FIRING: [3x] CloudVPSDesignateLeaks: Detected 21 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:16:48] FIRING: [3x] PuppetConstantChange: Puppet performing a change on every puppet run on cloudcontrol2004-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [14:00:03] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Kubernetes worker tools-k8s-worker-nfs-35 has many processes stuck on IO (probably NFS) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [17:12:57] FIRING: [3x] CloudVPSDesignateLeaks: Detected 22 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:17:03] FIRING: [3x] PuppetConstantChange: Puppet performing a change on every puppet run on cloudcontrol2004-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [18:02:26] (03open) 10legoktm: Remove reference to "User:Legoktm/toolforge_library" wiki page [toolforge-repos/python-toolforge] - 10https://gitlab.wikimedia.org/toolforge-repos/python-toolforge/-/merge_requests/23 [18:04:06] (03update) 10legoktm: Remove reference to "User:Legoktm/toolforge_library" wiki page [toolforge-repos/python-toolforge] - 10https://gitlab.wikimedia.org/toolforge-repos/python-toolforge/-/merge_requests/23 [18:04:11] (03update) 10legoktm: Remove reference to "User:Legoktm/toolforge_library" wiki page [toolforge-repos/python-toolforge] - 10https://gitlab.wikimedia.org/toolforge-repos/python-toolforge/-/merge_requests/23 [18:15:09] (03approved) 10lucaswerkmeister: Remove reference to "User:Legoktm/toolforge_library" wiki page [toolforge-repos/python-toolforge] - 10https://gitlab.wikimedia.org/toolforge-repos/python-toolforge/-/merge_requests/23 (owner: 10legoktm) [18:15:28] (03merge) 10lucaswerkmeister: Remove reference to "User:Legoktm/toolforge_library" wiki page [toolforge-repos/python-toolforge] - 10https://gitlab.wikimedia.org/toolforge-repos/python-toolforge/-/merge_requests/23 (owner: 10legoktm) [18:32:25] 10Cloud-VPS (Debian Buster Deprecation), 10Wikispeech: Cloud VPS "wikispeech" project Buster deprecation - https://phabricator.wikimedia.org/T367565#9895714 (10Lokal_Profil) @Sebastian_Berlin-WMSE Would you mind taking a look att this. Also probably worth taking a look at your settings and linking the account... [18:45:04] (03open) 10lucaswerkmeister: docs: Add assert_private_file() + load_private_yaml() [toolforge-repos/python-toolforge] - 10https://gitlab.wikimedia.org/toolforge-repos/python-toolforge/-/merge_requests/24 (https://phabricator.wikimedia.org/T333728) [21:12:57] FIRING: [3x] CloudVPSDesignateLeaks: Detected 21 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [21:17:03] FIRING: [3x] PuppetConstantChange: Puppet performing a change on every puppet run on cloudcontrol2004-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [22:51:28] FIRING: [2x] PuppetAgentNoResources: No Puppet resources found on instance cloudinfra-internal-puppetserver-1 on project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [22:56:28] FIRING: [3x] PuppetAgentNoResources: No Puppet resources found on instance cloudinfra-internal-puppetserver-1 on project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [22:56:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance cvn-nfs-1 on project cvn - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [23:04:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance gitlab-runners-puppetserver-01 on project gitlab-runners - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [23:06:28] FIRING: [3x] PuppetAgentNoResources: No Puppet resources found on instance cloudinfra-internal-puppetserver-1 on project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [23:11:28] RESOLVED: [3x] PuppetAgentNoResources: No Puppet resources found on instance cloudinfra-internal-puppetserver-1 on project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [23:11:28] RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance cvn-nfs-1 on project cvn - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [23:24:28] RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance gitlab-runners-puppetserver-01 on project gitlab-runners - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources