[00:02:42] 10PAWS: New upstream release for OpenRefine - https://phabricator.wikimedia.org/T363879#9759944 (10LibUp-bot) [00:13:46] (03PS1) 10Andrea Denisse: ssl: Delete dummy TLS key for the Grafana hosts [labs/private] - 10https://gerrit.wikimedia.org/r/1025877 (https://phabricator.wikimedia.org/T360414) [00:14:46] (03CR) 10Andrea Denisse: "This patch is to be merged after merging https://gerrit.wikimedia.org/r/c/operations/puppet/+/1025860" [labs/private] - 10https://gerrit.wikimedia.org/r/1025877 (https://phabricator.wikimedia.org/T360414) (owner: 10Andrea Denisse) [00:15:28] FIRING: InstanceDown: Project tf-infra-test instance tf-infra-test is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [00:20:28] RESOLVED: InstanceDown: Project tf-infra-test instance tf-infra-test is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [00:32:56] FIRING: [2x] SystemdUnitDown: The service unit wikitech_run_jobs.service is in failed status on host cloudweb1003. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [00:42:56] FIRING: [2x] SystemdUnitDown: The service unit wikitech_run_jobs.service is in failed status on host cloudweb1003. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [00:47:56] RESOLVED: [2x] SystemdUnitDown: The service unit wikitech_run_jobs.service is in failed status on host cloudweb1003. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [01:05:02] (03PS1) 10LWatson: releases: Bump Codex to 1.5.0 [labs/libraryupgrader/config] - 10https://gerrit.wikimedia.org/r/1025881 [01:08:41] (03CR) 10Catrope: [C:03+2] releases: Bump Codex to 1.5.0 [labs/libraryupgrader/config] - 10https://gerrit.wikimedia.org/r/1025881 (owner: 10LWatson) [01:09:17] (03Merged) 10jenkins-bot: releases: Bump Codex to 1.5.0 [labs/libraryupgrader/config] - 10https://gerrit.wikimedia.org/r/1025881 (owner: 10LWatson) [02:12:41] FIRING: [2x] CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [02:17:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [02:22:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [02:27:41] RESOLVED: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [02:31:34] RESOLVED: DiskSpace: Disk space cloudbackup1002-dev:9100:/srv/cinder-backups 1.758% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup1002-dev - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [02:44:38] RECOVERY - Disk space on cloudbackup1002-dev is OK: DISK OK https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space https://grafana.wikimedia.org/d/000000377/host-overview?var-server=cloudbackup1002-dev&var-datasource=eqiad+prometheus/ops [03:04:56] FIRING: [2x] SystemdUnitDown: The service unit wikitech_run_jobs.service is in failed status on host cloudweb1003. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [03:09:56] RESOLVED: [2x] SystemdUnitDown: The service unit wikitech_run_jobs.service is in failed status on host cloudweb1003. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [03:36:14] 10PAWS: gamescasino - https://phabricator.wikimedia.org/T363888 (10Soubag) 03NEW [03:39:22] 10PAWS: gamescasino - https://phabricator.wikimedia.org/T363888#9760225 (10Soubag) International payment fund 10000000$x2 [04:28:06] FIRING: [2x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_tool_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:33:06] RESOLVED: [4x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:42:41] FIRING: [2x] CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [04:47:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [04:52:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [04:57:41] RESOLVED: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [07:14:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [07:24:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [08:50:03] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Kubernetes worker tools-k8s-worker-nfs-42 has many processes stuck on IO (probably NFS) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [09:05:24] 10Cloud-VPS (Quota-requests): Project WP1.0/mwoffliner requests Trove instance with 75 GB - https://phabricator.wikimedia.org/T363901 (10Audiodude) 03NEW [09:43:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [09:53:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [10:45:32] 10Tools, 06Tech-Docs-Team, 07Documentation, 03Wikimedia-Hackathon-2024: [Hackathon 2024] Improve technical documentation of tools - https://phabricator.wikimedia.org/T358040#9760725 (10JorisDarlingtonQuarshie) https://www.mediawiki.org/wiki/Doc_Your_Tool:_Creating_user-friendly_documentation/Instructions... [11:12:41] FIRING: [2x] CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [11:17:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [11:22:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [11:27:41] RESOLVED: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:05:03] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Kubernetes worker tools-k8s-worker-nfs-42 has many processes stuck on IO (probably NFS) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [13:43:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:53:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [14:48:00] 06cloud-services-team, 10Cloud-VPS, 10Data-Services, 13Patch-For-Review: Support Trove + Swift integration - https://phabricator.wikimedia.org/T349651#9761131 (10Andrew) 05Open→03Resolved Inasmuch as Trove works for this, the integration is also working. [14:55:03] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Kubernetes worker tools-k8s-worker-nfs-42 has many processes stuck on IO (probably NFS) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [15:05:40] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Cloud-VPS: openstack: consider removing references to old hardware from the database - https://phabricator.wikimedia.org/T335978#9761181 (10Andrew) Removing hardware records from the DB seems a little bit dangerous as that could leave dangling references elsewhere (... [15:12:35] 10PAWS: New upstream release for OpenRefine - https://phabricator.wikimedia.org/T363879#9761214 (10rook) →14Duplicate dup:03T363732 [15:12:41] FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [15:12:43] 10PAWS: update openrefine - https://phabricator.wikimedia.org/T363732#9761216 (10rook) [15:21:23] 10PAWS, 10Pywikibot: add species to fam list - https://phabricator.wikimedia.org/T363738#9761242 (10rook) 05Open→03Resolved [15:22:36] 10PAWS: add species to fam list - https://phabricator.wikimedia.org/T363738#9761243 (10JJMC89) [15:22:41] FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [15:25:10] 10PAWS: Disable password based login on pwb on PAWS - https://phabricator.wikimedia.org/T144151#9761262 (10Xqt) T144157 is already solved and the file content is hidden for other users. Botpassword file or oauth token aren’t more save than the cookie file I guess. 2FA can be used together with the password which... [15:27:01] 10PAWS: Disable password based login on pwb on PAWS - https://phabricator.wikimedia.org/T144151#9761268 (10rook) 05Open→03Resolved [15:27:06] 10PAWS: Disable password based login on pwb on PAWS - https://phabricator.wikimedia.org/T144151#9761267 (10rook) >>! In T144151#9761262, @Xqt wrote: > What is left to do here? Seems like it is resolved. Can be re-opened if a hole is encountered. [15:27:41] RESOLVED: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [15:30:39] FIRING: [2x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:35:39] RESOLVED: [2x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:46:37] 10PAWS: Pod Security Policies - https://phabricator.wikimedia.org/T317787#9761336 (10rook) 05Resolved→03Open [15:52:38] 10PAWS: paws not connecting to any reconciliation services - https://phabricator.wikimedia.org/T363917 (10rook) 03NEW [15:52:42] 10PAWS: Pod Security Policies - https://phabricator.wikimedia.org/T317787#9761359 (10rook) 05Open→03Resolved [16:08:52] 10PAWS: paws not connecting to any reconciliation services - https://phabricator.wikimedia.org/T363917#9761403 (10Ponor) @rook Create a simple project with data from clipboard, like ` ID,taxon 120,Cedrus atlantica 121,Cedrus deodara 122,Cedrus libani 123,Larix decidua 124,Picea omorika 125,Pinus cembra ` Then in... [16:14:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [16:17:12] 10PAWS: paws not connecting to any reconciliation services - https://phabricator.wikimedia.org/T363917#9761413 (10github-toolforge-bot) vivian-rook opened https://github.com/toolforge/paws/pull/408 [16:17:19] vivian-rook opened https://github.com/toolforge/paws/pull/408 [16:24:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [16:27:01] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Cloud-VPS: openstack: consider removing references to old hardware from the database - https://phabricator.wikimedia.org/T335978#9761453 (10Andrew) Ok, I think I found them! These deleted hosts can be cleaned up with openstack resource provider list openstack reso... [16:36:25] 10PAWS: paws not connecting to any reconciliation services - https://phabricator.wikimedia.org/T363917#9761484 (10rook) @Ponor I've rolled back the version of OpenRefine to 3.7.9 and it seems to be working as expected, could you restart your container and verify that it is working for you again? I'm guessing th... [16:40:03] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Kubernetes worker tools-k8s-worker-nfs-42 has many processes stuck on IO (probably NFS) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [16:55:59] 10PAWS: paws not connecting to any reconciliation services - https://phabricator.wikimedia.org/T363917#9761532 (10Ponor) 3.7.9 working as expected. Thanks! @rook Hm... I'm a casual user and I can only guess what's happening behind the scenes. WikiDATA plugin? [16:57:22] (03CR) 10Andrew Bogott: [V:03+2 C:03+2] Disable backup strategies panel [openstack/horizon/trove-dashboard] (2024.1) - 10https://gerrit.wikimedia.org/r/1025841 (owner: 10Andrew Bogott) [16:57:22] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Cloud-VPS: openstack: consider removing references to old hardware from the database - https://phabricator.wikimedia.org/T335978#9761529 (10Andrew) 05Open→03Resolved I think this is now cleaned up and resolved for now. In the future, I suspect that deleting... [16:57:32] (03CR) 10Andrew Bogott: [V:03+2 C:03+2] Disable backup strategies panel [openstack/horizon/trove-dashboard] - 10https://gerrit.wikimedia.org/r/1025840 (owner: 10Andrew Bogott) [16:58:47] 10PAWS: paws not connecting to any reconciliation services - https://phabricator.wikimedia.org/T363917#9761535 (10rook) Looks like it is being covered in https://github.com/OpenRefine/OpenRefine/issues/6581 [17:00:28] FIRING: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance metricsinfra-puppetserver-1 in project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [17:03:43] 10Tool-itwiki: Aggiungere un'etichetta alle modifiche di JWB - https://phabricator.wikimedia.org/T363923 (10Titore) 03NEW [17:08:13] 10PAWS: paws not connecting to any reconciliation services - https://phabricator.wikimedia.org/T363917#9761565 (10Ponor) Also https://github.com/OpenRefine/OpenRefine/issues/6580 I'm guessing [17:42:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:52:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [20:12:42] FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [20:15:06] 10PAWS: paws not connecting to any reconciliation services - https://phabricator.wikimedia.org/T363917#9762071 (10github-toolforge-bot) vivian-rook closed https://github.com/toolforge/paws/pull/408 [20:15:06] 10PAWS: update openrefine - https://phabricator.wikimedia.org/T363732#9762073 (10rook) 05Resolved→03Open [20:15:14] vivian-rook closed https://github.com/toolforge/paws/pull/408 [20:15:29] 10PAWS: update openrefine - https://phabricator.wikimedia.org/T363732#9762074 (10rook) [20:15:29] 10PAWS: paws not connecting to any reconciliation services - https://phabricator.wikimedia.org/T363917#9762075 (10rook) [20:16:07] 10PAWS: update openrefine - https://phabricator.wikimedia.org/T363732#9762077 (10rook) Reopening as this appears to cause the wikimedia commons plugin to fail. T363917 describes more. [20:22:41] RESOLVED: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [21:08:00] 10superset.wmcloud.org, 07Regression: Lost saved queries - https://phabricator.wikimedia.org/T361822#9762193 (10Snaevar) I was able to find 60% of my queries by visiting "https://superset.wmcloud.org/sqllab/?savedQueryId=" urls from my browser history. I have those back through saving "a new query". Trying to... [21:22:44] 10Data-Services, 07User-notice: Remove deprecated abuse filter fields from Wiki Replicas - https://phabricator.wikimedia.org/T361996#9762235 (10Pppery) [22:10:43] 14Grid-Engine-to-K8s-Migration: Migrate parliamentdiagram from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319955#9762321 (10Gouvernathor) Hi, I'm having an issue porting a cron task we used to the new system. The task needs to remove old files, but there's a permission... [22:42:41] FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [22:47:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [22:52:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [22:57:41] RESOLVED: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks