[00:09:03] (InstanceDown) firing: Project tf-infra-test instance tf-infra-test is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [00:14:03] (InstanceDown) resolved: Project tf-infra-test instance tf-infra-test is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [00:21:30] (03CR) 10Majavah: [V: 03+2 C: 03+2] secret: dkim: move wmcs dkim keys to correct location [labs/private] - 10https://gerrit.wikimedia.org/r/969690 (owner: 10Majavah) [00:26:10] (03PS3) 10Majavah: hieradata: fix cloudinfra webproxy password location [labs/private] - 10https://gerrit.wikimedia.org/r/969689 [00:26:14] (03PS3) 10Majavah: hieradata: add fake metricsinfra grafana password [labs/private] - 10https://gerrit.wikimedia.org/r/969691 [00:35:39] (03CR) 10Majavah: [V: 03+2 C: 03+2] hieradata: fix cloudinfra webproxy password location [labs/private] - 10https://gerrit.wikimedia.org/r/969689 (owner: 10Majavah) [00:36:03] (InstanceDown) firing: Project toolsbeta instance toolsbeta-bastion-6 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [00:38:46] (03CR) 10Majavah: [V: 03+2 C: 03+2] hieradata: add fake metricsinfra grafana password [labs/private] - 10https://gerrit.wikimedia.org/r/969691 (owner: 10Majavah) [00:51:56] (ToolsToolsDBWritableState) firing: There should be exactly one writable MariaDB instance instead of 0 - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsToolsDBWritableState - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsToolsDBWritableState [00:57:03] (PuppetSyncFailure) firing: Failed to update Puppet repository /var/lib/git/labs/private on instance project-proxy-puppetmaster-01 in project project-proxy - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [01:01:56] (ToolsToolsDBWritableState) resolved: There should be exactly one writable MariaDB instance instead of 0 - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsToolsDBWritableState - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsToolsDBWritableState [01:02:03] (PuppetSyncFailure) resolved: Failed to update Puppet repository /var/lib/git/labs/private on instance project-proxy-puppetmaster-01 in project project-proxy - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [01:55:03] (TfInfraTestDestroyFailed) firing: Terraform failed to destroy the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestDestroyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestDestroyFailed [02:00:03] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [02:26:19] (HAProxyBackendUnavailable) firing: HAProxy service neutron-api_backend backend cloudcontrol1005.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [02:31:20] (HAProxyBackendUnavailable) resolved: HAProxy service neutron-api_backend backend cloudcontrol1005.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [03:05:05] 10Grid-Engine-to-K8s-Migration: Migrate botwikiawk from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319607 (10Green_Cardamom) Hi, my tools here are awk and tcsh scripts mostly, that further invoke other unix tools like sort, uniq etc.. that also depend on a library I wrote c... [03:36:03] (InstanceDown) firing: Project toolsbeta instance toolsbeta-bastion-6 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [03:43:29] (OpenstackAPIResponse) firing: (2) Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [04:55:03] (TfInfraTestDestroyFailed) firing: Terraform failed to destroy the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestDestroyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestDestroyFailed [05:00:03] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [06:36:03] (InstanceDown) firing: Project toolsbeta instance toolsbeta-bastion-6 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [07:09:56] (ToolsToolsDBReplicationError) firing: ToolsDB replication is broken on tools-db-2 (errno 2003) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBReplication - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsToolsDBReplicationError [07:09:56] (ToolsToolsDBWritableState) firing: There should be exactly one writable MariaDB instance instead of 0 - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsToolsDBWritableState - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsToolsDBWritableState [07:19:56] (ToolsToolsDBWritableState) resolved: There should be exactly one writable MariaDB instance instead of 0 - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsToolsDBWritableState - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsToolsDBWritableState [07:19:56] (ToolsToolsDBReplicationError) resolved: ToolsDB replication is broken on tools-db-2 (errno 2003) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBReplication - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsToolsDBReplicationError [07:30:36] 10cloud-services-team (FY2023/2024-Q1-Q2), 10Cloud-Services-Origin-Alert, 10Cloud-Services-Worktype-Unplanned, 10User-dcaro: [toolsdb] MariaDB process is killed by OOM killer (December 2023) - https://phabricator.wikimedia.org/T353093 (10dcaro) p:05Triage→03High [07:32:28] 10cloud-services-team (FY2023/2024-Q1-Q2), 10Cloud-Services-Origin-Alert, 10Cloud-Services-Worktype-Unplanned, 10User-dcaro: [toolsdb] MariaDB process is killed by OOM killer (December 2023) - https://phabricator.wikimedia.org/T353093 (10dcaro) Related to {T349695} and {T344298} [07:43:29] (OpenstackAPIResponse) firing: (2) Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [07:55:03] (TfInfraTestDestroyFailed) firing: Terraform failed to destroy the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestDestroyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestDestroyFailed [07:58:29] (OpenstackAPIResponse) firing: (2) Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [08:00:03] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [08:53:37] (CephSlowOps) firing: Ceph cluster in eqiad has 21 slow ops - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephSlowOps - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephSlowOps [08:53:41] 10cloud-services-team: CephSlowOps Ceph cluster in eqiad has slow ops, which might be blocking some writes - https://phabricator.wikimedia.org/T352570 (10phaultfinder) [08:58:37] (CephSlowOps) resolved: Ceph cluster in eqiad has 21 slow ops - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephSlowOps - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephSlowOps [09:33:40] 10Toolforge (Quota-requests): Request increased quota for anchor-corrector Toolforge tool - https://phabricator.wikimedia.org/T350484 (10Kanashimi) @taavi I think the reason is that wait has a timeout of 5 minutes. So it can't be used in combination for long tasks. Please help me to set up 8 Cron jobs at the sam... [09:36:03] (InstanceDown) firing: Project toolsbeta instance toolsbeta-bastion-6 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [09:57:01] 10Grid-Engine-to-K8s-Migration: Migrate uarchivebot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320104 (10Andriy.v) Hello! I'm the maintainer of the tool. I tried to performe the migration process by myself, but I have some difficulties to do it on Toolforge jobs framew... [10:04:30] 10Grid-Engine-to-K8s-Migration: Migrate redirtalkdeleter from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319998 (10Andriy.v) Hello! I'm the maintainer of the tool. I tried to performe the migration process by myself, but I have some difficulties to do it on Toolforge jobs f... [10:08:29] (OpenstackAPIResponse) resolved: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [10:12:37] 10Grid-Engine-to-K8s-Migration: Migrate resizeimagesbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320004 (10Andriy.v) Hello! I'm the maintainer of the tool. This tool is created to run a custom python script with pywikibot library and some others python libraries (inc... [10:51:55] 10Grid-Engine-to-K8s-Migration: Migrate video-cut-tool from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320123 (10Soda) cc @Gopavasanth I don't think we have anything run on these projects (ditto for {T320124}). We are currently hosted on a WMCS project. [10:55:03] (TfInfraTestDestroyFailed) firing: Terraform failed to destroy the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestDestroyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestDestroyFailed [10:55:23] 10Grid-Engine-to-K8s-Migration: Migrate unpatrollededitstats from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320105 (10bjh21) @Stang asked to adopt this tool a little over a year ago: T314167 [10:58:22] 10Grid-Engine-to-K8s-Migration: Migrate vrb from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320129 (10bjh21) @mdaniels5757 asked to adopt this tool a few months ago: T338556 [11:00:03] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [11:00:30] 10Grid-Engine-to-K8s-Migration: Migrate eatchabot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319713 (10bjh21) @mdaniels5757 asked to adopt this tool a few months ago: T338555 [11:01:57] 10Grid-Engine-to-K8s-Migration: Migrate fpcstats from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319752 (10bjh21) @mdaniels5757 asked to adopt this tools a few months ago: T338557 [11:10:42] 10Grid-Engine-to-K8s-Migration: Migrate video-cut-tool from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320123 (10Gopavasanth) Hi @Komla, @Soda Oops I somehow missed this ticket, yes we are not running our tool in Tool Forge, please proceed to delete any VideoCutTool instanc... [11:28:56] (ToolsToolsDBWritableState) firing: There should be exactly one writable MariaDB instance instead of 0 - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsToolsDBWritableState - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsToolsDBWritableState [11:53:56] (ToolsToolsDBWritableState) resolved: There should be exactly one writable MariaDB instance instead of 0 - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsToolsDBWritableState - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsToolsDBWritableState [11:59:27] (OpenstackAPIResponse) firing: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [12:36:03] (InstanceDown) firing: Project toolsbeta instance toolsbeta-bastion-6 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [13:55:03] (TfInfraTestDestroyFailed) firing: Terraform failed to destroy the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestDestroyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestDestroyFailed [14:00:03] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [15:07:16] 10Grid-Engine-to-K8s-Migration: Migrate panoviewer from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319953 (10Fuzheado) >>! In T319953#9392450, @tstarling wrote: >>>! In T319953#9392014, @Fuzheado wrote: >>Therefore I have only recently noticed that it is at risk of being de... [15:13:07] 10Grid-Engine-to-K8s-Migration: Migrate commons-maintenance-bot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319641 (10Fuzheado) FYI, this bot seems to be dormant as of September 2021. https://commons.wikimedia.org/wiki/Special:Contributions/CommonsMaintenanceBot [15:34:24] 10Toolforge-standards-committee: Adoption request for unpatrollededitstats - https://phabricator.wikimedia.org/T314167 (10Krinkle) I made a similar request in 206 at . * 15 April 2015: Sjoerd reports tool/bot is... [15:36:03] (InstanceDown) firing: Project toolsbeta instance toolsbeta-bastion-6 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [15:53:59] (PuppetConstantChange) firing: Puppet performing a change on every puppet run on cloudweb2002-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [15:59:27] (OpenstackAPIResponse) firing: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [16:13:58] 10Grid-Engine-to-K8s-Migration: Migrate dplbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319701 (10russblau) I do have some questions. For background, this tool consists of a PHP webserver that relies on a Toolsdb database, with a number of jobs (PHP scripts) run at v... [16:16:56] (ToolsToolsDBWritableState) firing: There should be exactly one writable MariaDB instance instead of 0 - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsToolsDBWritableState - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsToolsDBWritableState [16:40:56] (ToolsToolsDBReplicationError) firing: ToolsDB replication is broken on tools-db-2 (errno 2003) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBReplication - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsToolsDBReplicationError [16:43:25] 10cloud-services-team (FY2023/2024-Q1-Q2), 10Cloud-Services-Origin-Alert, 10Cloud-Services-Worktype-Unplanned, 10User-dcaro: [toolsdb] MariaDB process is killed by OOM killer (December 2023) - https://phabricator.wikimedia.org/T353093 (10dcaro) It has happened 4 times this saturday ` [16:43:57] 10cloud-services-team (FY2023/2024-Q1-Q2), 10Cloud-Services-Origin-Alert, 10Cloud-Services-Worktype-Unplanned, 10User-dcaro: [toolsdb] MariaDB process is killed by OOM killer (December 2023) - https://phabricator.wikimedia.org/T353093 (10dcaro) [16:45:56] (ToolsToolsDBReplicationError) resolved: ToolsDB replication is broken on tools-db-2 (errno 2003) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBReplication - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsToolsDBReplicationError [16:46:56] (ToolsToolsDBWritableState) resolved: There should be exactly one writable MariaDB instance instead of 0 - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsToolsDBWritableState - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsToolsDBWritableState [16:50:48] 10Grid-Engine-to-K8s-Migration: Migrate kanikbot-bn from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319836 (10Ahmad_Kanik) 05Open→03Resolved a:03Ahmad_Kanik I have migrated remaining scripts of this tool to kubernetes today. Thank you. [16:55:03] (TfInfraTestDestroyFailed) firing: Terraform failed to destroy the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestDestroyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestDestroyFailed [17:00:03] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [17:19:27] (OpenstackAPIResponse) firing: (2) Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [17:32:02] 10Grid-Engine-to-K8s-Migration: Migrate wikihistory from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320157 (10Aklapper) Is there a proper source code repository (Git etc), instead of some code-at-some-point-in-time-snapshot zip file? [17:55:30] 10Grid-Engine-to-K8s-Migration: Migrate video-cut-tool-back-end from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320124 (10Soda) 05Open→03Invalid Closing as invalid per statements [17:55:51] 10Grid-Engine-to-K8s-Migration: Migrate video-cut-tool from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320123 (10Soda) 05Open→03Invalid [17:58:48] 10Grid-Engine-to-K8s-Migration: Migrate apersonbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319561 (10Soda) @Enterprisey [18:01:21] 10Grid-Engine-to-K8s-Migration: Migrate afdstats from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319486 (10Soda) Tagging @Enterprisey as one of the people working on this :) [18:06:43] 10Grid-Engine-to-K8s-Migration: Migrate bub from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319610 (10Soda) @Samwilson Is bub even running/actively maintained, aren't we focussing on bub2 ? [18:36:03] (InstanceDown) firing: Project toolsbeta instance toolsbeta-bastion-6 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [18:36:19] 10Grid-Engine-to-K8s-Migration: Migrate pibot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319967 (10Mike_Peel) I'm the operator of Pi bot, I'll check into this as soon as I can. [19:21:29] 10Grid-Engine-to-K8s-Migration: Migrate video-cut-tool from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320123 (10Aklapper) 05Invalid→03Open In that case, please delete this tool and then resolve this ticket. Self-service tool deletion is available in Striker at https://... [19:45:30] 10Grid-Engine-to-K8s-Migration: Migrate wmf-sitematrix from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320180 (10Abbe98) Jumping in to say that I want the tool to stay up past the 14th of December. For the record, I do not appreciate the way this has been communicated, I h... [19:46:07] 10Grid-Engine-to-K8s-Migration: Migrate map-search from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319875 (10Abbe98) Jumping in to say that I want the tool to stay up past the 14th of December. For the record, I do not appreciate the way this has been communicated, I have... [19:46:22] 10Grid-Engine-to-K8s-Migration: Migrate articles-by-lat-lon-without-images from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319573 (10Abbe98) Jumping in to say that I want the tool to stay up past the 14th of December. For the record, I do not appreciate the way this has be... [19:46:37] 10Grid-Engine-to-K8s-Migration: Migrate abbe98tools from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319469 (10Abbe98) Jumping in to say that I want the tool to stay up past the 14th of December. For the record, I do not appreciate the way this has been communicated, I have... [19:54:00] (PuppetConstantChange) firing: Puppet performing a change on every puppet run on cloudweb2002-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [19:55:03] (TfInfraTestDestroyFailed) firing: Terraform failed to destroy the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestDestroyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestDestroyFailed [20:00:03] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [21:19:42] (OpenstackAPIResponse) firing: (2) Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [21:36:03] (InstanceDown) firing: Project toolsbeta instance toolsbeta-bastion-6 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [22:55:03] (TfInfraTestDestroyFailed) firing: Terraform failed to destroy the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestDestroyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestDestroyFailed [23:00:03] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [23:16:26] 10cloud-services-team: Implement 2022 Feedback/Comments for 2023 Cloud Survey - https://phabricator.wikimedia.org/T334818 (10Aklapper) @komla: Hi, the `Due Date` set for this open task passed a while ago. Could you please either update or reset the Due Date (by clicking {nav icon=pencil, name=Edit Task}), or set... [23:49:19] 10Grid-Engine-to-K8s-Migration: Migrate bracketbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319608 (10Jonesey95) This bot is long-dead. I helped with getting it set up on the user specification side. See https://en.wikipedia.org/wiki/User:BracketBot and https://en.... [23:54:00] (PuppetConstantChange) firing: Puppet performing a change on every puppet run on cloudweb2002-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange