[00:08:03] (InstanceDown) firing: Project tf-infra-test instance tf-infra-test is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [00:23:03] (InstanceDown) resolved: Project tf-infra-test instance tf-infra-test is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [00:30:09] 10Toolforge (Quota-requests): Request increased quota for anchor-corrector Toolforge tool - https://phabricator.wikimedia.org/T350484 (10Kanashimi) Thank you for your help! However, I'm getting this error when I run it. I thought the CPU quota limit was supposed to be a single task limit, maybe I'm wrong? ` # t... [00:30:29] (OpenstackAPIResponse) firing: (2) Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [00:37:13] 10Tool-iw, 10Toolforge: iw.toolforge.org does not support URL-encoded query parameters ([[toolforge:foo?bar]]) - https://phabricator.wikimedia.org/T345783 (10Mike_Peel) >>! In T345783#9333011, @Legoktm wrote: > Switching to path-based input is something that tool developers would need to support in each tool... [00:42:03] (TfInfraTestDestroyFailed) firing: Terraform failed to destroy the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestDestroyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestDestroyFailed [01:14:33] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [01:47:57] 10Grid-Engine-to-K8s-Migration: Migrate position-holder-history from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319972 (10komla) This is a reminder that the tool for which this ticket is created is still running on the Grid. The grid is deprecated and all remaining tools... [01:48:00] 10Grid-Engine-to-K8s-Migration: Migrate piglobot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319968 (10komla) This is a reminder that the tool for which this ticket is created is still running on the Grid. The grid is deprecated and all remaining tools need to migrate... [01:48:02] 10Grid-Engine-to-K8s-Migration: Migrate pibot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319967 (10komla) This is a reminder that the tool for which this ticket is created is still running on the Grid. The grid is deprecated and all remaining tools need to migrate to... [01:48:04] 10Grid-Engine-to-K8s-Migration: Migrate phetools from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319965 (10komla) This is a reminder that the tool for which this ticket is created is still running on the Grid. The grid is deprecated and all remaining tools need to migrate... [01:48:06] 10Grid-Engine-to-K8s-Migration: Migrate persondata from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319962 (10komla) This is a reminder that the tool for which this ticket is created is still running on the Grid. The grid is deprecated and all remaining tools need to migra... [01:48:08] 10Grid-Engine-to-K8s-Migration: Migrate peacock-finder from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319960 (10komla) This is a reminder that the tool for which this ticket is created is still running on the Grid. The grid is deprecated and all remaining tools need to m... [01:48:10] 10Grid-Engine-to-K8s-Migration: Migrate pb from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319957 (10komla) This is a reminder that the tool for which this ticket is created is still running on the Grid. The grid is deprecated and all remaining tools need to migrate to To... [01:48:12] 10Grid-Engine-to-K8s-Migration: Migrate panoviewer from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319953 (10komla) This is a reminder that the tool for which this ticket is created is still running on the Grid. The grid is deprecated and all remaining tools need to migra... [01:48:15] 10Grid-Engine-to-K8s-Migration: Migrate owintes from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319950 (10komla) This is a reminder that the tool for which this ticket is created is still running on the Grid. The grid is deprecated and all remaining tools need to migrate... [01:48:17] 10Grid-Engine-to-K8s-Migration: Migrate outreachy-recent-edits-tool from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319947 (10komla) This is a reminder that the tool for which this ticket is created is still running on the Grid. The grid is deprecated and all remaining to... [01:48:19] 10Grid-Engine-to-K8s-Migration: Migrate osmlint from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319944 (10komla) This is a reminder that the tool for which this ticket is created is still running on the Grid. The grid is deprecated and all remaining tools need to migrate... [01:48:21] 10Grid-Engine-to-K8s-Migration: Migrate os from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319942 (10komla) This is a reminder that the tool for which this ticket is created is still running on the Grid. The grid is deprecated and all remaining tools need to migrate to To... [01:48:23] 10Grid-Engine-to-K8s-Migration: Migrate orator-matcher from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319937 (10komla) This is a reminder that the tool for which this ticket is created is still running on the Grid. The grid is deprecated and all remaining tools need to m... [01:48:25] 10Grid-Engine-to-K8s-Migration: Migrate ooui-debug from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319936 (10komla) This is a reminder that the tool for which this ticket is created is still running on the Grid. The grid is deprecated and all remaining tools need to migra... [01:48:27] 10Grid-Engine-to-K8s-Migration: Migrate ocr4wikisource from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319933 (10komla) This is a reminder that the tool for which this ticket is created is still running on the Grid. The grid is deprecated and all remaining tools need to m... [01:48:29] 10Grid-Engine-to-K8s-Migration: Migrate nyandata from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319932 (10komla) This is a reminder that the tool for which this ticket is created is still running on the Grid. The grid is deprecated and all remaining tools need to migrate... [01:48:31] 10Grid-Engine-to-K8s-Migration: Migrate npp-lv from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319931 (10komla) This is a reminder that the tool for which this ticket is created is still running on the Grid. The grid is deprecated and all remaining tools need to migrate t... [01:48:33] 10Grid-Engine-to-K8s-Migration: Migrate npp from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319930 (10komla) This is a reminder that the tool for which this ticket is created is still running on the Grid. The grid is deprecated and all remaining tools need to migrate to T... [01:48:35] 10Grid-Engine-to-K8s-Migration: Migrate nokib-bot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319928 (10komla) This is a reminder that the tool for which this ticket is created is still running on the Grid. The grid is deprecated and all remaining tools need to migrat... [01:48:37] 10Grid-Engine-to-K8s-Migration: Migrate nn1l2bot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319926 (10komla) This is a reminder that the tool for which this ticket is created is still running on the Grid. The grid is deprecated and all remaining tools need to migrate... [01:48:39] 10Grid-Engine-to-K8s-Migration: Migrate neechal from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319922 (10komla) This is a reminder that the tool for which this ticket is created is still running on the Grid. The grid is deprecated and all remaining tools need to migrate... [01:48:42] 10Grid-Engine-to-K8s-Migration: Migrate nccroptool from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319921 (10komla) This is a reminder that the tool for which this ticket is created is still running on the Grid. The grid is deprecated and all remaining tools need to migra... [01:48:44] 10Grid-Engine-to-K8s-Migration: Migrate naggobot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319917 (10komla) This is a reminder that the tool for which this ticket is created is still running on the Grid. The grid is deprecated and all remaining tools need to migrate... [01:48:46] 10Grid-Engine-to-K8s-Migration: Migrate muninnbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319913 (10komla) This is a reminder that the tool for which this ticket is created is still running on the Grid. The grid is deprecated and all remaining tools need to migrat... [01:48:48] 10Grid-Engine-to-K8s-Migration: Migrate mrmetadata from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319911 (10komla) This is a reminder that the tool for which this ticket is created is still running on the Grid. The grid is deprecated and all remaining tools need to migra... [01:48:50] 10Grid-Engine-to-K8s-Migration: Migrate mp-count from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319910 (10komla) This is a reminder that the tool for which this ticket is created is still running on the Grid. The grid is deprecated and all remaining tools need to migrate... [01:48:52] 10Grid-Engine-to-K8s-Migration: Migrate mohib from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319902 (10komla) This is a reminder that the tool for which this ticket is created is still running on the Grid. The grid is deprecated and all remaining tools need to migrate to... [01:48:54] 10Grid-Engine-to-K8s-Migration: Migrate mmt from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319901 (10komla) This is a reminder that the tool for which this ticket is created is still running on the Grid. The grid is deprecated and all remaining tools need to migrate to T... [01:48:56] 10Grid-Engine-to-K8s-Migration: Migrate milhistbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319893 (10komla) This is a reminder that the tool for which this ticket is created is still running on the Grid. The grid is deprecated and all remaining tools need to migra... [01:48:58] 10Grid-Engine-to-K8s-Migration: Migrate mgp-cewbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319890 (10komla) This is a reminder that the tool for which this ticket is created is still running on the Grid. The grid is deprecated and all remaining tools need to migra... [02:24:33] (PuppetAgentFailure) firing: Puppet agent failure detected on instance toolsbeta-bastion-6 in project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [03:42:03] (TfInfraTestDestroyFailed) firing: Terraform failed to destroy the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestDestroyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestDestroyFailed [04:14:33] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [04:30:29] (OpenstackAPIResponse) firing: (2) Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [05:29:33] (PuppetAgentFailure) firing: Puppet agent failure detected on instance toolsbeta-bastion-6 in project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [06:33:05] 10Toolforge (Quota-requests): Request increased quota for Legoktm's Rust Toolforge tools - https://phabricator.wikimedia.org/T351604 (10Legoktm) [06:42:03] (TfInfraTestDestroyFailed) firing: Terraform failed to destroy the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestDestroyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestDestroyFailed [07:14:33] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [08:05:29] (OpenstackAPIResponse) firing: (3) Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [08:29:33] (PuppetAgentFailure) firing: Puppet agent failure detected on instance toolsbeta-bastion-6 in project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [09:42:03] (TfInfraTestDestroyFailed) firing: Terraform failed to destroy the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestDestroyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestDestroyFailed [10:14:33] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [10:22:37] (CephSlowOps) firing: Ceph cluster in eqiad has 3 slow ops - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephSlowOps - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephSlowOps [10:22:42] 10cloud-services-team: CephSlowOps Ceph cluster in eqiad has slow ops, which might be blocking some writes - https://phabricator.wikimedia.org/T349502 (10phaultfinder) [10:27:37] (CephSlowOps) resolved: Ceph cluster in eqiad has 1 slow ops - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephSlowOps - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephSlowOps [11:26:38] 10Tool-MajavahBot: Fix task approval/trial status being accidently cached in the web interface - https://phabricator.wikimedia.org/T251995 (10taavi) 05Open→03Declined [11:26:42] 10Tool-MajavahBot: Show runner host for job in MajavahBot interface - https://phabricator.wikimedia.org/T251976 (10taavi) 05Open→03Declined [11:26:44] 10Tool-MajavahBot: Add proper database migration framework in MajavahBot - https://phabricator.wikimedia.org/T251977 (10taavi) [11:26:46] 10Tool-MajavahBot: Add proper database migration framework in MajavahBot - https://phabricator.wikimedia.org/T251977 (10taavi) [11:26:48] 10Tool-MajavahBot: Show running continuous jobs in MajavahBot web interface - https://phabricator.wikimedia.org/T251975 (10taavi) 05Open→03Declined [11:29:33] (PuppetAgentFailure) firing: Puppet agent failure detected on instance toolsbeta-bastion-6 in project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [12:05:43] (OpenstackAPIResponse) firing: (3) Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [12:42:03] (TfInfraTestDestroyFailed) firing: Terraform failed to destroy the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestDestroyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestDestroyFailed [13:14:33] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [14:29:33] (PuppetAgentFailure) firing: Puppet agent failure detected on instance toolsbeta-bastion-6 in project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [15:32:08] 10Cloud-VPS (Project-requests): Request creation of tofu VPS project - https://phabricator.wikimedia.org/T351587 (10rook) +1 [15:32:19] 10Cloud-VPS (Project-requests): Request creation of tofu VPS project - https://phabricator.wikimedia.org/T351587 (10rook) 05Open→03In progress a:03rook [15:35:09] 10Cloud-VPS (Quota-requests): Additional storage for procbot - https://phabricator.wikimedia.org/T351608 (10rook) [15:38:49] 10Cloud-VPS (Project-requests): Request creation of tofu VPS project - https://phabricator.wikimedia.org/T351587 (10rook) C'est fait! ` root@cloudcontrol1005:~# openstack project create --description 'tofu' tofu --domain default Gateway Timeout (HTTP 504) root@cloudcontrol1005:~# openstack project list | grep t... [15:38:56] 10Cloud-VPS (Project-requests): Request creation of tofu VPS project - https://phabricator.wikimedia.org/T351587 (10rook) 05In progress→03Resolved [15:42:03] (TfInfraTestDestroyFailed) firing: Terraform failed to destroy the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestDestroyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestDestroyFailed [16:05:44] (OpenstackAPIResponse) firing: (3) Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [16:14:33] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [16:30:24] 10Grid-Engine-to-K8s-Migration: Migrate wikitasks from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320168 (10Vort) 05Open→03Resolved [16:37:44] 10Toolforge (Quota-requests): Request increased quota for Legoktm's Rust Toolforge tools - https://phabricator.wikimedia.org/T351604 (10rook) At a glance this feels as though it should already be (mostly) accessible to your system. As each project should already have access to 2 cores worth of cpu. At which poin... [16:45:36] 10Toolforge (Quota-requests): Request increased quota for Legoktm's Rust Toolforge tools - https://phabricator.wikimedia.org/T351604 (10taavi) The old default quotas had a K8s `LimitRange` that restricted pods to 1 CPU. The new quotas incremented in {T333979} increase them, but I still haven't fully rolled them... [16:45:45] 10Cloud-VPS (Quota-requests): Additional storage for procbot - https://phabricator.wikimedia.org/T351608 (10taavi) +1 [17:04:13] 10Cloud-VPS (Quota-requests): Additional storage for procbot - https://phabricator.wikimedia.org/T351608 (10Andrew) sgtm +1 [17:29:33] (PuppetAgentFailure) firing: Puppet agent failure detected on instance toolsbeta-bastion-6 in project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [18:42:03] (TfInfraTestDestroyFailed) firing: Terraform failed to destroy the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestDestroyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestDestroyFailed [19:07:16] RECOVERY - Check unit status of remove_dangling_cinder_snapshots on cloudbackup2001 is OK: OK: Status of the systemd unit remove_dangling_cinder_snapshots https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [19:14:33] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [19:19:02] PROBLEM - Check unit status of remove_dangling_cinder_snapshots on cloudbackup2001 is CRITICAL: CRITICAL: Status of the systemd unit remove_dangling_cinder_snapshots https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [19:40:29] (OpenstackAPIResponse) firing: (3) Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [19:45:19] 10Grid-Engine-to-K8s-Migration: Migrate mdwiki from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319887 (10komla) This is a reminder that the tool for which this ticket is created is still running on the Grid. The grid is deprecated and all remaining tools need to migrate t... [19:45:22] 10Grid-Engine-to-K8s-Migration: Migrate mbh from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319883 (10komla) This is a reminder that the tool for which this ticket is created is still running on the Grid. The grid is deprecated and all remaining tools need to migrate to T... [19:45:24] 10Grid-Engine-to-K8s-Migration: Migrate mathis-bot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319880 (10komla) This is a reminder that the tool for which this ticket is created is still running on the Grid. The grid is deprecated and all remaining tools need to migra... [19:45:26] 10Grid-Engine-to-K8s-Migration: Migrate mathbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319879 (10komla) This is a reminder that the tool for which this ticket is created is still running on the Grid. The grid is deprecated and all remaining tools need to migrate... [19:45:28] 10Grid-Engine-to-K8s-Migration: Migrate machtsinn from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319871 (10komla) This is a reminder that the tool for which this ticket is created is still running on the Grid. The grid is deprecated and all remaining tools need to migrat... [19:45:30] 10Grid-Engine-to-K8s-Migration: Migrate mabot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319870 (10komla) This is a reminder that the tool for which this ticket is created is still running on the Grid. The grid is deprecated and all remaining tools need to migrate to... [19:45:32] 10Grid-Engine-to-K8s-Migration: Migrate lziad-cvnbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319869 (10komla) This is a reminder that the tool for which this ticket is created is still running on the Grid. The grid is deprecated and all remaining tools need to mig... [19:45:34] 10Grid-Engine-to-K8s-Migration: Migrate lziad from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319868 (10komla) This is a reminder that the tool for which this ticket is created is still running on the Grid. The grid is deprecated and all remaining tools need to migrate to... [19:45:39] 10Grid-Engine-to-K8s-Migration: Migrate lyan from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319867 (10komla) This is a reminder that the tool for which this ticket is created is still running on the Grid. The grid is deprecated and all remaining tools need to migrate to... [19:45:41] 10Grid-Engine-to-K8s-Migration: Migrate lokasbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319864 (10komla) This is a reminder that the tool for which this ticket is created is still running on the Grid. The grid is deprecated and all remaining tools need to migrate... [19:45:43] 10Grid-Engine-to-K8s-Migration: Migrate linkchecker from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319858 (10komla) This is a reminder that the tool for which this ticket is created is still running on the Grid. The grid is deprecated and all remaining tools need to migr... [19:45:47] 10Grid-Engine-to-K8s-Migration: Migrate lihaohong-bot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319856 (10komla) This is a reminder that the tool for which this ticket is created is still running on the Grid. The grid is deprecated and all remaining tools need to mi... [19:45:49] 10Grid-Engine-to-K8s-Migration: Migrate languageproofing-ui from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319852 (10komla) This is a reminder that the tool for which this ticket is created is still running on the Grid. The grid is deprecated and all remaining tools need... [19:45:56] 10Grid-Engine-to-K8s-Migration: Migrate labelimgohs from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319850 (10komla) This is a reminder that the tool for which this ticket is created is still running on the Grid. The grid is deprecated and all remaining tools need to migr... [19:45:58] 10Grid-Engine-to-K8s-Migration: Migrate kolbert from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319846 (10komla) This is a reminder that the tool for which this ticket is created is still running on the Grid. The grid is deprecated and all remaining tools need to migrate... [19:46:00] 10Grid-Engine-to-K8s-Migration: Migrate khanamalumat from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319842 (10komla) This is a reminder that the tool for which this ticket is created is still running on the Grid. The grid is deprecated and all remaining tools need to mig... [19:46:02] 10Grid-Engine-to-K8s-Migration: Migrate karsilayici from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319840 (10komla) This is a reminder that the tool for which this ticket is created is still running on the Grid. The grid is deprecated and all remaining tools need to migr... [19:46:04] 10Grid-Engine-to-K8s-Migration: Migrate kanikbot-bn from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319836 (10komla) This is a reminder that the tool for which this ticket is created is still running on the Grid. The grid is deprecated and all remaining tools need to migr... [19:46:07] 10Grid-Engine-to-K8s-Migration: Migrate kaleem-bot-i from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319835 (10komla) This is a reminder that the tool for which this ticket is created is still running on the Grid. The grid is deprecated and all remaining tools need to mig... [19:46:09] 10Grid-Engine-to-K8s-Migration: Migrate kaleem-bot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319834 (10komla) This is a reminder that the tool for which this ticket is created is still running on the Grid. The grid is deprecated and all remaining tools need to migra... [19:46:11] 10Grid-Engine-to-K8s-Migration: Migrate jpxg-test from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319833 (10komla) This is a reminder that the tool for which this ticket is created is still running on the Grid. The grid is deprecated and all remaining tools need to migrat... [19:46:14] 10Grid-Engine-to-K8s-Migration: Migrate jogobot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319831 (10komla) This is a reminder that the tool for which this ticket is created is still running on the Grid. The grid is deprecated and all remaining tools need to migrate... [19:46:16] 10Grid-Engine-to-K8s-Migration: Migrate jhstools from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319829 (10komla) This is a reminder that the tool for which this ticket is created is still running on the Grid. The grid is deprecated and all remaining tools need to migrate... [19:46:18] 10Grid-Engine-to-K8s-Migration: Migrate jembot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319828 (10komla) This is a reminder that the tool for which this ticket is created is still running on the Grid. The grid is deprecated and all remaining tools need to migrate t... [19:46:20] 10Grid-Engine-to-K8s-Migration: Migrate itemlister from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319822 (10komla) This is a reminder that the tool for which this ticket is created is still running on the Grid. The grid is deprecated and all remaining tools need to migra... [19:46:22] 10Grid-Engine-to-K8s-Migration: Migrate isbn from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319819 (10komla) This is a reminder that the tool for which this ticket is created is still running on the Grid. The grid is deprecated and all remaining tools need to migrate to... [19:46:24] 10Grid-Engine-to-K8s-Migration: Migrate isa from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319818 (10komla) This is a reminder that the tool for which this ticket is created is still running on the Grid. The grid is deprecated and all remaining tools need to migrate to T... [19:46:31] 10Grid-Engine-to-K8s-Migration: Migrate ipp from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319817 (10komla) This is a reminder that the tool for which this ticket is created is still running on the Grid. The grid is deprecated and all remaining tools need to migrate to T... [19:46:34] 10Grid-Engine-to-K8s-Migration: Migrate integraality from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319813 (10komla) This is a reminder that the tool for which this ticket is created is still running on the Grid. The grid is deprecated and all remaining tools need to mig... [19:46:36] 10Grid-Engine-to-K8s-Migration: Migrate ideasbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319805 (10komla) This is a reminder that the tool for which this ticket is created is still running on the Grid. The grid is deprecated and all remaining tools need to migrate... [19:46:38] 10Grid-Engine-to-K8s-Migration: Migrate huntleybots from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319802 (10komla) This is a reminder that the tool for which this ticket is created is still running on the Grid. The grid is deprecated and all remaining tools need to migr... [19:46:40] 10Grid-Engine-to-K8s-Migration: Migrate htools from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319796 (10komla) This is a reminder that the tool for which this ticket is created is still running on the Grid. The grid is deprecated and all remaining tools need to migrate t... [19:46:42] 10Grid-Engine-to-K8s-Migration: Migrate hsfbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319795 (10komla) This is a reminder that the tool for which this ticket is created is still running on the Grid. The grid is deprecated and all remaining tools need to migrate t... [19:46:44] 10Grid-Engine-to-K8s-Migration: Migrate himo from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319789 (10komla) This is a reminder that the tool for which this ticket is created is still running on the Grid. The grid is deprecated and all remaining tools need to migrate to... [19:46:47] 10Grid-Engine-to-K8s-Migration: Migrate hazard-bot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319785 (10komla) This is a reminder that the tool for which this ticket is created is still running on the Grid. The grid is deprecated and all remaining tools need to migra... [19:46:49] 10Grid-Engine-to-K8s-Migration: Migrate heritage from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319787 (10komla) This is a reminder that the tool for which this ticket is created is still running on the Grid. The grid is deprecated and all remaining tools need to migrate... [19:46:51] 10Grid-Engine-to-K8s-Migration: Migrate hashtagwatcher from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319784 (10komla) This is a reminder that the tool for which this ticket is created is still running on the Grid. The grid is deprecated and all remaining tools need to m... [19:46:53] 10Grid-Engine-to-K8s-Migration: Migrate hamishbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319783 (10komla) This is a reminder that the tool for which this ticket is created is still running on the Grid. The grid is deprecated and all remaining tools need to migrat... [19:46:55] 10Grid-Engine-to-K8s-Migration: Migrate grapedog from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319780 (10komla) This is a reminder that the tool for which this ticket is created is still running on the Grid. The grid is deprecated and all remaining tools need to migrate... [19:46:59] 10Grid-Engine-to-K8s-Migration, 10Community-Tech, 10Event Metrics: Migrate grantmetrics from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319779 (10komla) This is a reminder that the tool for which this ticket is created is still running on the Grid. The grid is depreca... [19:50:43] 10Grid-Engine-to-K8s-Migration: Migrate milhistbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319893 (10Hawkeye7) 05Open→03Resolved a:03Hawkeye7 Tool has been migrated to using the kubernetes containers. However the task to create a proper dotnet build environmen... [20:34:33] (PuppetAgentFailure) firing: Puppet agent failure detected on instance toolsbeta-bastion-6 in project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [21:42:03] (TfInfraTestDestroyFailed) firing: Terraform failed to destroy the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestDestroyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestDestroyFailed [22:14:33] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [23:17:03] (PuppetCertificateAboutToExpire) firing: Puppet CA certificate tools-sgegrid-shadow.tools.eqiad.wmflabs is about to expire in 27d 23h 58m 14s - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire [23:34:33] (PuppetAgentFailure) firing: Puppet agent failure detected on instance toolsbeta-bastion-6 in project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [23:40:29] (OpenstackAPIResponse) firing: (2) Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [23:48:54] 10Tool-inteGraality: Internal Server Error on InteGraality - https://phabricator.wikimedia.org/T351574 (10PKM) The error also occurs when selecting "manually update" from the top menu.