[00:07:03] <wmcs-alerts>	 (InstanceDown) firing: Project tf-infra-test instance tf-infra-test is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[00:17:03] <wmcs-alerts>	 (InstanceDown) resolved: Project tf-infra-test instance tf-infra-test is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[01:11:40] <wmcs-alerts>	 (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed
[03:35:42] <jinxer-wm>	 (OpenstackAPIResponse) firing: (2) Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse
[03:57:03] <wmcs-alerts>	 (PuppetAgentStaleLastRun) firing: Last Puppet run was over 24 hours ago on instance metricsinfra-alertmanager-1 in project metricsinfra   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun
[04:02:03] <wmcs-alerts>	 (PuppetAgentStaleLastRun) resolved: Last Puppet run was over 24 hours ago on instance metricsinfra-alertmanager-1 in project metricsinfra   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun
[04:11:40] <wmcs-alerts>	 (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed
[04:12:03] <wmcs-alerts>	 (PuppetAgentStaleLastRun) firing: Last Puppet run was over 24 hours ago on instance metricsinfra-alertmanager-1 in project metricsinfra   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun
[04:17:03] <wmcs-alerts>	 (PuppetAgentStaleLastRun) resolved: Last Puppet run was over 24 hours ago on instance metricsinfra-alertmanager-1 in project metricsinfra   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun
[04:30:28] <jinxer-wm>	 (OpenstackAPIResponse) firing: (2) Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse
[05:01:03] <wmcs-alerts>	 (PuppetAgentStaleLastRun) firing: Last Puppet run was over 24 hours ago on instance metricsinfra-alertmanager-1 in project metricsinfra   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun
[05:06:03] <wmcs-alerts>	 (PuppetAgentStaleLastRun) resolved: Last Puppet run was over 24 hours ago on instance metricsinfra-alertmanager-1 in project metricsinfra   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun
[05:24:03] <wmcs-alerts>	 (PuppetAgentStaleLastRun) firing: Last Puppet run was over 24 hours ago on instance metricsinfra-alertmanager-1 in project metricsinfra   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun
[05:29:03] <wmcs-alerts>	 (PuppetAgentStaleLastRun) resolved: Last Puppet run was over 24 hours ago on instance metricsinfra-alertmanager-1 in project metricsinfra   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun
[05:58:03] <wmcs-alerts>	 (PuppetAgentStaleLastRun) firing: Last Puppet run was over 24 hours ago on instance metricsinfra-alertmanager-1 in project metricsinfra   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun
[06:03:03] <wmcs-alerts>	 (PuppetAgentStaleLastRun) resolved: Last Puppet run was over 24 hours ago on instance metricsinfra-alertmanager-1 in project metricsinfra   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun
[07:11:40] <wmcs-alerts>	 (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed
[08:30:42] <jinxer-wm>	 (OpenstackAPIResponse) firing: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse
[08:46:16] <wikibugs>	 10Tool-inteGraality: Internal Server Error on InteGraality - https://phabricator.wikimedia.org/T351574 (10Aklapper)
[10:11:41] <wmcs-alerts>	 (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed
[10:58:03] <wmcs-alerts>	 (PuppetAgentStaleLastRun) firing: Last Puppet run was over 24 hours ago on instance metricsinfra-alertmanager-1 in project metricsinfra   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun
[11:03:03] <wmcs-alerts>	 (PuppetAgentStaleLastRun) resolved: Last Puppet run was over 24 hours ago on instance metricsinfra-alertmanager-1 in project metricsinfra   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun
[11:14:37] <jinxer-wm>	 (CephSlowOps) firing: Ceph cluster in eqiad has 1 slow ops - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephSlowOps - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephSlowOps
[11:14:42] <wikibugs>	 10cloud-services-team: CephSlowOps  Ceph cluster in eqiad has slow ops, which might be blocking some writes - https://phabricator.wikimedia.org/T349502 (10phaultfinder)
[11:19:37] <jinxer-wm>	 (CephSlowOps) resolved: Ceph cluster in eqiad has 1 slow ops - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephSlowOps - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephSlowOps
[11:35:03] <wmcs-alerts>	 (PuppetAgentNoResources) firing: No Puppet resources found on instance toolsbeta-sgewebgen-10-2 on project toolsbeta   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[11:39:03] <wmcs-alerts>	 (PuppetAgentFailure) firing: Puppet agent failure detected on instance toolsbeta-bastion-6 in project toolsbeta   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure
[11:45:03] <wmcs-alerts>	 (PuppetAgentNoResources) firing: (3) No Puppet resources found on instance toolsbeta-sgewebgen-10-2 on project toolsbeta   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[11:50:03] <wmcs-alerts>	 (PuppetAgentNoResources) firing: (5) No Puppet resources found on instance toolsbeta-sgeexec-10-5 on project toolsbeta   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[11:55:03] <wmcs-alerts>	 (PuppetAgentNoResources) firing: (6) No Puppet resources found on instance toolsbeta-sgecron-02 on project toolsbeta   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[12:00:03] <wmcs-alerts>	 (PuppetAgentNoResources) firing: (6) No Puppet resources found on instance toolsbeta-sgecron-02 on project toolsbeta   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[12:10:03] <wmcs-alerts>	 (PuppetAgentNoResources) firing: (6) No Puppet resources found on instance toolsbeta-sgecron-02 on project toolsbeta   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[12:22:03] <wmcs-alerts>	 (PuppetAgentStaleLastRun) firing: Last Puppet run was over 24 hours ago on instance metricsinfra-alertmanager-1 in project metricsinfra   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun
[12:25:03] <wmcs-alerts>	 (PuppetAgentNoResources) resolved: (3) No Puppet resources found on instance toolsbeta-sgecron-02 on project toolsbeta   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[12:27:03] <wmcs-alerts>	 (PuppetAgentStaleLastRun) resolved: Last Puppet run was over 24 hours ago on instance metricsinfra-alertmanager-1 in project metricsinfra   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun
[12:30:42] <jinxer-wm>	 (OpenstackAPIResponse) firing: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse
[12:54:03] <wmcs-alerts>	 (PuppetAgentFailure) firing: (2) Puppet agent failure detected on instance toolsbeta-bastion-6 in project toolsbeta   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure
[13:11:41] <wmcs-alerts>	 (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed
[13:17:03] <wmcs-alerts>	 (PuppetAgentStaleLastRun) firing: Last Puppet run was over 24 hours ago on instance metricsinfra-alertmanager-1 in project metricsinfra   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun
[13:22:03] <wmcs-alerts>	 (PuppetAgentStaleLastRun) resolved: Last Puppet run was over 24 hours ago on instance metricsinfra-alertmanager-1 in project metricsinfra   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun
[13:26:03] <wmcs-alerts>	 (PuppetAgentStaleLastRun) firing: Last Puppet run was over 24 hours ago on instance metricsinfra-alertmanager-1 in project metricsinfra   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun
[13:31:03] <wmcs-alerts>	 (PuppetAgentStaleLastRun) resolved: Last Puppet run was over 24 hours ago on instance metricsinfra-alertmanager-1 in project metricsinfra   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun
[14:14:33] <wmcs-alerts>	 (PuppetAgentFailure) firing: (2) Puppet agent failure detected on instance toolsbeta-bastion-6 in project toolsbeta   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure
[14:40:05] <wikibugs>	 10Cloud-VPS (Project-requests): Request creation of tofu VPS project - https://phabricator.wikimedia.org/T351587 (10taavi)
[14:47:51] <wikibugs>	 10Wikibugs, 10Patch-For-Review: Better message than "This change is ready for review" when patch stops being WIP - https://phabricator.wikimedia.org/T350778 (10valhallasw) Took an initial stab at adding the scaffolding for changing the behavior in the WIP patchset above. Implementation still to be done (and op...
[16:14:33] <wmcs-alerts>	 (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed
[16:30:43] <jinxer-wm>	 (OpenstackAPIResponse) firing: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse
[16:30:45] <wikibugs>	 10Toolforge (Quota-requests): Request increased quota for anchor-corrector Toolforge tool - https://phabricator.wikimedia.org/T350484 (10taavi) Ok, I bumped that quota as well. You should be all set now.
[17:19:33] <wmcs-alerts>	 (PuppetAgentFailure) firing: Puppet agent failure detected on instance toolsbeta-bastion-6 in project toolsbeta   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure
[18:22:19] <jinxer-wm>	 (HAProxyBackendUnavailable) firing: HAProxy service neutron-api_backend backend cloudcontrol1007.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable
[18:27:19] <jinxer-wm>	 (HAProxyBackendUnavailable) resolved: HAProxy service neutron-api_backend backend cloudcontrol1007.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable
[18:29:14] <wikibugs>	 10Grid-Engine-to-K8s-Migration: Migrate veblenbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320119 (10komla)  This is a reminder that the tool for which this ticket is created is still running on the Grid.  The grid is deprecated and all remaining tools need to migrat...
[18:29:18] <wikibugs>	 10Grid-Engine-to-K8s-Migration: Migrate userimpact from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320116 (10komla)  This is a reminder that the tool for which this ticket is created is still running on the Grid.  The grid is deprecated and all remaining tools need to migra...
[18:29:20] <wikibugs>	 10Grid-Engine-to-K8s-Migration: Migrate urlinktranslator from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320112 (10komla)  This is a reminder that the tool for which this ticket is created is still running on the Grid.  The grid is deprecated and all remaining tools need to...
[18:29:27] <wikibugs>	 10Grid-Engine-to-K8s-Migration: Migrate urduspellchecker from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320110 (10komla)  This is a reminder that the tool for which this ticket is created is still running on the Grid.  The grid is deprecated and all remaining tools need to...
[18:29:34] <wikibugs>	 10Grid-Engine-to-K8s-Migration: Migrate urdubot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320109 (10komla)  This is a reminder that the tool for which this ticket is created is still running on the Grid.  The grid is deprecated and all remaining tools need to migrate...
[18:29:38] <wikibugs>	 10Grid-Engine-to-K8s-Migration: Migrate upload-stats-bot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320106 (10komla)  This is a reminder that the tool for which this ticket is created is still running on the Grid.  The grid is deprecated and all remaining tools need to...
[18:29:40] <wikibugs>	 10Grid-Engine-to-K8s-Migration: Migrate uarchivebot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320104 (10komla)  This is a reminder that the tool for which this ticket is created is still running on the Grid.  The grid is deprecated and all remaining tools need to migr...
[18:29:43] <wikibugs>	 10Grid-Engine-to-K8s-Migration: Migrate twltools from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320102 (10komla)  This is a reminder that the tool for which this ticket is created is still running on the Grid.  The grid is deprecated and all remaining tools need to migrate...
[18:29:48] <wikibugs>	 10Grid-Engine-to-K8s-Migration: Migrate twl from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320098 (10komla)  This is a reminder that the tool for which this ticket is created is still running on the Grid.  The grid is deprecated and all remaining tools need to migrate to T...
[18:30:02] <wikibugs>	 10Grid-Engine-to-K8s-Migration: Migrate twitterbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320097 (10komla)  This is a reminder that the tool for which this ticket is created is still running on the Grid.  The grid is deprecated and all remaining tools need to migra...
[18:32:26] <wikibugs>	 10Toolforge (Quota-requests): Request increased quota for anchor-corrector Toolforge tool - https://phabricator.wikimedia.org/T350484 (10komla) @Kanashimi once you test and this works for you, mark this ticket as resolved.
[18:34:51] <wikibugs>	 10Grid-Engine-to-K8s-Migration: Migrate tulsibot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320095 (10komla)  This is a reminder that the tool for which this ticket is created is still running on the Grid.  The grid is deprecated and all remaining tools need to migrate...
[18:34:54] <wikibugs>	 10Grid-Engine-to-K8s-Migration: Migrate title-search from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320087 (10komla)  This is a reminder that the tool for which this ticket is created is still running on the Grid.  The grid is deprecated and all remaining tools need to mig...
[18:34:56] <wikibugs>	 10Grid-Engine-to-K8s-Migration: Migrate test-vvv from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320085 (10komla)  This is a reminder that the tool for which this ticket is created is still running on the Grid.  The grid is deprecated and all remaining tools need to migrate...
[18:34:59] <wikibugs>	 10Grid-Engine-to-K8s-Migration: Migrate telegram-wikilinksbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320082 (10komla)  This is a reminder that the tool for which this ticket is created is still running on the Grid.  The grid is deprecated and all remaining tools ne...
[18:35:01] <wikibugs>	 10Grid-Engine-to-K8s-Migration, 10User-revi: Migrate tc-rc from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320079 (10komla)  This is a reminder that the tool for which this ticket is created is still running on the Grid.  The grid is deprecated and all remaining tools nee...
[18:35:03] <wikibugs>	 10Grid-Engine-to-K8s-Migration: Migrate t187305-demo from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320075 (10komla)  This is a reminder that the tool for which this ticket is created is still running on the Grid.  The grid is deprecated and all remaining tools need to mig...
[18:35:05] <wikibugs>	 10Grid-Engine-to-K8s-Migration: Migrate suha from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320069 (10komla)  This is a reminder that the tool for which this ticket is created is still running on the Grid.  The grid is deprecated and all remaining tools need to migrate to...
[18:35:07] <wikibugs>	 10Grid-Engine-to-K8s-Migration: Migrate suggestbotbn from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320068 (10komla)  This is a reminder that the tool for which this ticket is created is still running on the Grid.  The grid is deprecated and all remaining tools need to mig...
[18:35:09] <wikibugs>	 10Grid-Engine-to-K8s-Migration: Migrate suggestbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320067 (10komla)  This is a reminder that the tool for which this ticket is created is still running on the Grid.  The grid is deprecated and all remaining tools need to migra...
[18:35:12] <wikibugs>	 10Grid-Engine-to-K8s-Migration: Migrate submitter from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320066 (10komla)  This is a reminder that the tool for which this ticket is created is still running on the Grid.  The grid is deprecated and all remaining tools need to migrat...
[18:35:14] <wikibugs>	 10Grid-Engine-to-K8s-Migration: Migrate studiesworld from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320064 (10komla)  This is a reminder that the tool for which this ticket is created is still running on the Grid.  The grid is deprecated and all remaining tools need to mig...
[18:35:19] <wikibugs>	 10Grid-Engine-to-K8s-Migration: Migrate stockholm-mania from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320063 (10komla)  This is a reminder that the tool for which this ticket is created is still running on the Grid.  The grid is deprecated and all remaining tools need to...
[18:35:21] <wikibugs>	 10Grid-Engine-to-K8s-Migration: Migrate stereoskopie from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320061 (10komla)  This is a reminder that the tool for which this ticket is created is still running on the Grid.  The grid is deprecated and all remaining tools need to mig...
[18:35:23] <wikibugs>	 10Grid-Engine-to-K8s-Migration: Migrate status from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320058 (10komla)  This is a reminder that the tool for which this ticket is created is still running on the Grid.  The grid is deprecated and all remaining tools need to migrate t...
[18:35:25] <wikibugs>	 10Grid-Engine-to-K8s-Migration: Migrate sqid from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320056 (10komla)  This is a reminder that the tool for which this ticket is created is still running on the Grid.  The grid is deprecated and all remaining tools need to migrate to...
[18:35:30] <wikibugs>	 10Grid-Engine-to-K8s-Migration: Migrate spellcheck from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320053 (10komla)  This is a reminder that the tool for which this ticket is created is still running on the Grid.  The grid is deprecated and all remaining tools need to migra...
[18:35:32] <wikibugs>	 10Grid-Engine-to-K8s-Migration: Migrate smallem from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320048 (10komla)  This is a reminder that the tool for which this ticket is created is still running on the Grid.  The grid is deprecated and all remaining tools need to migrate...
[18:35:34] <wikibugs>	 10Grid-Engine-to-K8s-Migration: Migrate sigma from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320041 (10komla)  This is a reminder that the tool for which this ticket is created is still running on the Grid.  The grid is deprecated and all remaining tools need to migrate to...
[18:35:37] <wikibugs>	 10Grid-Engine-to-K8s-Migration: Migrate shumariyat from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320040 (10komla)  This is a reminder that the tool for which this ticket is created is still running on the Grid.  The grid is deprecated and all remaining tools need to migra...
[18:35:39] <wikibugs>	 10Grid-Engine-to-K8s-Migration: Migrate shuaib-bot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320039 (10komla)  This is a reminder that the tool for which this ticket is created is still running on the Grid.  The grid is deprecated and all remaining tools need to migra...
[18:58:19] <jinxer-wm>	 (HAProxyBackendUnavailable) firing: HAProxy service neutron-api_backend backend cloudcontrol1005.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable
[19:03:19] <jinxer-wm>	 (HAProxyBackendUnavailable) resolved: HAProxy service neutron-api_backend backend cloudcontrol1005.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable
[19:04:04] <icinga-wm>	 RECOVERY - Check unit status of remove_dangling_cinder_snapshots on cloudbackup2001 is OK: OK: Status of the systemd unit remove_dangling_cinder_snapshots https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
[19:14:33] <wmcs-alerts>	 (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed
[19:15:48] <icinga-wm>	 PROBLEM - Check unit status of remove_dangling_cinder_snapshots on cloudbackup2001 is CRITICAL: CRITICAL: Status of the systemd unit remove_dangling_cinder_snapshots https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
[20:24:33] <wmcs-alerts>	 (PuppetAgentFailure) firing: Puppet agent failure detected on instance toolsbeta-bastion-6 in project toolsbeta   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure
[20:30:28] <jinxer-wm>	 (OpenstackAPIResponse) firing: (2) Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse
[21:33:33] <wikibugs>	 10Tools, 10Diffusion-Repository-Administrators, 10Projects-Cleanup: Consider archiving Gerrit repository "labs/tools/multichill" (20140806) - https://phabricator.wikimedia.org/T351599 (10Aklapper)
[21:33:49] <wikibugs>	 10Tools, 10Diffusion-Repository-Administrators, 10Projects-Cleanup: Consider archiving Gerrit repository "labs/tools/pywikibugs" (20141017) - https://phabricator.wikimedia.org/T351601 (10Aklapper)
[22:14:33] <wmcs-alerts>	 (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed
[23:24:33] <wmcs-alerts>	 (PuppetAgentFailure) firing: Puppet agent failure detected on instance toolsbeta-bastion-6 in project toolsbeta   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure
[23:43:19] <wikibugs>	 10Grid-Engine-to-K8s-Migration: Migrate shuaib from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320038 (10komla)  This is a reminder that the tool for which this ticket is created is still running on the Grid.  The grid is deprecated and all remaining tools need to migrate t...
[23:43:21] <wikibugs>	 10Grid-Engine-to-K8s-Migration: Migrate shrinitools from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320037 (10komla)  This is a reminder that the tool for which this ticket is created is still running on the Grid.  The grid is deprecated and all remaining tools need to migr...
[23:43:23] <wikibugs>	 10Grid-Engine-to-K8s-Migration: Migrate section-watchlist from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320033 (10komla)  This is a reminder that the tool for which this ticket is created is still running on the Grid.  The grid is deprecated and all remaining tools need t...
[23:43:25] <wikibugs>	 10Grid-Engine-to-K8s-Migration: Migrate sanaa from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320028 (10komla)  This is a reminder that the tool for which this ticket is created is still running on the Grid.  The grid is deprecated and all remaining tools need to migrate to...
[23:43:28] <wikibugs>	 10Grid-Engine-to-K8s-Migration: Migrate sammour from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320027 (10komla)  This is a reminder that the tool for which this ticket is created is still running on the Grid.  The grid is deprecated and all remaining tools need to migrate...
[23:43:30] <wikibugs>	 10Grid-Engine-to-K8s-Migration: Migrate saami from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320026 (10komla)  This is a reminder that the tool for which this ticket is created is still running on the Grid.  The grid is deprecated and all remaining tools need to migrate to...
[23:43:32] <wikibugs>	 10Grid-Engine-to-K8s-Migration: Migrate ruwn-misc from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320024 (10komla)  This is a reminder that the tool for which this ticket is created is still running on the Grid.  The grid is deprecated and all remaining tools need to migrat...
[23:43:34] <wikibugs>	 10Grid-Engine-to-K8s-Migration: Migrate ruwikisource from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320023 (10komla)  This is a reminder that the tool for which this ticket is created is still running on the Grid.  The grid is deprecated and all remaining tools need to mig...
[23:43:36] <wikibugs>	 10Grid-Engine-to-K8s-Migration: Migrate ru-unreliable-music-sources from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320017 (10komla)  This is a reminder that the tool for which this ticket is created is still running on the Grid.  The grid is deprecated and all remaining to...
[23:43:38] <wikibugs>	 10Grid-Engine-to-K8s-Migration: Migrate rluts from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320013 (10komla)  This is a reminder that the tool for which this ticket is created is still running on the Grid.  The grid is deprecated and all remaining tools need to migrate to...
[23:43:40] <wikibugs>	 10Grid-Engine-to-K8s-Migration: Migrate rileybot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320012 (10komla)  This is a reminder that the tool for which this ticket is created is still running on the Grid.  The grid is deprecated and all remaining tools need to migrate...
[23:43:42] <wikibugs>	 10Grid-Engine-to-K8s-Migration, 10User-revi: Migrate revibot-ii from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320008 (10komla)  This is a reminder that the tool for which this ticket is created is still running on the Grid.  The grid is deprecated and all remaining tool...
[23:43:45] <wikibugs>	 10Grid-Engine-to-K8s-Migration, 10User-revi: Migrate revibot-i from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320007 (10komla)  This is a reminder that the tool for which this ticket is created is still running on the Grid.  The grid is deprecated and all remaining tools...
[23:43:50] <wikibugs>	 10Grid-Engine-to-K8s-Migration, 10User-revi: Migrate revibot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320006 (10komla)  This is a reminder that the tool for which this ticket is created is still running on the Grid.  The grid is deprecated and all remaining tools n...
[23:43:55] <wikibugs>	 10Grid-Engine-to-K8s-Migration: Migrate resizeimagesbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320004 (10komla)  This is a reminder that the tool for which this ticket is created is still running on the Grid.  The grid is deprecated and all remaining tools need to...
[23:43:59] <wikibugs>	 10Grid-Engine-to-K8s-Migration: Migrate request from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320003 (10komla)  This is a reminder that the tool for which this ticket is created is still running on the Grid.  The grid is deprecated and all remaining tools need to migrate...
[23:44:03] <wikibugs>	 10Grid-Engine-to-K8s-Migration: Migrate render-tests from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320002 (10komla)  This is a reminder that the tool for which this ticket is created is still running on the Grid.  The grid is deprecated and all remaining tools need to mig...
[23:44:07] <wikibugs>	 10Grid-Engine-to-K8s-Migration: Migrate render from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320001 (10komla)  This is a reminder that the tool for which this ticket is created is still running on the Grid.  The grid is deprecated and all remaining tools need to migrate t...
[23:44:11] <wikibugs>	 10Grid-Engine-to-K8s-Migration: Migrate relgen from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320000 (10komla)  This is a reminder that the tool for which this ticket is created is still running on the Grid.  The grid is deprecated and all remaining tools need to migrate t...
[23:44:15] <wikibugs>	 10Grid-Engine-to-K8s-Migration: Migrate redirtalkdeleter from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319998 (10komla)  This is a reminder that the tool for which this ticket is created is still running on the Grid.  The grid is deprecated and all remaining tools need to...
[23:44:19] <wikibugs>	 10Grid-Engine-to-K8s-Migration: Migrate rchv from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319994 (10komla)  This is a reminder that the tool for which this ticket is created is still running on the Grid.  The grid is deprecated and all remaining tools need to migrate to...
[23:44:23] <wikibugs>	 10Grid-Engine-to-K8s-Migration: Migrate raymond from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319993 (10komla)  This is a reminder that the tool for which this ticket is created is still running on the Grid.  The grid is deprecated and all remaining tools need to migrate...
[23:44:27] <wikibugs>	 10Grid-Engine-to-K8s-Migration: Migrate rangecontrib from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319991 (10komla)  This is a reminder that the tool for which this ticket is created is still running on the Grid.  The grid is deprecated and all remaining tools need to mig...
[23:44:31] <wikibugs>	 10Grid-Engine-to-K8s-Migration: Migrate qic from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319982 (10komla)  This is a reminder that the tool for which this ticket is created is still running on the Grid.  The grid is deprecated and all remaining tools need to migrate to T...
[23:44:35] <wikibugs>	 10Grid-Engine-to-K8s-Migration: Migrate ptbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319980 (10komla)  This is a reminder that the tool for which this ticket is created is still running on the Grid.  The grid is deprecated and all remaining tools need to migrate to...
[23:44:39] <wikibugs>	 10Grid-Engine-to-K8s-Migration: Migrate prompter from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319979 (10komla)  This is a reminder that the tool for which this ticket is created is still running on the Grid.  The grid is deprecated and all remaining tools need to migrate...
[23:44:45] <wikibugs>	 10Grid-Engine-to-K8s-Migration, 10Pywikibot: Migrate pywikibot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319981 (10komla)  This is a reminder that the tool for which this ticket is created is still running on the Grid.  The grid is deprecated and all remaining tools...
[23:44:51] <wikibugs>	 10Grid-Engine-to-K8s-Migration: Migrate projektneuheiten-feed from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319978 (10komla)  This is a reminder that the tool for which this ticket is created is still running on the Grid.  The grid is deprecated and all remaining tools ne...
[23:44:55] <wikibugs>	 10Grid-Engine-to-K8s-Migration: Migrate procbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319977 (10komla)  This is a reminder that the tool for which this ticket is created is still running on the Grid.  The grid is deprecated and all remaining tools need to migrate...
[23:44:59] <wikibugs>	 10Grid-Engine-to-K8s-Migration: Migrate postcardcatfinder from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319973 (10komla)  This is a reminder that the tool for which this ticket is created is still running on the Grid.  The grid is deprecated and all remaining tools need t...