[00:09:03] (TfInfraTestDestroyFailed) resolved: Terraform failed to destroy the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestDestroyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestDestroyFailed [00:09:03] (InstanceDown) firing: Project tf-infra-test instance tf-infra-test is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [00:14:03] (InstanceDown) resolved: Project tf-infra-test instance tf-infra-test is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [01:35:03] (InstanceDown) firing: Project tools instance tools-k8s-worker-84 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [01:40:23] (ToolforgeKubernetesNodeNotReady) firing: Kubernetes node tools-k8s-worker-84 is not ready - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesNodeNotReady - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesNodeNotReady [01:48:03] (PuppetAgentNoResources) firing: No Puppet resources found on instance toolsbeta-bastion-6 on project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [01:50:03] (InstanceDown) resolved: Project tools instance tools-k8s-worker-84 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [01:50:23] (ToolforgeKubernetesNodeNotReady) resolved: Kubernetes node tools-k8s-worker-84 is not ready - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesNodeNotReady - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesNodeNotReady [02:01:22] (HAProxyBackendUnavailable) firing: HAProxy service neutron-api_backend backend cloudcontrol1007.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [02:06:22] (HAProxyBackendUnavailable) resolved: HAProxy service neutron-api_backend backend cloudcontrol1007.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [04:32:09] 10Tools: uncategorized articles - https://phabricator.wikimedia.org/T355014 (10Bugreporter) [04:48:04] (PuppetAgentNoResources) firing: No Puppet resources found on instance toolsbeta-bastion-6 on project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:18:03] (InstanceDown) firing: Project tools instance tools-k8s-worker-84 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [07:23:23] (ToolforgeKubernetesNodeNotReady) firing: Kubernetes node tools-k8s-worker-84 is not ready - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesNodeNotReady - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesNodeNotReady [07:41:45] (ProbeDown) firing: Service tools-k8s-haproxy-4:30000 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-k8s-haproxy-4:30000 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [07:46:45] (ProbeDown) resolved: Service tools-k8s-haproxy-4:30000 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-k8s-haproxy-4:30000 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [07:48:03] (PuppetAgentNoResources) firing: No Puppet resources found on instance toolsbeta-bastion-6 on project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [07:53:03] (InstanceDown) resolved: Project tools instance tools-k8s-worker-84 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [07:54:49] 10Tools: uncategorized articles - https://phabricator.wikimedia.org/T355014 (10Peachey88) 05Open→03Invalid The page in question, directs (in the footer) contact queries to https://en.wikipedia.org/wiki/User_talk:JaGa. Please raise as appropriate on JaGa's talk page. [07:58:23] (ToolforgeKubernetesNodeNotReady) resolved: Kubernetes node tools-k8s-worker-84 is not ready - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesNodeNotReady - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesNodeNotReady [10:48:04] (PuppetAgentNoResources) firing: No Puppet resources found on instance toolsbeta-bastion-6 on project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [13:53:04] (PuppetAgentNoResources) firing: No Puppet resources found on instance toolsbeta-bastion-6 on project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [14:08:25] PROBLEM - Wikitech-static main page has content on wikitech-static.wikimedia.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - string Wikitech not found on https://wikitech-static.wikimedia.org:443/wiki/Main_Page?debug=true - 4836 bytes in 0.130 second response time https://wikitech.wikimedia.org/wiki/Wikitech-static [14:13:21] 10Tools: uncategorized articles - https://phabricator.wikimedia.org/T355014 (10Bearcat) 05Invalid→03Open JaGa is a virtually retired editor who was on Wikipedia just four times in all of 2023, and has not responded to any posts on his talk page since //2016//. So that's not a solution; this is not a thing th... [14:14:45] 10Tools: uncategorized articles - https://phabricator.wikimedia.org/T355014 (10Bearcat) >>! In T355014#9458414, @Bearcat wrote: > JaGa is a virtually retired editor who was on Wikipedia just four times in all of 2023, and has not responded to any posts on his talk page since //2016//. So that's not a solution; t... [14:44:27] RECOVERY - Wikitech-static main page has content on wikitech-static.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 26882 bytes in 2.944 second response time https://wikitech.wikimedia.org/wiki/Wikitech-static [15:03:12] 10Toolforge: There are some files that I cannot view, delete, or do anything to - https://phabricator.wikimedia.org/T355022 (10GergesSH) [15:28:37] 10Toolforge: There are some files that I cannot view, delete, or do anything to - https://phabricator.wikimedia.org/T355022 (10Andrew) After becoming gergesbot2 I'm unable to reproduce these issues on either sge-bastion-10 or sge-bastion-11. Do you have any hints about what's different on your end? What user ac... [16:06:10] (CephSlowOps) firing: Ceph cluster in eqiad has 41 slow ops - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephSlowOps - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephSlowOps [16:06:15] 10cloud-services-team: CephSlowOps Ceph cluster in eqiad has slow ops, which might be blocking some writes - https://phabricator.wikimedia.org/T352570 (10phaultfinder) [16:11:09] (CephSlowOps) resolved: Ceph cluster in eqiad has 7 slow ops - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephSlowOps - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephSlowOps [16:53:04] (PuppetAgentNoResources) firing: No Puppet resources found on instance toolsbeta-bastion-6 on project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [17:42:48] 10Toolforge: There are some files that I cannot view, delete, or do anything to - https://phabricator.wikimedia.org/T355022 (10GergesSH) a:03GergesSH There is still a problem. Use a command: composer update [17:44:01] 10Toolforge: There are some files that I cannot view, delete, or do anything to - https://phabricator.wikimedia.org/T355022 (10GergesSH) There is still a problem. Use a command: composer update [17:59:49] 10Toolforge: There are some files that I cannot view, delete, or do anything to - https://phabricator.wikimedia.org/T355022 (10GergesSH) a:05GergesSH→03None [18:16:24] 10Toolforge (Toolforge iteration 03), 10Toolforge Build Service: Toolforge Build Service: add the locale buildpack - https://phabricator.wikimedia.org/T354128 (10Earwig) 05Resolved→03Open [18:16:47] 10Toolforge (Toolforge iteration 03), 10Toolforge Build Service: Toolforge Build Service: add the locale buildpack - https://phabricator.wikimedia.org/T354128 (10Earwig) @dcaro We've added the `.locales` file, but we now get a build failure due to a missing package (`language-pack-zh-base`): ` [step-build] 20... [19:53:04] (PuppetAgentNoResources) firing: No Puppet resources found on instance toolsbeta-bastion-6 on project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [20:05:35] PROBLEM - Wikitech-static main page has content on wikitech-static.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Wikitech-static [20:13:19] RECOVERY - Wikitech-static main page has content on wikitech-static.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 26874 bytes in 1.681 second response time https://wikitech.wikimedia.org/wiki/Wikitech-static [21:55:38] 10Grid-Engine-to-K8s-Migration: Migrate dvorapabot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319709 (10Dvorapa) a:03Dvorapa [22:45:22] (HAProxyBackendUnavailable) firing: HAProxy service neutron-api_backend backend cloudcontrol1007.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [22:50:22] (HAProxyBackendUnavailable) resolved: HAProxy service neutron-api_backend backend cloudcontrol1007.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [22:58:04] (PuppetAgentNoResources) firing: No Puppet resources found on instance toolsbeta-bastion-6 on project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [23:18:03] (InstanceDown) firing: Project tools instance tools-k8s-worker-84 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [23:22:23] (ToolforgeKubernetesNodeNotReady) firing: Kubernetes node tools-k8s-worker-84 is not ready - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesNodeNotReady - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesNodeNotReady [23:32:23] (ToolforgeKubernetesNodeNotReady) resolved: Kubernetes node tools-k8s-worker-84 is not ready - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesNodeNotReady - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesNodeNotReady [23:33:03] (InstanceDown) resolved: Project tools instance tools-k8s-worker-84 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown