[02:02:19] (HAProxyBackendUnavailable) firing: HAProxy service neutron-api_backend backend cloudcontrol1007.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [02:07:20] (HAProxyBackendUnavailable) resolved: HAProxy service neutron-api_backend backend cloudcontrol1007.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [04:02:03] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [07:02:03] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [08:44:19] (HAProxyBackendUnavailable) firing: HAProxy service neutron-api_backend backend cloudcontrol1007.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [08:49:19] (HAProxyBackendUnavailable) resolved: HAProxy service neutron-api_backend backend cloudcontrol1007.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [09:53:04] 10Grid-Engine-to-K8s-Migration: Migrate mathis-bot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319880 (10Mathis_Benguigui) 05In progress→03Resolved Migration done. [09:57:40] 10Grid-Engine-to-K8s-Migration: Migrate naggobot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319917 (10El_pitareio) Migration to k8s completed. [10:02:03] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [11:11:49] 10Grid-Engine-to-K8s-Migration: Migrate naggobot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319917 (10taavi) 05Open→03Resolved a:03taavi Thank you! [11:13:43] 10Grid-Engine-to-K8s-Migration: Migrate naggobot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319917 (10taavi) a:05taavi→03None [12:20:56] (ToolsToolsDBReplicationError) firing: ToolsDB replication is broken on tools-db-2 (errno 1595) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBReplication - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsToolsDBReplicationError [12:20:56] (ToolsToolsDBReplicationMissing) firing: ToolsDB replication is not running on tools-db-1 (errno 0) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBReplication - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsToolsDBReplicationMissing [12:30:56] (ToolsToolsDBReplicationError) resolved: ToolsDB replication is broken on tools-db-2 (errno 1595) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBReplication - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsToolsDBReplicationError [12:30:56] (ToolsToolsDBReplicationMissing) resolved: ToolsDB replication is not running on tools-db-1 (errno 0) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBReplication - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsToolsDBReplicationMissing [12:33:56] (ToolsToolsDBReplicationLagIsTooHigh) firing: ToolsDB replication on tools-db-2 is lagging behind the primary, the current lag is 3709 - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBReplication - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsToolsDBReplicationLagIsTooHigh [13:02:03] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [13:22:10] 10Tool-Pageviews, 10Data-Engineering, 10Pageviews-API: 429 Too Many Requests hit despite throttling to 100 req/sec - https://phabricator.wikimedia.org/T219857 (10TheDJ) @MusikAnimal is this still an issue ? Since there hasn't happened anything in this ticket for 3 years (if you ignore the workboard/team shuf... [14:07:55] 10VPS-project-Codesearch, 10Patch-For-Review: Remove WikiMANNia repositories from MediaWiki code search - https://phabricator.wikimedia.org/T323956 (10UlfDunkel) I see no connection between the possibility that people use the MediaWiki engine to set up a wiki that also conveys opinions, statements and claims t... [14:15:56] 10Cloud-VPS, 10cloud-services-team (Hardware), 10SRE, 10ops-eqiad: Cloudvirt1063.eqiad.wmnet overheating - https://phabricator.wikimedia.org/T353408 (10Jclark-ctr) @Andrew Dell would like to replace cpu and reapply thermal paste they would like to preform service today is server still down? [15:18:21] 10Cloud-VPS, 10cloud-services-team (Hardware), 10SRE, 10ops-eqiad: Cloudvirt1063.eqiad.wmnet overheating - https://phabricator.wikimedia.org/T353408 (10taavi) >>! In T353408#9423493, @Jclark-ctr wrote: > @Andrew Dell would like to replace cpu and reapply thermal paste they would like to preform service t... [15:33:56] (ToolsToolsDBReplicationLagIsTooHigh) firing: ToolsDB replication on tools-db-2 is lagging behind the primary, the current lag is 8264 - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBReplication - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsToolsDBReplicationLagIsTooHigh [16:02:03] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [16:07:05] vivian-rook opened https://github.com/toolforge/superset-deploy/pull/14 [16:10:49] vivian-rook closed https://github.com/toolforge/superset-deploy/pull/14 [16:28:04] 10Grid-Engine-to-K8s-Migration: Migrate assessor from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319577 (10Edgars2007) 05Open→03Resolved a:03Edgars2007 [16:28:18] 10Grid-Engine-to-K8s-Migration: Migrate assessor from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319577 (10Edgars2007) moved to toolforge jobs. [16:36:35] 10Grid-Engine-to-K8s-Migration: Migrate enboten from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319721 (10Lejonel) 05Open→03Resolved a:03Lejonel Fixed following the instructions at https://wikitech.wikimedia.org/wiki/Help:Toolforge/Running_Pywikibot_scripts_(advanced) [16:38:30] 10Grid-Engine-to-K8s-Migration: Migrate booster from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319600 (10Edgars2007) moved to toolforge jobs. @komla: do i have to anything with the TOOL_DISABLED file? [16:38:47] 10Grid-Engine-to-K8s-Migration: Migrate booster from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319600 (10Edgars2007) a:03Edgars2007 [16:44:57] 10Grid-Engine-to-K8s-Migration: Migrate completer from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319643 (10Edgars2007) 05Open→03Resolved a:03Edgars2007 moved to toolforge jobs. [16:48:30] 10Grid-Engine-to-K8s-Migration: Migrate redirtalkdeleter from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319998 (10Lejonel) https://wikitech.wikimedia.org/wiki/Help:Toolforge/Running_Pywikibot_scripts_(advanced) might be better for running your own pywikibot scripts. > ` >... [16:53:56] (ToolsToolsDBReplicationLagIsTooHigh) resolved: ToolsDB replication on tools-db-2 is lagging behind the primary, the current lag is 5319 - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBReplication - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsToolsDBReplicationLagIsTooHigh [16:55:26] (ToolsToolsDBReplicationLagIsTooHigh) firing: ToolsDB replication on tools-db-2 is lagging behind the primary, the current lag is 5425 - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBReplication - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsToolsDBReplicationLagIsTooHigh [17:35:15] 10VPS-project-Codesearch, 10Special:NewLexeme revival, 10wmde-wikidata-tech: Please add wmde/new-lexeme-special-page to codesearch index - https://phabricator.wikimedia.org/T351938 (10Lucas_Werkmeister_WMDE) I don’t see the repo in codesearch yet, does the config change need to be deployed or something? (FWI... [17:38:03] 10VPS-project-Codesearch, 10Special:NewLexeme revival, 10wmde-wikidata-tech: Please add wmde/new-lexeme-special-page to codesearch index - https://phabricator.wikimedia.org/T351938 (10Ladsgroup) It should be deployed automatically. I'll check if something is broken. [19:01:54] PROBLEM - Host cloudvirt1063 is DOWN: PING CRITICAL - Packet loss = 100% [19:02:03] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [19:03:35] ACKNOWLEDGEMENT - Host cloudvirt1063 is DOWN: PING CRITICAL - Packet loss = 100% Andrew Bogott replacing CPU, T353408 [19:17:17] 10Grid-Engine-to-K8s-Migration: Migrate dawikibot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319661 (10Steenth) I need a copy of crontab to work with migration. My plan to use crontab to create a new crontab that uses Toolforge Kubernetes. [19:18:35] 10Grid-Engine-to-K8s-Migration: Migrate dawikitool from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319662 (10Steenth) I need a copy of crontab to work with migration. My plan to use crontab to create a new crontab that uses Toolforge Kubernetes. [19:23:12] RECOVERY - Host cloudvirt1063 is UP: PING WARNING - Packet loss = 60%, RTA = 1510.45 ms [19:25:13] 10Grid-Engine-to-K8s-Migration: Migrate dawikibot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319661 (10taavi) >>! In T319661#9423998, @Steenth wrote: > I need a copy of crontab to work with migration. My plan to use crontab to create a new crontab that uses Toolforge K... [19:55:26] (ToolsToolsDBReplicationLagIsTooHigh) firing: ToolsDB replication on tools-db-2 is lagging behind the primary, the current lag is 4088 - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBReplication - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsToolsDBReplicationLagIsTooHigh [20:20:26] (ToolsToolsDBReplicationLagIsTooHigh) resolved: ToolsDB replication on tools-db-2 is lagging behind the primary, the current lag is 3620 - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBReplication - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsToolsDBReplicationLagIsTooHigh [22:02:03] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [22:27:50] (PawsJupyterHubDown) firing: PAWS JupyterHub is down https://wikitech.wikimedia.org/wiki/PAWS/Admin - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPawsJupyterHubDown [22:32:50] (PawsJupyterHubDown) resolved: PAWS JupyterHub is down https://wikitech.wikimedia.org/wiki/PAWS/Admin - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPawsJupyterHubDown [22:41:27] 10Toolforge Build Service: apt buildpack (Aptfile support) doesn’t really work - https://phabricator.wikimedia.org/T353847 (10LucasWerkmeister) This is kind of a strange task (hence the bleh title). I suspect several of the issues mentioned in the task description aren’t really fixable without fundamentally chan...