[00:15:03] (InstanceDown) firing: Project tf-infra-test instance tf-infra-test is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [00:20:03] (InstanceDown) resolved: Project tf-infra-test instance tf-infra-test is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [00:29:58] 10Toolforge (Software install/update): Create a kubernetes container with mono and dotnet - https://phabricator.wikimedia.org/T311466 (10Hawkeye7) I don't quite understand this. I know how to build locally: pack build liftwing --buildpack paketo-buildpacks/dotnet-core --builder paketobuildpacks/builder:base... [00:51:04] (InstanceDown) firing: Project toolsbeta instance toolsbeta-bastion-6 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [01:04:28] 10Toolforge (Software install/update): Create a kubernetes container with mono and dotnet - https://phabricator.wikimedia.org/T311466 (10bd808) >>! In T311466#9414549, @Hawkeye7 wrote: > But how will a toolforge build start know to user the dotnet buildpack? The buildpack that @dcaro ended up adding is https://... [01:29:28] (OpenstackAPIResponse) resolved: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [01:29:56] (OpenstackAPIResponse) firing: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [01:34:42] (OpenstackAPIResponse) resolved: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [01:43:14] 10Striker, 10GitLab (Auth & Access), 10Patch-For-Review, 10Release-Engineering-Team (Radar): Automatically approve GitLab accounts created by Striker integration - https://phabricator.wikimedia.org/T344667 (10CodeReviewBot) bd808 opened https://gitlab.wikimedia.org/toolforge-repos/gitlab-account-approval/-... [02:50:04] (TfInfraTestDestroyFailed) firing: Terraform failed to destroy the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestDestroyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestDestroyFailed [02:55:04] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [03:33:37] (CephSlowOps) firing: Ceph cluster in eqiad has 87 slow ops - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephSlowOps - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephSlowOps [03:33:41] 10cloud-services-team: CephSlowOps Ceph cluster in eqiad has slow ops, which might be blocking some writes - https://phabricator.wikimedia.org/T352570 (10phaultfinder) [03:38:37] (CephSlowOps) resolved: Ceph cluster in eqiad has 33 slow ops - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephSlowOps - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephSlowOps [03:51:04] (InstanceDown) firing: Project toolsbeta instance toolsbeta-bastion-6 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [05:50:04] (TfInfraTestDestroyFailed) firing: Terraform failed to destroy the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestDestroyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestDestroyFailed [05:55:04] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [06:51:04] (InstanceDown) firing: Project toolsbeta instance toolsbeta-bastion-6 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [08:50:04] (TfInfraTestDestroyFailed) firing: Terraform failed to destroy the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestDestroyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestDestroyFailed [08:55:04] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [09:22:21] 10Toolforge, 10Fiwiki-Wikidata-Commons, 10Documentation: Django socialauth OAUTH login fails with mediawiki backend - https://phabricator.wikimedia.org/T353593 (10Zache) [09:41:24] 10Quarry: Timer that counts up as the query is running - https://phabricator.wikimedia.org/T353690 (10Novem_Linguae) [09:51:04] (InstanceDown) firing: Project toolsbeta instance toolsbeta-bastion-6 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [10:12:14] 10Toolforge (Quota-requests): Request increased quota for anchor-corrector Toolforge tool - https://phabricator.wikimedia.org/T350484 (10Kanashimi) @taavi I initially intended to execute the three tasks sequentially. However, I later realized that `wait’ is not suitable for this purpose. Consequently, I had to r... [10:42:00] 10Toolforge (Quota-requests): Request increased quota for anchor-corrector Toolforge tool - https://phabricator.wikimedia.org/T350484 (10Kanashimi) 05Openβ†’03Resolved [10:42:02] 10Grid-Engine-to-K8s-Migration: Migrate anchor-corrector from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319555 (10Kanashimi) [10:42:49] 10Toolforge (Quota-requests): Request increased quota for cewbot, toc, signature-checker, mgp-cewbot Toolforge tool - https://phabricator.wikimedia.org/T353104 (10Kanashimi) Thank you for your help. The anchor-corrector looks good now. [11:15:54] 10VPS-project-Phabricator, 10Phabricator, 10Release-Engineering-Team (Escape Goats🐐), 10User-brennen: After a deployment, Phabricator errors out with `Unable to load the "Arcanist" library. Put "arcanist/" next to "phabricator/" on disk.` - https://phabricator.wikimedia.org/T314460 (10Aklapper) >>! In T314... [11:28:19] (HAProxyBackendUnavailable) firing: HAProxy service neutron-api_backend backend cloudcontrol1006.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [11:33:19] (HAProxyBackendUnavailable) resolved: HAProxy service neutron-api_backend backend cloudcontrol1006.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [11:50:04] (TfInfraTestDestroyFailed) firing: Terraform failed to destroy the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestDestroyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestDestroyFailed [11:55:04] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [12:51:04] (InstanceDown) firing: Project toolsbeta instance toolsbeta-bastion-6 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [13:06:46] 10Cloud-VPS (Quota-requests), 10cloud-services-team, 10GitLab-Test, 10Release-Engineering-Team, 10collaboration-services: Request additional resources for devtools project - https://phabricator.wikimedia.org/T353671 (10Jelto) [13:11:14] (DiskSpace) firing: Disk space cloudbackup2001:9100:/srv/cinder-backups 5.974% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup2001 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [13:17:58] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api [13:18:11] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api [13:23:14] 10Toolforge (Toolforge iteration 02), 10Patch-For-Review: Add `toolforge build quota` command - https://phabricator.wikimedia.org/T341068 (10taavi) 05Resolvedβ†’03Open This does not work on Toolforge: `lang=shell-session tools.taavi-test-tool@tools-sgebastion-11:~$ toolforge build quota BuildClientError: Err... [13:23:19] 10Toolforge (Toolforge iteration 02): [tbs] Improve Harbor quota handling and docs - https://phabricator.wikimedia.org/T351092 (10taavi) [13:30:46] 10Cloud-VPS, 10Toolforge Build Service, 10cloud-services-team: dynamicproxy client_max_body_size blocks large Harbor uploads - https://phabricator.wikimedia.org/T353698 (10taavi) [13:45:07] 10Grid-Engine-to-K8s-Migration: Migrate wd-shex-infer from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320140 (10taavi) [13:45:57] 10Cloud-VPS, 10Toolforge Build Service, 10cloud-services-team, 10Patch-For-Review: dynamicproxy client_max_body_size blocks large Harbor uploads - https://phabricator.wikimedia.org/T353698 (10taavi) 05Openβ†’03Resolved Increasing the max size to 512M worked for this specific case. I'm sure some other too... [13:47:09] 10Toolforge (Toolforge iteration 02), 10Toolforge Build Service: Add `toolforge build quota` command - https://phabricator.wikimedia.org/T341068 (10taavi) [13:48:28] 10Toolforge Build Service: `build quota` fails if tool has no builds - https://phabricator.wikimedia.org/T353701 (10taavi) [13:48:43] 10Toolforge (Toolforge iteration 02): [tbs] Improve Harbor quota handling and docs - https://phabricator.wikimedia.org/T351092 (10taavi) [13:48:46] 10cloud-services-team (FY2023/2024-Q1-Q2): openstack: eqiad1: cleanup leaks from the cloudlb migration - https://phabricator.wikimedia.org/T346630 (10fnegri) [13:48:52] 10Toolforge Build Service: `build quota` fails if tool has no builds - https://phabricator.wikimedia.org/T353701 (10taavi) [13:49:04] 10Toolforge (Toolforge iteration 02), 10Toolforge Build Service: Add `toolforge build quota` command - https://phabricator.wikimedia.org/T341068 (10taavi) 05Openβ†’03Resolved Nevermind, filed {T353701} for that. [13:49:15] 10Toolforge Build Service: `build quota` fails if tool has no builds - https://phabricator.wikimedia.org/T353701 (10taavi) [14:05:23] 10cloud-services-team, 10Infrastructure-Foundations, 10SRE, 10netbox, 10Patch-For-Review: Netbox: Add support for our complex host network setups in provision script - https://phabricator.wikimedia.org/T346428 (10Volans) Thanks for the patch, it would take a bit to do a full pass given the size. I agree... [14:48:52] (03PS1) 10Krinkle: write_config: Add operations/software/benchmw [labs/codesearch] - 10https://gerrit.wikimedia.org/r/984208 [14:50:28] (03CR) 10Krinkle: [C: 03+2] write_config: Add operations/software/benchmw [labs/codesearch] - 10https://gerrit.wikimedia.org/r/984208 (owner: 10Krinkle) [14:51:32] (03Merged) 10jenkins-bot: write_config: Add operations/software/benchmw [labs/codesearch] - 10https://gerrit.wikimedia.org/r/984208 (owner: 10Krinkle) [14:55:04] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [14:55:04] (TfInfraTestDestroyFailed) firing: Terraform failed to destroy the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestDestroyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestDestroyFailed [15:22:56] 10Grid-Engine-to-K8s-Migration: Migrate comidentgen from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319637 (10komla) 05Openβ†’03Invalid deleted tool [15:23:33] 10Grid-Engine-to-K8s-Migration: Migrate dashboard from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319658 (10komla) 05Openβ†’03Invalid deleted tool [15:24:00] 10Cloud-VPS, 10cloud-services-team (Hardware), 10SRE, 10ops-eqiad: Cloudvirt1063.eqiad.wmnet overheating - https://phabricator.wikimedia.org/T353408 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=0cee941c-9871-4463-b392-d45794163f4d) set by taavi@cumin1001 for 30 days, 0:00:00 on 1 hos... [15:24:09] 10Grid-Engine-to-K8s-Migration: Migrate declare from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319666 (10komla) 05Openβ†’03Invalid deleted tool [15:32:28] 10Grid-Engine-to-K8s-Migration: Migrate devlibrarycard from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319672 (10komla) 05Openβ†’03Invalid deleted tool [15:33:47] 10Grid-Engine-to-K8s-Migration: Migrate edits from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319719 (10komla) 05Openβ†’03Invalid deleted tool [15:37:08] 10Grid-Engine-to-K8s-Migration: Migrate loltrs from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319865 (10komla) 05Openβ†’03Invalid deleted tool [15:37:54] 10Grid-Engine-to-K8s-Migration: Migrate ores-afc from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319938 (10komla) 05Openβ†’03Invalid deleted tool [15:39:17] 10Grid-Engine-to-K8s-Migration: Migrate rank4 from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319992 (10komla) 05Openβ†’03Invalid deleted tool [15:40:18] 10Grid-Engine-to-K8s-Migration: Migrate reasomics from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319995 (10komla) 05Openβ†’03Invalid deleted tool [15:41:56] 10Grid-Engine-to-K8s-Migration: Migrate signature-manquante-bot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320043 (10komla) 05Openβ†’03Invalid deleted tool [15:43:20] 10Grid-Engine-to-K8s-Migration: Migrate user-activity from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320113 (10komla) 05Openβ†’03Invalid deleted tool [15:44:44] 10Grid-Engine-to-K8s-Migration: Migrate useredit from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320114 (10komla) 05Openβ†’03Invalid deleted tool [15:45:43] 10Grid-Engine-to-K8s-Migration: Migrate useredits1 from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320115 (10komla) 05Openβ†’03Invalid deleted tool [15:47:11] 10cloud-services-team (FY2023/2024-Q1-Q2): openstack: eqiad1: cleanup leaks from the cloudlb migration - https://phabricator.wikimedia.org/T346630 (10taavi) a:03taavi [15:47:13] 10Grid-Engine-to-K8s-Migration: Migrate yemen from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320196 (10komla) 05Openβ†’03Invalid deleted tool [15:51:04] (InstanceDown) firing: Project toolsbeta instance toolsbeta-bastion-6 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [16:01:37] 10Grid-Engine-to-K8s-Migration, 10Pywikibot: Migrate pywikibot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319981 (10bd808) >>! In T319981#9404622, @JJMC89 wrote: > The nightly cronjobs need to be migrated from Grid Engine to the jobs framework. `lang=shell-session $... [17:04:00] 10Cloud-VPS: Cloud VPS custom provider does not have arm Mac os package - https://phabricator.wikimedia.org/T353736 (10Chicocvenancio) [17:10:27] 10Cloud-VPS: Cloud-VPS OpenTofu provider is not working on M1 Macs - https://phabricator.wikimedia.org/T353019 (10taavi) [17:10:38] 10Cloud-VPS: Cloud VPS custom provider does not have arm Mac os package - https://phabricator.wikimedia.org/T353736 (10taavi) [17:11:10] 10Cloud-VPS: Cloud-VPS OpenTofu provider is not working on M1 Macs - https://phabricator.wikimedia.org/T353019 (10taavi) If I recall correctly Go can cross-compile programs pretty easily. So the real problem to solve here is to automate the release process to make that happen. [17:11:28] (DiskSpace) firing: Disk space cloudbackup2001:9100:/srv/cinder-backups 4.958% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup2001 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [17:54:12] 10Toolforge (Toolforge iteration 02): [toolforge-cd] discuss the possibility of removing tests from merge request ci/cd pipelines - https://phabricator.wikimedia.org/T353740 (10Raymond_Ndibe) [17:55:04] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [17:55:04] (TfInfraTestDestroyFailed) firing: Terraform failed to destroy the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestDestroyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestDestroyFailed [17:57:09] 10Toolforge (Toolforge iteration 02), 10Patch-For-Review, 10User-Raymond_Ndibe: [gitlab,toolforge-deploy] Create a process to open an MR to toolforge-deploy when a new release ofa component happens - https://phabricator.wikimedia.org/T347392 (10Raymond_Ndibe) [17:57:15] 10Toolforge (Toolforge iteration 02): [toolforge,gitlab] ensure we have a release before creating the mr on toolforge-deploy - https://phabricator.wikimedia.org/T353425 (10Raymond_Ndibe) 05In progressβ†’03Resolved [18:00:58] 10Toolforge (Toolforge iteration 02), 10Toolforge Build Service, 10Patch-For-Review: Add Rust buildpack to Toolforge build service - https://phabricator.wikimedia.org/T337066 (10CodeReviewBot) raymond-ndibe merged https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/22 rust: ad... [18:02:33] 10Toolforge (Toolforge iteration 02), 10Toolforge Build Service, 10Patch-For-Review: Add Rust buildpack to Toolforge build service - https://phabricator.wikimedia.org/T337066 (10CodeReviewBot) project_1317_bot_df3177307bed93c3f34e421e26c86e38 opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforg... [18:12:00] 10Toolforge Build Service, 10Patch-For-Review: `build delete` gives a confusing error message on a non-existent build - https://phabricator.wikimedia.org/T353583 (10CodeReviewBot) raymond-ndibe merged https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/68 auth: Clarify error message [18:18:07] 10Toolforge Build Service, 10Patch-For-Review: `build delete` gives a confusing error message on a non-existent build - https://phabricator.wikimedia.org/T353583 (10CodeReviewBot) project_1317_bot_df3177307bed93c3f34e421e26c86e38 opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merg... [18:39:29] 10Toolforge (Toolforge iteration 02), 10Toolforge Build Service, 10Patch-For-Review: Add Rust buildpack to Toolforge build service - https://phabricator.wikimedia.org/T337066 (10bd808) [18:50:55] 10Toolforge Build Service: Add builder support for Perl runtime projects - https://phabricator.wikimedia.org/T353744 (10bd808) [18:51:04] (InstanceDown) firing: Project toolsbeta instance toolsbeta-bastion-6 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [18:55:37] (CephSlowOps) firing: Ceph cluster in eqiad has 11 slow ops - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephSlowOps - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephSlowOps [18:55:43] 10cloud-services-team: CephSlowOps Ceph cluster in eqiad has slow ops, which might be blocking some writes - https://phabricator.wikimedia.org/T352570 (10phaultfinder) [18:59:37] (CephClusterInWarning) firing: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [19:04:37] (CephClusterInWarning) resolved: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [19:05:37] (CephSlowOps) resolved: Ceph cluster in eqiad has 20 slow ops - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephSlowOps - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephSlowOps [19:36:05] 10Cloud-VPS, 10Moderator-Tools-Team (Kanban): enable lists.wikimedia.org or wikimedia.org email addresses to receive dmarc reports for *.wmflabs.org - https://phabricator.wikimedia.org/T352902 (10jsn.sherman) 05In progressβ†’03Declined The original request for an EDV record has not been addressed so I'm mark... [19:44:58] 10cloud-services-team, 10DC-Ops, 10ops-codfw: Test new hardware candidate for cloudbackup replacement - https://phabricator.wikimedia.org/T353746 (10Andrew) [20:34:17] 10Cloud-VPS (Quota-requests), 10cloud-services-team (Kanban), 10Community-Tech, 10WikiWho: Request increased quota for wikiwho Cloud VPS project (volume storage) - https://phabricator.wikimedia.org/T297446 (10Andrew) Just a note for posterity: given the lack of response to my latest comment I am excluding... [20:51:18] 10Toolforge, 10cloud-services-team: Do something to Toolforge tools with no non-blocked maintainers - https://phabricator.wikimedia.org/T320342 (10bd808) I think we should archive all of the currently unowned tools for sure. I would be fine with either an automated archiving of such tools in the future or a ch... [21:00:04] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [21:00:04] (TfInfraTestDestroyFailed) firing: Terraform failed to destroy the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestDestroyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestDestroyFailed [21:18:03] 10Toolforge (Software install/update): Create a kubernetes container with mono and dotnet - https://phabricator.wikimedia.org/T311466 (10Hawkeye7) @bd808 Thanks for that. I will ensure that there is a Program.cs element in the top level of the repository. [21:21:48] 10Toolforge (Toolforge iteration 02), 10Toolforge Build Service, 10Patch-For-Review: Add Rust buildpack to Toolforge build service - https://phabricator.wikimedia.org/T337066 (10CodeReviewBot) raymond-ndibe merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/160 build... [21:24:11] 10Toolforge Build Service, 10Patch-For-Review: `build delete` gives a confusing error message on a non-existent build - https://phabricator.wikimedia.org/T353583 (10CodeReviewBot) raymond-ndibe merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/161 builds-api: bump to... [21:51:04] (InstanceDown) firing: Project toolsbeta instance toolsbeta-bastion-6 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [22:34:01] PROBLEM - Disk space on cloudbackup2001 is CRITICAL: DISK CRITICAL - free space: /srv/cinder-backups 3163966 MB (3% inode=98%): https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space https://grafana.wikimedia.org/d/000000377/host-overview?var-server=cloudbackup2001&var-datasource=codfw+prometheus/ops [23:04:53] 10Toolforge (Toolforge iteration 02), 10Toolforge Build Service, 10Patch-For-Review: Add Rust buildpack to Toolforge build service - https://phabricator.wikimedia.org/T337066 (10CodeReviewBot) raymond-ndibe opened https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/45 [builds-cli]... [23:35:33] 10Toolforge Build Service, 10Upstream: Python buildpack does not detect requirements from pyproject.toml - https://phabricator.wikimedia.org/T353762 (10bd808)