[00:09:03] (InstanceDown) firing: Project tf-infra-test instance tf-infra-test is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [00:09:34] (TfInfraTestApplyFailed) resolved: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [00:10:03] (TfInfraTestDestroyFailed) firing: Terraform failed to destroy the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestDestroyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestDestroyFailed [00:13:33] (SystemdUnitDown) firing: The service unit purge_vm_backup.service is in failed status on host cloudbackup1003. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudbackup1003 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [00:16:41] PROBLEM - Check unit status of purge_vm_backup on cloudbackup1003 is CRITICAL: CRITICAL: Status of the systemd unit purge_vm_backup https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [01:32:42] (OpenstackAPIResponse) firing: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [02:05:19] (HAProxyBackendUnavailable) firing: HAProxy service neutron-api_backend backend cloudcontrol1006.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [02:08:33] (SystemdUnitDownForLong) firing: The systemd unit purge_vm_backup.service on node cloudbackup1003 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDownForLong - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudbackup1003 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDownForLong [02:08:39] 10cloud-services-team: SystemdUnitDownForLong cloudbackup1003:9100 Unit purge_vm_backup.service on node cloudbackup1003 has been down for long. - https://phabricator.wikimedia.org/T352132 (10phaultfinder) [02:10:19] (HAProxyBackendUnavailable) resolved: HAProxy service neutron-api_backend backend cloudcontrol1006.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [03:03:37] (CephSlowOps) firing: Ceph cluster in eqiad has 26 slow ops - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephSlowOps - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephSlowOps [03:03:42] 10cloud-services-team: CephSlowOps Ceph cluster in eqiad has slow ops, which might be blocking some writes - https://phabricator.wikimedia.org/T352082 (10phaultfinder) [03:07:37] (CephClusterInWarning) firing: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [03:09:03] (InstanceDown) firing: Project tf-infra-test instance tf-infra-test is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [03:10:03] (TfInfraTestDestroyFailed) firing: Terraform failed to destroy the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestDestroyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestDestroyFailed [03:12:37] (CephClusterInWarning) resolved: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [03:13:37] (CephSlowOps) resolved: Ceph cluster in eqiad has 1 slow ops - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephSlowOps - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephSlowOps [04:12:31] 10Cloud-Services, 10MinT, 10Language-Team (Language-2023-October-December): Create large instance for MinT - https://phabricator.wikimedia.org/T352136 (10KartikMistry) The #Cloud-Services project tag is not intended to have any tasks. Please check the list on https://phabricator.wikimedia.org/project/profile... [04:13:37] 10Cloud-VPS (Project-requests), 10MinT, 10Language-Team (Language-2023-October-December): Create large instance for MinT - https://phabricator.wikimedia.org/T352136 (10KartikMistry) [05:14:25] 10Cloud-VPS (Project-requests), 10MinT, 10Language-Team (Language-2023-October-December): Create large instance for MinT - https://phabricator.wikimedia.org/T352136 (10KartikMistry) [05:32:42] (OpenstackAPIResponse) firing: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [05:59:52] PROBLEM - Check unit status of backup_cinder_volumes on cloudbackup2001 is CRITICAL: CRITICAL: Status of the systemd unit backup_cinder_volumes https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [06:08:49] (SystemdUnitDownForLong) firing: The systemd unit purge_vm_backup.service on node cloudbackup1003 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDownForLong - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudbackup1003 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDownForLong [06:09:03] (InstanceDown) firing: Project tf-infra-test instance tf-infra-test is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [06:10:03] (TfInfraTestDestroyFailed) firing: Terraform failed to destroy the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestDestroyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestDestroyFailed [09:09:03] (InstanceDown) firing: Project tf-infra-test instance tf-infra-test is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [09:10:03] (TfInfraTestDestroyFailed) firing: Terraform failed to destroy the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestDestroyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestDestroyFailed [09:22:31] 10Toolforge (Toolforge iteration 02): [builds-api] Use admin user credentials for Harbor API auth in dev - https://phabricator.wikimedia.org/T352022 (10Slst2020) [09:22:37] 10Tool-bub2, 10Internet-Archive, 10Outreachy (Round 27): Proposal: Integrate Wikimedia Ecosystem within BUB2 tool - https://phabricator.wikimedia.org/T352150 (10Okerekechinweotito) [09:23:16] 10Toolforge (Toolforge iteration 02): [builds-api] Use admin user credentials for Harbor API auth in dev - https://phabricator.wikimedia.org/T352022 (10Slst2020) [09:24:51] 10Tool-bub2, 10Internet-Archive, 10Outreachy (Round 27): Proposal: Integrate Wikimedia Ecosystem within BUB2 tool - https://phabricator.wikimedia.org/T352150 (10Okerekechinweotito) [09:27:11] 10Tool-bub2, 10Internet-Archive, 10Outreachy (Round 27): Proposal: Integrate Wikimedia Ecosystem within BUB2 tool - https://phabricator.wikimedia.org/T352150 (10Okerekechinweotito) [09:28:29] 10Toolforge (Toolforge iteration 02), 10Patch-For-Review: [builds-api] Use admin user credentials for Harbor API auth in dev - https://phabricator.wikimedia.org/T352022 (10CodeReviewBot) sstefanova opened https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/61 dev: Use admin creds in... [09:32:42] (OpenstackAPIResponse) firing: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [09:41:16] 10cloud-services-team, 10Infrastructure-Foundations, 10SRE-tools, 10Spicerack: [spicerack] Add remote command output to log file - https://phabricator.wikimedia.org/T347093 (10Volans) 05Open→03Declined As there is already a workaround to do that in the cookbooks on demand and it will be even simpler wi... [09:52:58] 10Toolforge (Toolforge iteration 02), 10Patch-For-Review: [builds-api] Use admin user credentials for Harbor API auth in dev - https://phabricator.wikimedia.org/T352022 (10CodeReviewBot) sstefanova merged https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/61 dev: Use admin creds in... [10:08:49] (SystemdUnitDownForLong) firing: The systemd unit purge_vm_backup.service on node cloudbackup1003 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDownForLong - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudbackup1003 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDownForLong [10:09:57] (SystemdUnitDownForLong) firing: The systemd unit purge_vm_backup.service on node cloudbackup1003 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDownForLong - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudbackup1003 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDownForLong [10:09:57] (OpenstackAPIResponse) firing: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [10:10:02] 10cloud-services-team: SystemdUnitDownForLong Unit purge_vm_backup.service on node cloudbackup1003 has been down for long. - https://phabricator.wikimedia.org/T352158 (10phaultfinder) [10:21:21] 10Cloud-VPS (Project-requests), 10MinT, 10Language-Team (Language-2023-October-December): Create large instance for MinT - https://phabricator.wikimedia.org/T352136 (10santhosh) We need 2TB scratch volume mounted too. [10:28:31] 10Cloud-VPS (Quota-requests), 10MinT, 10Language-Team (Language-2023-October-December): Create large instance for MinT - https://phabricator.wikimedia.org/T352136 (10taavi) Please fill in the template from #cloud-vps-quota-requests. [10:28:58] 10cloud-services-team: CephSlowOps Ceph cluster in eqiad has slow ops, which might be blocking some writes - https://phabricator.wikimedia.org/T352082 (10taavi) 05Open→03Resolved a:03taavi [10:29:12] 10cloud-services-team: SystemdUnitDownForLong cloudbackup1003:9100 Unit purge_vm_backup.service on node cloudbackup1003 has been down for long. - https://phabricator.wikimedia.org/T352132 (10taavi) 05Open→03Resolved a:03taavi [11:42:03] (WidespreadPuppetAgentFailure) firing: Widespread puppet agent failures in project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DWidespreadPuppetAgentFailure [11:57:03] (PuppetAgentFailure) firing: Puppet agent failure detected on instance cloudinfra-internal-puppetmaster-02 in project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [12:09:03] (InstanceDown) firing: Project tf-infra-test instance tf-infra-test is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [12:10:03] (TfInfraTestDestroyFailed) firing: Terraform failed to destroy the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestDestroyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestDestroyFailed [12:12:03] (PuppetAgentFailure) resolved: Puppet agent failure detected on instance cloudinfra-internal-puppetmaster-02 in project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [12:14:12] 10PAWS: tofu state file to object storage - https://phabricator.wikimedia.org/T352164 (10rook) [13:08:59] (SystemdUnitDown) firing: The service unit backup_glance_images.service is in failed status on host cloudbackup1003. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudbackup1003 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [13:10:31] PROBLEM - Check unit status of backup_glance_images on cloudbackup1003 is CRITICAL: CRITICAL: Status of the systemd unit backup_glance_images https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [13:22:20] (ProbeDown) firing: Service toolserver-proxy-01:443 has failed probes (http_toolserver_org_redirects_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#toolserver-proxy-01:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [13:27:20] (ProbeDown) resolved: Service toolserver-proxy-01:443 has failed probes (http_toolserver_org_redirects_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#toolserver-proxy-01:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [13:44:51] 10Toolforge: Toolforge Kubernetes quota requests.memory was reduced - https://phabricator.wikimedia.org/T352055 (10taavi) p:05Triage→03High a:03taavi Good catch, I did indeed miss this when working on the quota increases. As far as I can tell, there are three options on how to deal with this: 1. Increase t... [14:09:57] (OpenstackAPIResponse) firing: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [14:09:57] (SystemdUnitDownForLong) firing: The systemd unit purge_vm_backup.service on node cloudbackup1003 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDownForLong - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudbackup1003 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDownForLong [15:04:00] (SystemdUnitDownForLong) firing: (2) The systemd unit backup_glance_images.service on node cloudbackup1003 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDownForLong - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudbackup1003 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDownForLong [15:04:05] 10cloud-services-team: SystemdUnitDownForLong - https://phabricator.wikimedia.org/T352185 (10phaultfinder) [15:09:03] (InstanceDown) firing: Project tf-infra-test instance tf-infra-test is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [15:10:03] (TfInfraTestDestroyFailed) firing: Terraform failed to destroy the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestDestroyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestDestroyFailed [16:23:42] 10Toolforge, 10cloud-services-team: Migrate remaining tools off Gridengine - https://phabricator.wikimedia.org/T313405 (10nskaggs) [16:52:37] 10Toolforge (Software install/update): Create a kubernetes container with mono and dotnet - https://phabricator.wikimedia.org/T311466 (10bd808) >>! In T311466#9353995, @Hawkeye7 wrote: > I have multiple C# projects in the one git repository. It was never intended to be built in this manner. These are reasonable... [16:57:30] 10Toolforge: Toolforge Kubernetes quota requests.memory was reduced - https://phabricator.wikimedia.org/T352055 (10Bamyers99) The reason it uses the "unstable" interface is because it dynamically dispatches jobs from a pod. It used to use the Jobs framework api, but access to this from a pod was blocked sometime... [17:01:26] 10Toolforge (Software install/update): Create a kubernetes container with mono and dotnet - https://phabricator.wikimedia.org/T311466 (10bd808) >>! In T311466#9364202, @bd808 wrote: > This reads to me like splitting your "mono/Liftwing/" directory into a dedicated git repo would make the Toolforge supported syst... [17:22:54] 10Cloud-VPS, 10cloud-services-team (FY2023/2024-Q1-Q2): [openstack] Fix ceph-common version in Bookworm - https://phabricator.wikimedia.org/T350188 (10fnegri) 05In progress→03Declined We opted for the simple solution: upgrading to ceph-common v16 in both cloudvirt* and cloudcontrol* eqiad hosts, as we did... [17:23:00] 10Cloud-VPS, 10cloud-services-team (FY2023/2024-Q1-Q2): [openstack] Upgrade eqiad hosts to bookworm - https://phabricator.wikimedia.org/T345811 (10fnegri) [17:23:07] 10PAWS: Remove db_password variable - https://phabricator.wikimedia.org/T351255 (10github-toolforge-bot) vivian-rook opened https://github.com/toolforge/paws/pull/352 [17:23:13] vivian-rook opened https://github.com/toolforge/paws/pull/352 [17:32:25] 10PAWS: Remove db_password variable - https://phabricator.wikimedia.org/T351255 (10github-toolforge-bot) vivian-rook closed https://github.com/toolforge/paws/pull/352 [17:32:29] vivian-rook closed https://github.com/toolforge/paws/pull/352 [17:32:40] 10PAWS: Remove db_password variable - https://phabricator.wikimedia.org/T351255 (10rook) 05Open→03Resolved [17:41:23] vivian-rook opened https://github.com/toolforge/paws/pull/353 [17:41:24] 10PAWS: update opentofu version - https://phabricator.wikimedia.org/T351402 (10github-toolforge-bot) vivian-rook opened https://github.com/toolforge/paws/pull/353 [17:53:30] 10Data-Services: [toolsdb] Upgrade to MariaDB 10.6 - https://phabricator.wikimedia.org/T352206 (10fnegri) [17:53:39] 10Data-Services: [toolsdb] Upgrade to MariaDB 10.6 - https://phabricator.wikimedia.org/T352206 (10fnegri) p:05Triage→03Medium [17:54:19] 10Data-Services, 10cloud-services-team (FY2023/2024-Q1-Q2), 10Cloud-Services-Origin-Alert: [toolsdb] MariaDB process is killed by OOM killer (October 2023) - https://phabricator.wikimedia.org/T349695 (10fnegri) 05In progress→03Resolved Two weeks later, the free memory is quite stable around 12GB, and has... [18:09:03] (InstanceDown) firing: Project tf-infra-test instance tf-infra-test is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [18:10:03] (TfInfraTestDestroyFailed) firing: Terraform failed to destroy the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestDestroyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestDestroyFailed [18:13:59] (OpenstackAPIResponse) firing: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [18:39:03] (InstanceDown) resolved: Project tf-infra-test instance tf-infra-test is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [18:40:03] (TfInfraTestDestroyFailed) resolved: Terraform failed to destroy the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestDestroyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestDestroyFailed [18:51:43] 10Quarry: [bug] query/77794: "This query was stopped" - https://phabricator.wikimedia.org/T352211 (10Boshomi_Phabricator) [19:12:47] 10Quarry: [feedback] Allow search within SQL - https://phabricator.wikimedia.org/T352212 (10Boshomi_Phabricator) [20:32:30] (03PS1) 10Hashar: Add CORS header for the json API [labs/tools/train-blockers] - 10https://gerrit.wikimedia.org/r/978139 [20:42:19] (03CR) 10Majavah: [C: 04-1] "thanks! see inline" [labs/tools/train-blockers] - 10https://gerrit.wikimedia.org/r/978139 (owner: 10Hashar) [22:14:00] (OpenstackAPIResponse) firing: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [23:29:05] 10Toolforge (Software install/update): Create a kubernetes container with mono and dotnet - https://phabricator.wikimedia.org/T311466 (10Hawkeye7) As you say, the biggest obstacle is that there is no dotnet buildpack available. I don't know if any work has been done on this. Or indeed whether you are proceedin... [23:53:39] 10Toolforge (Software install/update): Create a kubernetes container with mono and dotnet - https://phabricator.wikimedia.org/T311466 (10bd808) >>! In T311466#9365479, @Hawkeye7 wrote: > As you say, the biggest obstacle is that there is no dotnet buildpack available. > > I don't know if any work has been done...