[00:04:53] 10Toolforge (Toolforge iteration 02), 10Toolforge Build Service, 10Patch-For-Review: Add Rust buildpack to Toolforge build service - https://phabricator.wikimedia.org/T337066 (10Legoktm) 05Resolved→03Open Here are the issues I see so far: * rust-toolchain has a different format now, as a TOML file (and o... [00:10:03] (TfInfraTestApplyFailed) resolved: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [00:15:03] (InstanceDown) firing: Project tf-infra-test instance tf-infra-test is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [00:20:03] (InstanceDown) resolved: Project tf-infra-test instance tf-infra-test is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [04:05:33] (SystemdUnitDown) firing: The service unit nova-fullstack.service is in failed status on host cloudcontrol1006. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1006 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [04:20:33] (SystemdUnitDown) resolved: The service unit nova-fullstack.service is in failed status on host cloudcontrol1006. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1006 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [05:17:04] 10Grid-Engine-to-K8s-Migration: Migrate potd from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319974 (10Legoktm) Manually fixed a few issues and sent out today's mail, will see that it works tomorrow before closing it. [05:26:26] 10Grid-Engine-to-K8s-Migration: Migrate afdstats from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319486 (10Legoktm) a:05Sigma→03Legoktm Thanks @Ahecht! We've moved the Git repository to https://gitlab.wikimedia.org/toolforge-repos/afdstats/ - I'm going to port over your... [05:42:55] 10Grid-Engine-to-K8s-Migration: Migrate afdstats from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319486 (10CodeReviewBot) legoktm merged https://gitlab.wikimedia.org/toolforge-repos/afdstats/-/merge_requests/1 Migrate to Python 3.9 + k8s [05:51:54] 10Grid-Engine-to-K8s-Migration: Migrate afdstats from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319486 (10Legoktm) 05Open→03Resolved ` tools.afdstats@tools-sgebastion-10:~$ webservice status Your webservice of type php7.4 is running on backend kubernetes ` [06:11:15] 10Grid-Engine-to-K8s-Migration: Migrate apersonbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319561 (10Legoktm) Started poking at this today, there's some Python stuff and some Rust things. The Rust stuff is statically linked with musl, so it should be trivial to move... [08:10:09] 10Toolforge (Toolforge iteration 02), 10Toolforge Build Service, 10Patch-For-Review: Add Rust buildpack to Toolforge build service - https://phabricator.wikimedia.org/T337066 (10dcaro) > Is there a reason you didn't use that one in the first place then? I'm a bit confused. Yes, using that buildpack means us... [08:19:44] 10Toolforge (Toolforge iteration 02), 10Toolforge Build Service, 10Patch-For-Review: Add Rust buildpack to Toolforge build service - https://phabricator.wikimedia.org/T337066 (10dcaro) 05Open→03Resolved Can you open separated tasks for each thing you need? You can already use rust as is, though it might... [08:20:54] 10Toolforge (Toolforge iteration 02), 10cloud-services-team (FY2023/2024-Q1-Q2), 10User-dcaro: [harbor] Redis using all available memory - https://phabricator.wikimedia.org/T354176 (10dcaro) Memory usage has started flattening out \o/, crossing fingers: {F41650292} [08:36:10] 10Grid-Engine-to-K8s-Migration: Migrate dplbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319701 (10dcaro) I can reproduce yes, the vars are set in the container that runs the application: ` tools.jupytest@tools-sgebastion-10:~$ kubectl get pods -o yaml | grep -A 2 TOO... [08:39:55] 10Grid-Engine-to-K8s-Migration, 10Toolforge (Toolforge iteration 02): [webservice] php 7.4 containers don't pass through the environment variables to the scripts - https://phabricator.wikimedia.org/T354320 (10dcaro) [08:40:23] 10Toolforge (Toolforge iteration 02): [webservice] php 7.4 containers don't pass through the environment variables to the scripts - https://phabricator.wikimedia.org/T354320 (10dcaro) [08:40:44] 10Toolforge (Toolforge iteration 02): [webservice] php 7.4 containers don't pass through the environment variables to the scripts - https://phabricator.wikimedia.org/T354320 (10dcaro) a:03dcaro [08:45:22] 10Toolforge (Toolforge iteration 02): [buildservice] Feature request - Indicate when long envvars are cutoff when listing - https://phabricator.wikimedia.org/T353287 (10Slst2020) @Amorymeltzer would this be better UX in your opinion? ` toolsbeta.tf-test@bullseye:~/envvars-cli$ toolforge envvars list --help Usag... [08:49:48] 10Toolforge (Toolforge iteration 02): [webservice] php 7.4 containers don't pass through the environment variables to the scripts - https://phabricator.wikimedia.org/T354320 (10dcaro) Yep, this is it: https://gerrit.wikimedia.org/r/plugins/gitiles/operations/software/tools-webservice/+/06e418c6952fcee64d2e63e6d7... [09:16:20] 10Toolforge (Toolforge iteration 02), 10Patch-For-Review: [webservice] php 7.4 containers don't pass through the environment variables to the scripts - https://phabricator.wikimedia.org/T354320 (10CodeReviewBot) dcaro opened https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/2... [09:20:30] 10Toolforge (Toolforge iteration 02), 10Patch-For-Review, 10User-dcaro: [envvars-cli] Ask for the contents of the envvar when setting it if not passed - https://phabricator.wikimedia.org/T354196 (10CodeReviewBot) dcaro merged https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/15... [09:43:43] 10Toolforge (Toolforge iteration 02), 10Patch-For-Review: [envvars-cli] use toolforge-weld for error handling - https://phabricator.wikimedia.org/T351459 (10CodeReviewBot) dcaro opened https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/17 d/changelog: bump to 0.0.3 [09:44:30] 10Toolforge (Toolforge iteration 02), 10Patch-For-Review, 10User-dcaro: [envvars-cli] Ask for the contents of the envvar when setting it if not passed - https://phabricator.wikimedia.org/T354196 (10CodeReviewBot) dcaro opened https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/17... [09:52:42] 10Toolforge (Toolforge iteration 02), 10Toolforge Build Service: Toolforge Build Service: add the locale buildpack - https://phabricator.wikimedia.org/T354128 (10dcaro) a:03dcaro [09:59:03] 10Toolforge (Toolforge iteration 02): `build quota` fails if tool has no builds - https://phabricator.wikimedia.org/T353701 (10Slst2020) 05Open→03In progress [10:06:22] 10Toolforge (Toolforge iteration 02), 10Patch-For-Review, 10User-dcaro: [envvars-cli] Ask for the contents of the envvar when setting it if not passed - https://phabricator.wikimedia.org/T354196 (10CodeReviewBot) dcaro merged https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/17... [10:06:28] 10Toolforge (Toolforge iteration 02), 10Patch-For-Review: [envvars-cli] use toolforge-weld for error handling - https://phabricator.wikimedia.org/T351459 (10CodeReviewBot) dcaro merged https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/17 d/changelog: bump to 0.0.3 [11:17:21] 10Grid-Engine-to-K8s-Migration: Migrate dplbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319701 (10dcaro) [11:17:42] 10Toolforge (Toolforge iteration 02), 10User-dcaro: [envvars-cli] Ask for the contents of the envvar when setting it if not passed - https://phabricator.wikimedia.org/T354196 (10dcaro) 05In progress→03Resolved [11:18:02] 10Toolforge (Toolforge iteration 02), 10Patch-For-Review: [webservice] php 7.4 containers don't pass through the environment variables to the scripts - https://phabricator.wikimedia.org/T354320 (10dcaro) 05Open→03In progress [11:31:12] 10Toolforge (Toolforge iteration 02): [bulids-builder] upgrade builder to lastest with >=2.6.1 buildpacks - https://phabricator.wikimedia.org/T354330 (10dcaro) [11:31:31] 10Toolforge (Toolforge iteration 02): [bulids-builder] upgrade builder to lastest with >=2.6.1 buildpacks - https://phabricator.wikimedia.org/T354330 (10dcaro) [11:31:33] 10Tool-hitaden, 10Toolforge Build Service: [buildservice,nodejs] nodejs buildpack does not take envvars into account - https://phabricator.wikimedia.org/T353557 (10dcaro) [11:31:47] 10Toolforge (Toolforge iteration 02): [bulids-builder] upgrade builder to lastest with >=2.6.1 buildpacks - https://phabricator.wikimedia.org/T354330 (10dcaro) [11:31:49] 10Toolforge Build Service, 10cloud-services-team, 10Documentation: Document how to update heroku-builder:22 container image - https://phabricator.wikimedia.org/T353566 (10dcaro) [11:31:51] 10Tool-hitaden, 10Toolforge Build Service: [buildservice,nodejs] nodejs buildpack does not take envvars into account - https://phabricator.wikimedia.org/T353557 (10dcaro) [11:32:12] 10Toolforge (Toolforge iteration 02), 10Toolforge Build Service, 10cloud-services-team, 10Documentation: Document how to update heroku-builder:22 container image - https://phabricator.wikimedia.org/T353566 (10dcaro) [12:34:57] 10Toolforge (Toolforge iteration 02), 10Toolforge Build Service: Toolforge Build Service: add the locale buildpack - https://phabricator.wikimedia.org/T354128 (10dcaro) Yep, I think there's no easy workaround for this one, will add the buildpack to the list [12:35:37] (CephSlowOps) firing: Ceph cluster in eqiad has 1 slow ops - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephSlowOps - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephSlowOps [12:35:42] 10cloud-services-team: CephSlowOps Ceph cluster in eqiad has slow ops, which might be blocking some writes - https://phabricator.wikimedia.org/T352570 (10phaultfinder) [12:40:37] (CephSlowOps) resolved: Ceph cluster in eqiad has 1 slow ops - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephSlowOps - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephSlowOps [12:43:03] 10Toolforge (Toolforge iteration 02), 10Patch-For-Review, 10Upstream: [maintain-harbor] Manage project quotas via maintain-harbor - https://phabricator.wikimedia.org/T352417 (10Slst2020) >>! In T352417#9432507, @Slst2020 wrote: >I've asked in the cncf harbor slack channel if I'm missing something and there's... [13:07:17] 10Toolforge (Toolforge iteration 02): [builds-builder] move buildpacks under their own namespace in gitlab - https://phabricator.wikimedia.org/T354349 (10dcaro) [13:07:22] 10Toolforge (Toolforge iteration 02): [builds-builder] move buildpacks under their own namespace in gitlab - https://phabricator.wikimedia.org/T354349 (10dcaro) a:03dcaro [13:08:04] 10Toolforge (Toolforge iteration 02): [builds-builder] move buildpacks under their own namespace in gitlab - https://phabricator.wikimedia.org/T354349 (10dcaro) 05Open→03In progress [13:12:41] 10Toolforge (Toolforge iteration 02), 10Patch-For-Review: [builds-builder] move buildpacks under their own namespace in gitlab - https://phabricator.wikimedia.org/T354349 (10CodeReviewBot) dcaro opened https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/23 buildpacks: use the ne... [13:13:32] 10Tool-ducttape, 10Abstract Wikipedia team: Add documentation for retriggering a failed pipeline to AW developer cheatsheet - https://phabricator.wikimedia.org/T333192 (10Jdforrester-WMF) 05Open→03In progress https://meta.wikimedia.org/w/index.php?diff=26024825&oldid=25870526&title=Abstract_Wikipedia/Abstr... [13:16:27] 10Toolforge (Toolforge iteration 02): [toolforge,ci] generate versions fails with dubious git directory error - https://phabricator.wikimedia.org/T354350 (10dcaro) [13:16:50] 10Toolforge (Toolforge iteration 02): [toolforge,ci] generate versions fails with dubious git directory error - https://phabricator.wikimedia.org/T354350 (10dcaro) 05Open→03In progress [13:18:33] 10Toolforge (Toolforge iteration 02), 10Patch-For-Review: [toolforge,ci] generate versions fails with dubious git directory error - https://phabricator.wikimedia.org/T354350 (10CodeReviewBot) dcaro opened https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/25 version_generation: make sour... [13:22:51] 10Toolforge (Toolforge iteration 02): [buildservice] Feature request - Indicate when long envvars are cutoff when listing - https://phabricator.wikimedia.org/T353287 (10Amorymeltzer) Yes! ---- I see the `--truncate` option now, is it new, or did I miss it previously? If the latter, I'm quite sorry for taking... [13:25:55] 10Toolforge (Toolforge iteration 02), 10Toolforge Build Service: Toolforge Build Service: add the locale buildpack - https://phabricator.wikimedia.org/T354128 (10dcaro) 05Open→03In progress [13:35:41] 10Toolforge (Toolforge iteration 02), 10Patch-For-Review: [toolforge,ci] generate versions fails with dubious git directory error - https://phabricator.wikimedia.org/T354350 (10CodeReviewBot) dcaro merged https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/25 version_generation: make sour... [13:41:56] 10Toolforge (Toolforge iteration 02): [buildservice] Feature request - Indicate when long envvars are cutoff when listing - https://phabricator.wikimedia.org/T353287 (10Slst2020) >>! In T353287#9435165, @Amorymeltzer wrote: > Yes! > > ---- > > I see the `--truncate` option now, is it new, or did I miss it prev... [13:42:59] 10Toolforge (Toolforge iteration 02), 10Patch-For-Review: [toolforge,ci] generate versions fails with dubious git directory error - https://phabricator.wikimedia.org/T354350 (10dcaro) hmpf... the issue is that the directory is owned by a different user (now the image runs as nobody). And the nobody user has n... [13:48:16] 10Tool-bub2: Uploads failing for Panjab Digital Library - https://phabricator.wikimedia.org/T354351 (10wassan.anmol117) [13:49:30] 10Tool-bub2: Integrate DSpace based libraries - https://phabricator.wikimedia.org/T354352 (10wassan.anmol117) [13:50:05] 10Tool-bub2: Integrate Hathitrust library - https://phabricator.wikimedia.org/T354353 (10wassan.anmol117) [13:50:29] 10Tool-bub2: Integrate Hathitrust library - https://phabricator.wikimedia.org/T354353 (10wassan.anmol117) a:03wassan.anmol117 [13:56:01] 10Toolforge (Toolforge iteration 02), 10Patch-For-Review: [toolforge,ci] generate versions fails with dubious git directory error - https://phabricator.wikimedia.org/T354350 (10CodeReviewBot) dcaro opened https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/26 version_generation: second tr... [13:56:08] 10Tool-bub2: Integrate Internet Archive as one of the libraries - https://phabricator.wikimedia.org/T354354 (10wassan.anmol117) [13:56:44] 10Tool-bub2: Integrate Internet Archive as one of the libraries - https://phabricator.wikimedia.org/T354354 (10wassan.anmol117) Allow direct upload to Commons, metadata entries in Wikidata and Wikisource index page creation [14:04:43] 10Toolforge (Toolforge iteration 02), 10Patch-For-Review: [toolforge,ci] generate versions fails with dubious git directory error - https://phabricator.wikimedia.org/T354350 (10CodeReviewBot) dcaro merged https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/26 version_generation: second tr... [14:11:40] 10Toolforge (Toolforge iteration 02), 10Patch-For-Review: [toolforge,ci] generate versions fails with dubious git directory error - https://phabricator.wikimedia.org/T354350 (10dcaro) 05In progress→03Resolved [14:18:51] 10Grid-Engine-to-K8s-Migration: Migrate mbh from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319883 (10nskaggs) @MBH I suggested that approach as a close approximation of what type of environment you had on grid engine. Using apt to add mono would mean /usr/bin/mono is pres... [14:22:47] 10Toolforge (Toolforge iteration 02), 10Toolforge Build Service, 10Patch-For-Review: Toolforge Build Service: add the locale buildpack - https://phabricator.wikimedia.org/T354128 (10CodeReviewBot) dcaro opened https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/24 buildpacks:... [14:47:15] 10Toolforge (Toolforge iteration 02): [ci] Investigate discrepancy between different CI envs - https://phabricator.wikimedia.org/T353044 (10Slst2020) [14:47:51] 10Toolforge (Toolforge iteration 02): [ci] Investigate discrepancy between different CI envs - https://phabricator.wikimedia.org/T353044 (10Slst2020) Has been observed in builds-cli and envvars-cli [14:51:35] 10Toolforge (Toolforge iteration 02): [ci] Investigate discrepancy between different CI envs - https://phabricator.wikimedia.org/T353044 (10dcaro) One source of this is the two different sources of truth for installed packages, `poetry.toml` and `.pre-commit-config.yaml`, specifically for `toolforge-weld` as the... [14:53:26] 10Toolforge (Toolforge iteration 02): [ci] Investigate discrepancy between different CI envs - https://phabricator.wikimedia.org/T353044 (10dcaro) The issue for the script shown here is due to the old version of python (3.7) that comes with an old version of `unittest.mock`. We still have python3.7 on the basti... [14:55:04] 10Cloud-VPS (Project-requests): Request creation of OpenVAS VPS project - https://phabricator.wikimedia.org/T354192 (10fnegri) 05Open→03In progress a:03fnegri [14:56:28] 10Toolforge (Toolforge iteration 02), 10Patch-For-Review: [ci] Investigate discrepancy between different CI envs - https://phabricator.wikimedia.org/T353044 (10CodeReviewBot) sstefanova opened https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/18 dev: update toolforge-weld version [14:59:22] 10Cloud-VPS (Project-requests): Request creation of OpenVAS VPS project - https://phabricator.wikimedia.org/T354192 (10Slst2020) +1 [15:03:20] !log fnegri@cloudcumin1001 openvas START - Cookbook wmcs.vps.create_project for project openvas in eqiad1 (T354192) [15:03:22] fnegri@cloudcumin1001: Unknown project "openvas" [15:03:22] T354192: Request creation of OpenVAS VPS project - https://phabricator.wikimedia.org/T354192 [15:05:12] 10Toolforge (Toolforge iteration 02), 10Patch-For-Review: [ci] Investigate discrepancy between different CI envs - https://phabricator.wikimedia.org/T353044 (10CodeReviewBot) sstefanova merged https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/18 dev: update toolforge-weld version [15:07:35] 10Toolforge (Toolforge iteration 02), 10Patch-For-Review: [ci] Investigate discrepancy between different CI envs - https://phabricator.wikimedia.org/T353044 (10CodeReviewBot) sstefanova opened https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/46 dev: update toolforge-weld to 1.4.0+ [15:08:53] !log fnegri@cloudcumin1001 openvas END (PASS) - Cookbook wmcs.vps.create_project (exit_code=0) for project openvas in eqiad1 (T354192) [15:08:57] T354192: Request creation of OpenVAS VPS project - https://phabricator.wikimedia.org/T354192 [15:10:45] 10Cloud-VPS: [openstack] Creating a new project returns Gateway Timeout (HTTP 504) - https://phabricator.wikimedia.org/T353829 (10fnegri) I've just successfully created a project, Cumin reports the CLI took 5 mins 32 seconds to create the project: ` fnegri@cloudcumin1001:~$ sudo cookbook wmcs.vps.create_project... [15:11:25] 10Cloud-VPS: [openstack] Creating a new project returns Gateway Timeout (HTTP 504) - https://phabricator.wikimedia.org/T353829 (10fnegri) 05Open→03Resolved a:03fnegri [15:12:00] 10Toolforge (Toolforge iteration 02), 10Patch-For-Review: [ci] Investigate discrepancy between different CI envs - https://phabricator.wikimedia.org/T353044 (10CodeReviewBot) sstefanova merged https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/46 dev: update toolforge-weld to 1.4.0+ [15:19:28] 10Cloud-VPS (Project-requests): Request creation of OpenVAS VPS project - https://phabricator.wikimedia.org/T354192 (10fnegri) @KHurd-WMF I have created the project, please note that project names are all-lowercase, so I created it as `openvas` instead of `OpenVAS`. To give you access, I need your [Developer ac... [15:28:15] 10Toolforge (Toolforge iteration 02), 10Patch-For-Review: [buildservice] Feature request - Indicate when long envvars are cutoff when listing - https://phabricator.wikimedia.org/T353287 (10CodeReviewBot) sstefanova opened https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/19 list:... [16:34:47] 10cloud-services-team: cloud-vps: monitor dns record leaks - https://phabricator.wikimedia.org/T354365 (10Andrew) [16:35:16] 10Cloud-VPS, 10cloud-services-team: cloud-vps: monitor dns record leaks - https://phabricator.wikimedia.org/T354365 (10Andrew) [17:10:28] 10Toolforge (Toolforge iteration 02): [bulids-builder] upgrade builder to lastest with >=2.6.1 buildpacks - https://phabricator.wikimedia.org/T354330 (10dcaro) [17:11:15] 10Toolforge (Toolforge iteration 02), 10Toolforge Build Service, 10cloud-services-team, 10Documentation: Document how to update heroku-builder:22 container image - https://phabricator.wikimedia.org/T353566 (10dcaro) 05Open→03Resolved a:03dcaro Done https://wikitech.wikimedia.org/wiki/Portal:Toolforge... [17:39:53] 10Cloud-VPS, 10cloud-services-team (FY2023/2024-Q1-Q2): [openstack] Creating a new project returns Gateway Timeout (HTTP 504) - https://phabricator.wikimedia.org/T353829 (10fnegri) [18:01:29] 10Grid-Engine-to-K8s-Migration: Migrate redirtalkdeleter from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319998 (10Andriy.v) 05Open→03Resolved a:03Andriy.v Scheduling works too. Migration to k8s is completed. [18:12:15] 10Cloud-VPS (Project-requests): Request creation of OpenVAS VPS project - https://phabricator.wikimedia.org/T354192 (10KHurd-WMF) KHurd, please [20:28:58] 10Tool-bub2: Integrate Gallica (Bibliothèque Nationale de France) - https://phabricator.wikimedia.org/T354381 (10Poslovitch) [20:36:35] 10Tool-bub2: Integrate Limédia Kiosque/Galeries (Sillon Lorrain, France) - https://phabricator.wikimedia.org/T354382 (10Poslovitch) [20:38:02] 10Tool-bub2: Integrate Limédia Kiosque/Galeries (Sillon Lorrain, France) - https://phabricator.wikimedia.org/T354382 (10Poslovitch) [20:55:19] (HAProxyBackendUnavailable) firing: HAProxy service neutron-api_backend backend cloudcontrol1006.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [21:00:19] (HAProxyBackendUnavailable) resolved: HAProxy service neutron-api_backend backend cloudcontrol1006.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable