[00:02:40] (GaleraClusterSizeMismatch) firing: Galera in has 2 nodes - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/GaleraClusterSizeMismatch - https://grafana.wikimedia.org/d/galera-cluster-summary/wmcs-openstack-eqiad-galera-cluster-summary - https://alerts.wikimedia.org/?q=alertname%3DGaleraClusterSizeMismatch [00:03:19] (HAProxyBackendUnavailable) firing: HAProxy service mysql backend cloudcontrol1005.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [00:03:40] (GaleraNotEnabled) firing: Galera not enabled on cloudcontrol1005:9104 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/GaleraNotEnabled - https://grafana.wikimedia.org/d/galera-cluster-summary/wmcs-openstack-eqiad-galera-cluster-summary - https://alerts.wikimedia.org/?q=alertname%3DGaleraNotEnabled [00:03:40] (GaleraNodeOutOfSync) firing: Galera node cloudcontrol1005:9104 is out of sync - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/GaleraNodeOutOfSync - https://grafana.wikimedia.org/d/galera-cluster-summary/wmcs-openstack-eqiad-galera-cluster-summary - https://alerts.wikimedia.org/?q=alertname%3DGaleraNodeOutOfSync [00:05:04] (TfInfraTestDestroyFailed) firing: Terraform failed to destroy the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestDestroyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestDestroyFailed [00:07:40] (GaleraClusterSizeMismatch) firing: (3) Galera in has 0 nodes - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/GaleraClusterSizeMismatch - https://grafana.wikimedia.org/d/galera-cluster-summary/wmcs-openstack-eqiad-galera-cluster-summary - https://alerts.wikimedia.org/?q=alertname%3DGaleraClusterSizeMismatch [00:07:40] (GaleraNoRecentWrites) firing: Galera node cloudcontrol1005:9104 has no writes - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/GaleraNoRecentWrites - https://grafana.wikimedia.org/d/galera-cluster-summary/wmcs-openstack-eqiad-galera-cluster-summary - https://alerts.wikimedia.org/?q=alertname%3DGaleraNoRecentWrites [00:07:56] 10Cloud-VPS (Quota-requests), 10Community-Tech, 10WS Export: Increase wikisource project quota in order to upgrade WS Export - https://phabricator.wikimedia.org/T353770 (10Samwilson) Thank you! [00:10:04] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [00:11:03] (PuppetAgentStaleLastRun) firing: Last Puppet run was over 24 hours ago on instance tf-infra-test in project tf-infra-test - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [00:51:04] (InstanceDown) firing: Project toolsbeta instance toolsbeta-bastion-6 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [02:33:20] (HAProxyBackendUnavailable) firing: (2) HAProxy service mysql backend cloudcontrol1005.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [02:35:41] PROBLEM - Check systemd state on cloudservices1005 is CRITICAL: CRITICAL - degraded: The following units failed: labs-ip-alias-dump.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [02:38:19] (HAProxyBackendUnavailable) firing: (2) HAProxy service mysql backend cloudcontrol1005.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [02:38:33] (SystemdUnitDown) firing: The service unit labs-ip-alias-dump.service is in failed status on host cloudservices1005. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudservices1005 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [03:02:40] (GaleraClusterSizeMismatch) resolved: (3) Galera in has 0 nodes - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/GaleraClusterSizeMismatch - https://grafana.wikimedia.org/d/galera-cluster-summary/wmcs-openstack-eqiad-galera-cluster-summary - https://alerts.wikimedia.org/?q=alertname%3DGaleraClusterSizeMismatch [03:03:40] (GaleraNodeOutOfSync) resolved: Galera node cloudcontrol1005:9104 is out of sync - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/GaleraNodeOutOfSync - https://grafana.wikimedia.org/d/galera-cluster-summary/wmcs-openstack-eqiad-galera-cluster-summary - https://alerts.wikimedia.org/?q=alertname%3DGaleraNodeOutOfSync [03:03:40] (GaleraNotEnabled) resolved: Galera not enabled on cloudcontrol1005:9104 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/GaleraNotEnabled - https://grafana.wikimedia.org/d/galera-cluster-summary/wmcs-openstack-eqiad-galera-cluster-summary - https://alerts.wikimedia.org/?q=alertname%3DGaleraNotEnabled [03:07:40] (GaleraNoRecentWrites) resolved: Galera node cloudcontrol1005:9104 has no writes - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/GaleraNoRecentWrites - https://grafana.wikimedia.org/d/galera-cluster-summary/wmcs-openstack-eqiad-galera-cluster-summary - https://alerts.wikimedia.org/?q=alertname%3DGaleraNoRecentWrites [03:08:19] (HAProxyBackendUnavailable) resolved: HAProxy service mysql backend cloudcontrol1005.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [03:10:04] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [03:10:04] (TfInfraTestDestroyFailed) firing: Terraform failed to destroy the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestDestroyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestDestroyFailed [03:11:03] (PuppetAgentStaleLastRun) firing: Last Puppet run was over 24 hours ago on instance tf-infra-test in project tf-infra-test - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [03:30:31] RECOVERY - Check systemd state on cloudservices1005 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [03:39:38] 10Grid-Engine-to-K8s-Migration: Migrate sigma from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320041 (10AntiCompositeNumber) This is a critical tool for anti-abuse workflows on multiple wikis. Sigma is inactive, and I asked Earwig about it on their talk page: > @AntiComposi... [03:39:56] 10Grid-Engine-to-K8s-Migration: Migrate sigma from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320041 (10AntiCompositeNumber) a:05Sigma→03None [03:46:18] (SystemdUnitDown) resolved: The service unit labs-ip-alias-dump.service is in failed status on host cloudservices1005. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudservices1005 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [03:51:04] (InstanceDown) firing: Project toolsbeta instance toolsbeta-bastion-6 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [05:23:33] 10Grid-Engine-to-K8s-Migration, 10Pywikibot: Migrate pywikibot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319981 (10JJMC89) [05:23:39] 10Toolforge (Software install/update): Add zip to Kubernetes base images - https://phabricator.wikimedia.org/T353769 (10JJMC89) [05:23:43] 10Toolforge Jobs framework, 10cloud-services-team: toolforge-jobs should be able to run git with ssh - https://phabricator.wikimedia.org/T307847 (10JJMC89) [05:23:47] 10Toolforge (Software install/update), 10User-QEDK, 10User-bd808: Add ssh-client support to Kubernetes containers - https://phabricator.wikimedia.org/T258841 (10JJMC89) [05:48:46] 10Tool-masto-collab: Add i18n support - https://phabricator.wikimedia.org/T353831 (10Legoktm) Thanks for filing this, it's definitely something I want to work on soon. I looked around for Rust i18n libraries like 3-4 months ago and was pretty disappointed by what I saw, so we might need to write one ourselves. [06:10:04] (TfInfraTestDestroyFailed) firing: Terraform failed to destroy the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestDestroyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestDestroyFailed [06:10:04] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [06:11:03] (PuppetAgentStaleLastRun) firing: Last Puppet run was over 24 hours ago on instance tf-infra-test in project tf-infra-test - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [06:27:17] 10Tool-masto-collab: masto-collab: Allow setting content warning - https://phabricator.wikimedia.org/T353861 (10Frostly) [06:51:04] (InstanceDown) firing: Project toolsbeta instance toolsbeta-bastion-6 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [07:18:26] 10Grid-Engine-to-K8s-Migration: Migrate sigma from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320041 (100xDeadbeef) I have migrated the jobs and webservices to kubernetes. This should be resolved now. [09:10:04] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [09:10:04] (TfInfraTestDestroyFailed) firing: Terraform failed to destroy the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestDestroyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestDestroyFailed [09:11:03] (PuppetAgentStaleLastRun) firing: Last Puppet run was over 24 hours ago on instance tf-infra-test in project tf-infra-test - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [09:51:04] (InstanceDown) firing: Project toolsbeta instance toolsbeta-bastion-6 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [10:10:53] 10Grid-Engine-to-K8s-Migration: Migrate sigma from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320041 (10taavi) 05Open→03Resolved a:030xDeadbeef Thank you! [11:06:22] (03PS1) 10Majavah: bookworm-sssd: install zip and unzip [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/984813 (https://phabricator.wikimedia.org/T353769) [11:06:33] 10Toolforge (Software install/update), 10Patch-For-Review: Add zip to Kubernetes base images - https://phabricator.wikimedia.org/T353769 (10taavi) a:03taavi `zip` and `unzip` (yes, those are separate packages. no, `zip` does not have a flag to extract an archive.) are about a megabyte extracted together, so... [11:24:07] 10wikitech.wikimedia.org, 10Parsoid: Wikitech job runner often OOMs on parsoidCachePrewarm - https://phabricator.wikimedia.org/T353587 (10matmarex) [12:10:04] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [12:10:04] (TfInfraTestDestroyFailed) firing: Terraform failed to destroy the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestDestroyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestDestroyFailed [12:11:03] (PuppetAgentStaleLastRun) firing: Last Puppet run was over 24 hours ago on instance tf-infra-test in project tf-infra-test - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [12:25:14] (03CR) 10CI reject: [V: 04-1] Localisation updates from https://translatewiki.net. [labs/tools/commons-mass-description] - 10https://gerrit.wikimedia.org/r/984832 (owner: 10L10n-bot) [12:51:04] (InstanceDown) firing: Project toolsbeta instance toolsbeta-bastion-6 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [14:56:37] (03CR) 10BryanDavis: [C: 03+1] bookworm-sssd: install zip and unzip [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/984813 (https://phabricator.wikimedia.org/T353769) (owner: 10Majavah) [14:57:54] 10Toolforge, 10cloud-services-team, 10Patch-For-Review: Toolforge: Decomission the Grid Engine infrastructure - https://phabricator.wikimedia.org/T314664 (10taavi) [14:58:27] (03CR) 10Majavah: [C: 03+2] bookworm-sssd: install zip and unzip [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/984813 (https://phabricator.wikimedia.org/T353769) (owner: 10Majavah) [14:58:51] 10Toolforge, 10cloud-services-team, 10Patch-For-Review: Provide tools for disabling the grid for specific tools - https://phabricator.wikimedia.org/T353351 (10taavi) 05Open→03Resolved a:03Andrew [14:59:09] (03Merged) 10jenkins-bot: bookworm-sssd: install zip and unzip [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/984813 (https://phabricator.wikimedia.org/T353769) (owner: 10Majavah) [15:11:03] (PuppetAgentStaleLastRun) firing: Last Puppet run was over 24 hours ago on instance tf-infra-test in project tf-infra-test - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [15:15:04] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [15:15:04] (TfInfraTestDestroyFailed) firing: Terraform failed to destroy the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestDestroyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestDestroyFailed [15:31:19] (HAProxyBackendUnavailable) firing: HAProxy service keystone-public-api_backend backend cloudcontrol1005.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [15:31:45] 10Toolforge (Software install/update): Add zip to Kubernetes base images - https://phabricator.wikimedia.org/T353769 (10taavi) 05Open→03Resolved [15:31:49] 10Grid-Engine-to-K8s-Migration, 10Pywikibot: Migrate pywikibot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319981 (10taavi) [15:36:19] (HAProxyBackendUnavailable) resolved: HAProxy service keystone-public-api_backend backend cloudcontrol1005.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [15:47:58] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeexec-10-16 [15:51:04] (InstanceDown) firing: Project toolsbeta instance toolsbeta-bastion-6 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [16:23:29] 10Cloud-VPS, 10cloud-services-team (FY2023/2024-Q1-Q2), 10User-dcaro: OpenStack API response time gets slower over time - https://phabricator.wikimedia.org/T345084 (10fnegri) p:05Triage→03Medium [16:25:29] 10Cloud-VPS, 10Patch-For-Review: [openstack] Creating a new project returns Gateway Timeout (HTTP 504) - https://phabricator.wikimedia.org/T353829 (10fnegri) Possibly related to {T345084} [16:33:20] (HAProxyBackendUnavailable) firing: HAProxy service neutron-api_backend backend cloudcontrol1007.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [16:38:19] (HAProxyBackendUnavailable) resolved: HAProxy service neutron-api_backend backend cloudcontrol1007.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [16:53:45] 10cloud-services-team: NodeDown cloudvirt1063 - https://phabricator.wikimedia.org/T353406 (10fnegri) 05Open→03Resolved a:03fnegri I deactivated the node to resolve the alerts until the server is replaced in {T353408} [17:01:04] (InstanceDown) resolved: Project toolsbeta instance toolsbeta-bastion-6 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [17:22:50] !log fnegri@cloudcumin1001 tf-infra-test START - Cookbook wmcs.vps.add_user_to_project for user 'fnegri' in role 'member' [17:22:57] !log fnegri@cloudcumin1001 tf-infra-test END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'fnegri' in role 'member' [17:36:03] (PuppetAgentStaleLastRun) resolved: Last Puppet run was over 24 hours ago on instance tf-infra-test in project tf-infra-test - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [17:37:19] 10Grid-Engine-to-K8s-Migration, 10Pywikibot: Migrate pywikibot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319981 (10JJMC89) My version of the script is working now. `lang=shell-session [17:16:43]tools.jjmc89-bot-dev@tools-sgebastion-11:~$ cat pre_nightly rm -rf nigh... [18:35:04] (TfInfraTestApplyFailed) resolved: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [18:35:05] (TfInfraTestDestroyFailed) resolved: Terraform failed to destroy the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestDestroyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestDestroyFailed [18:36:42] 10Cloud-VPS: [tf-infra-test] alerts are pointing to the wrong metrics - https://phabricator.wikimedia.org/T353893 (10fnegri) [18:39:13] 10Cloud-VPS: [tf-infra-test] alerts are pointing to the wrong metrics - https://phabricator.wikimedia.org/T353893 (10fnegri) 05Open→03Resolved a:03fnegri I fixed this by editing the alerts in `metricsinfra-controller-2.metricsinfra.eqiad1.wikimedia.cloud`: ` MariaDB [prometheusconfig]> SELECT * FROM alert... [18:39:34] 10Cloud-VPS, 10cloud-services-team (FY2023/2024-Q1-Q2): [tf-infra-test] alerts are pointing to the wrong metrics - https://phabricator.wikimedia.org/T353893 (10fnegri) [20:53:20] (ProbeDown) firing: Service toolserver-proxy-01:443 has failed probes (http_toolserver_org_redirects_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#toolserver-proxy-01:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [20:58:20] (ProbeDown) resolved: Service toolserver-proxy-01:443 has failed probes (http_toolserver_org_redirects_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#toolserver-proxy-01:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [21:15:03] (03PS1) 10Umherirrender: branches: Remove REL1_35 [labs/libraryupgrader/config] - 10https://gerrit.wikimedia.org/r/985014 [21:16:01] (03CR) 10Umherirrender: [C: 03+2] "https://lists.wikimedia.org/hyperkitty/list/mediawiki-l@lists.wikimedia.org/thread/72B3GNSDFUJEM23W54735OW34IHCZL6F/" [labs/libraryupgrader/config] - 10https://gerrit.wikimedia.org/r/985014 (owner: 10Umherirrender) [21:16:38] (03Merged) 10jenkins-bot: branches: Remove REL1_35 [labs/libraryupgrader/config] - 10https://gerrit.wikimedia.org/r/985014 (owner: 10Umherirrender) [22:00:53] 10Tool-gitlab-account-approval, 10gitlab-settings, 10GitLab (Auth & Access), 10Release-Engineering-Team (Priority Backlog 📥), 10User-brennen: Create an approved-accounts project in GitLab for distributing account approval process to non-administrators - https://phabricator.wikimedia.org/T350832 (10bd808) [22:40:48] 10Grid-Engine-to-K8s-Migration: Migrate mdwiki from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319887 (10Mr.Ibrahem) We used "jsub" command to start some tasks and now we are using "toolforge-jobs" instead, and it didn't work through php page [22:41:58] 10Grid-Engine-to-K8s-Migration: Migrate lyan from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319867 (10Mr.Ibrahem) Done [22:42:26] 10Grid-Engine-to-K8s-Migration: Migrate lyan from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319867 (10Mr.Ibrahem) 05Open→03Resolved a:03Mr.Ibrahem [22:44:46] 10Grid-Engine-to-K8s-Migration: Migrate sanaa from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320028 (10Mr.Ibrahem) Done [22:44:59] 10Grid-Engine-to-K8s-Migration: Migrate sanaa from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320028 (10Mr.Ibrahem) 05Open→03Resolved a:03Mr.Ibrahem [22:45:24] 10Grid-Engine-to-K8s-Migration: Migrate suha from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320069 (10Mr.Ibrahem) Done [22:45:44] 10Grid-Engine-to-K8s-Migration: Migrate suha from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320069 (10Mr.Ibrahem) 05Open→03Resolved a:03Mr.Ibrahem [22:46:13] 10Grid-Engine-to-K8s-Migration: Migrate nccroptool from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319921 (10Mr.Ibrahem) 05Open→03Resolved a:03Mr.Ibrahem [22:51:37] 10Tool-gitlab-account-approval: Consider adding Gerrit Trusted-Users group as source of trust - https://phabricator.wikimedia.org/T353914 (10bd808)