[00:01:49] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [00:16:56] (SystemdUnitDown) firing: The service unit nova-fullstack.service is in failed status on host cloudcontrol1006. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1006 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [00:51:56] (SystemdUnitDown) resolved: The service unit nova-fullstack.service is in failed status on host cloudcontrol1006. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1006 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [01:00:12] 10Tool-gitlab-account-approval, 10User-bd808: gitlab-account-approval bot stalled on 2024-01-09 - https://phabricator.wikimedia.org/T356097 (10Legoktm) Wanted to process T356586 but it's stuck again :( ` tools.gitlab-account-approval@tools-sgebastion-10:~$ kubectl get pods NAME READY STA... [01:03:45] 10Tool-gitlab-account-approval, 10Patch-For-Review, 10User-bd808: gitlab-account-approval bot stalled on 2024-01-09 - https://phabricator.wikimedia.org/T356097 (10CodeReviewBot) legoktm opened https://gitlab.wikimedia.org/toolforge-repos/gitlab-account-approval/-/merge_requests/11 Run `approve` under `timeo... [01:52:37] (HAProxyBackendUnavailable) firing: (2) HAProxy service nova-api_backend backend cloudcontrol1005.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [02:01:56] (SystemdUnitDown) firing: The service unit nova-fullstack.service is in failed status on host cloudcontrol1006. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1006 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [02:21:56] (SystemdUnitDown) resolved: The service unit nova-fullstack.service is in failed status on host cloudcontrol1006. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1006 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [02:36:17] (OpenstackAPIResponse) firing: (3) Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [02:46:56] (SystemdUnitDown) firing: The service unit nova-fullstack.service is in failed status on host cloudcontrol1006. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1006 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [03:01:49] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [03:21:56] (SystemdUnitDown) resolved: The service unit nova-fullstack.service is in failed status on host cloudcontrol1006. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1006 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [04:11:56] (SystemdUnitDown) firing: The service unit nova-fullstack.service is in failed status on host cloudcontrol1006. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1006 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [04:21:56] (SystemdUnitDown) resolved: The service unit nova-fullstack.service is in failed status on host cloudcontrol1006. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1006 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [04:31:56] (SystemdUnitDown) firing: The service unit nova-fullstack.service is in failed status on host cloudcontrol1006. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1006 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [04:51:56] (SystemdUnitDown) resolved: The service unit nova-fullstack.service is in failed status on host cloudcontrol1006. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1006 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [04:53:26] (SystemdUnitDown) firing: The service unit nova-fullstack.service is in failed status on host cloudcontrol1006. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1006 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [05:23:26] (SystemdUnitDown) resolved: The service unit nova-fullstack.service is in failed status on host cloudcontrol1006. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1006 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [05:31:56] (SystemdUnitDown) firing: The service unit nova-fullstack.service is in failed status on host cloudcontrol1006. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1006 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [05:51:56] (SystemdUnitDown) resolved: The service unit nova-fullstack.service is in failed status on host cloudcontrol1006. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1006 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [05:52:38] (HAProxyBackendUnavailable) firing: (2) HAProxy service nova-api_backend backend cloudcontrol1005.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [06:01:49] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [06:11:56] (SystemdUnitDown) firing: The service unit nova-fullstack.service is in failed status on host cloudcontrol1006. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1006 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [06:21:56] (SystemdUnitDown) resolved: The service unit nova-fullstack.service is in failed status on host cloudcontrol1006. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1006 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [06:27:22] (HAProxyBackendUnavailable) firing: (3) HAProxy service neutron-api_backend backend cloudcontrol1007.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [06:31:56] (SystemdUnitDown) firing: The service unit nova-fullstack.service is in failed status on host cloudcontrol1006. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1006 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [06:36:17] (OpenstackAPIResponse) firing: (3) Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [06:38:41] (CloudVPSDesignateLeaks) firing: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:43:41] (CloudVPSDesignateLeaks) firing: (2) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:48:41] (CloudVPSDesignateLeaks) firing: (2) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:51:56] (SystemdUnitDown) resolved: The service unit nova-fullstack.service is in failed status on host cloudcontrol1006. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1006 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [06:53:41] (CloudVPSDesignateLeaks) resolved: (2) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [07:16:56] (SystemdUnitDown) firing: The service unit nova-fullstack.service is in failed status on host cloudcontrol1006. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1006 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [07:21:56] (SystemdUnitDown) resolved: The service unit nova-fullstack.service is in failed status on host cloudcontrol1006. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1006 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [08:06:56] (SystemdUnitDown) firing: The service unit nova-fullstack.service is in failed status on host cloudcontrol1006. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1006 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [08:21:56] (SystemdUnitDown) resolved: The service unit nova-fullstack.service is in failed status on host cloudcontrol1006. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1006 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [08:41:56] (SystemdUnitDown) firing: The service unit nova-fullstack.service is in failed status on host cloudcontrol1006. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1006 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [08:42:22] (HAProxyBackendUnavailable) firing: (3) HAProxy service neutron-api_backend backend cloudcontrol1006.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [08:51:56] (SystemdUnitDown) resolved: The service unit nova-fullstack.service is in failed status on host cloudcontrol1006. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1006 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [09:01:49] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [09:01:57] (SystemdUnitDown) firing: The service unit nova-fullstack.service is in failed status on host cloudcontrol1006. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1006 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [09:21:56] (SystemdUnitDown) resolved: The service unit nova-fullstack.service is in failed status on host cloudcontrol1006. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1006 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [09:32:22] (HAProxyBackendUnavailable) firing: (2) HAProxy service nova-api_backend backend cloudcontrol1005.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [09:52:22] (HAProxyBackendUnavailable) resolved: HAProxy service nova-api_backend backend cloudcontrol1007.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [10:24:11] (03PS1) 10Ladsgroup: Always compress text blobs [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/997333 (https://phabricator.wikimedia.org/T356565) [10:26:41] (03CR) 10CI reject: [V: 04-1] Always compress text blobs [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/997333 (https://phabricator.wikimedia.org/T356565) (owner: 10Ladsgroup) [10:36:17] (OpenstackAPIResponse) firing: (3) Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [10:36:17] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Cloud-Services-Origin-Alert, 10Cloud-Services-Worktype-Unplanned, 10User-dcaro: [nova-api,cloudrabbit] Connectivity issues from all cloudcontrols to all cloudrabbit nodes - https://phabricator.wikimedia.org/T356621 (10dcaro) p:05Triage→03High [10:37:10] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Cloud-Services-Origin-Alert, 10Cloud-Services-Worktype-Unplanned, 10User-dcaro: [nova-api,cloudrabbit] Connectivity issues from all cloudcontrols to all cloudrabbit nodes - https://phabricator.wikimedia.org/T356621 (10dcaro) [10:42:48] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Cloud-Services-Origin-Alert, 10Cloud-Services-Worktype-Unplanned, 10User-aborrero, 10User-dcaro: [nova-api,cloudrabbit] Connectivity issues from all cloudcontrols to all cloudrabbit nodes - https://phabricator.wikimedia.org/T356621 (10aborrero) [11:28:17] 10cloud-services-team (Hardware), 10DC-Ops, 10SRE, 10ops-eqiad: Q3:rack/setup/install cloudcephosd10(3[5-9]|40) - https://phabricator.wikimedia.org/T324998 (10Volans) @wiki_willy @Jclark-ctr @RobH As I see that Rob is out this week, to unblock the rest of SREs with any DNS-related change in Netbox I'm runn... [11:34:40] 10Toolforge, 10cloud-services-team, 10User-aborrero: kubelet: cannot work with docker >= v25 - https://phabricator.wikimedia.org/T356629 (10aborrero) [11:34:55] 10cloud-services-team (Hardware), 10DC-Ops, 10SRE, 10ops-eqiad: Q3:rack/setup/install cloudcephosd10(3[5-9]|40) - https://phabricator.wikimedia.org/T324998 (10Volans) Done. FYI `cloudcephosd1040` had a wrong WMF asset tag `wmf108805`, I guess it was supposed to be `wmf10805` [11:39:09] (CephSlowOps) firing: Ceph cluster in eqiad has 102 slow ops - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephSlowOps - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephSlowOps [11:39:15] 10cloud-services-team: CephSlowOps Ceph cluster in eqiad has slow ops, which might be blocking some writes - https://phabricator.wikimedia.org/T352570 (10phaultfinder) [11:41:25] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Cloud-Services-Origin-Alert, 10Cloud-Services-Worktype-Unplanned, 10User-aborrero, 10User-dcaro: [nova-api,cloudrabbit] Connectivity issues from all cloudcontrols to all cloudrabbit nodes - https://phabricator.wikimedia.org/T356621 (10dcaro) The retransmitted... [11:43:29] 10Toolforge, 10cloud-services-team, 10Patch-For-Review, 10User-aborrero: kubelet: cannot work with docker >= v25 - https://phabricator.wikimedia.org/T356629 (10aborrero) p:05Triage→03Medium [11:44:09] (CephSlowOps) resolved: Ceph cluster in eqiad has 102 slow ops - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephSlowOps - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephSlowOps [12:01:49] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [12:10:32] (03PS2) 10Ladsgroup: Always compress text blobs [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/997333 (https://phabricator.wikimedia.org/T356565) [12:10:51] (03CR) 10Ladsgroup: "The actual file didn't get checked in :face-palm:" [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/997333 (https://phabricator.wikimedia.org/T356565) (owner: 10Ladsgroup) [12:12:50] (03CR) 10CI reject: [V: 04-1] Always compress text blobs [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/997333 (https://phabricator.wikimedia.org/T356565) (owner: 10Ladsgroup) [12:15:53] (03PS3) 10Ladsgroup: Always compress text blobs [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/997333 (https://phabricator.wikimedia.org/T356565) [12:18:08] (03CR) 10CI reject: [V: 04-1] Always compress text blobs [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/997333 (https://phabricator.wikimedia.org/T356565) (owner: 10Ladsgroup) [12:22:30] (03PS4) 10Ladsgroup: Always compress text blobs [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/997333 (https://phabricator.wikimedia.org/T356565) [12:24:33] 10Grid-Engine-to-K8s-Migration: Migrate glamify from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319772 (10dcaro) I had a quick look at this tool, it's built using ruby 2, that has been EOL for a bit, and uses a very old library (mediawiki-gateway) that does not work anymore... [12:25:09] (03CR) 10CI reject: [V: 04-1] Localisation updates from https://translatewiki.net. [labs/tools/commons-mass-description] - 10https://gerrit.wikimedia.org/r/997385 (owner: 10L10n-bot) [12:50:34] (DiskSpace) firing: Disk space cloudbackup1004:9100:/ 5.905% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup1004 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [13:05:34] (DiskSpace) resolved: Disk space cloudbackup1004:9100:/ 5.887% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup1004 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [13:09:05] (03CR) 10Nikerabbit: [V: 03+2] Localisation updates from https://translatewiki.net. [labs/tools/commons-mass-description] - 10https://gerrit.wikimedia.org/r/997385 (owner: 10L10n-bot) [13:39:44] (03CR) 10Jforrester: [C: 03+1] Always compress text blobs (031 comment) [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/997333 (https://phabricator.wikimedia.org/T356565) (owner: 10Ladsgroup) [13:40:06] (03CR) 10Jforrester: [C: 03+1] "(Also are we doing dev. here or on GitLab, we've got both repos live right now?)" [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/997333 (https://phabricator.wikimedia.org/T356565) (owner: 10Ladsgroup) [13:40:15] (03PS5) 10Jforrester: Always compress text blobs [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/997333 (https://phabricator.wikimedia.org/T356565) (owner: 10Ladsgroup) [13:49:27] 10Tool-Global-user-contributions, 10Stewards-and-global-tools, 10Temporary accounts, 10XTools, and 2 others: [Design] UX exploration and wireframes - https://phabricator.wikimedia.org/T354531 (10KColeman-WMF) [14:16:54] 10Toolforge (Toolforge iteration 04), 10User-aborrero: [toolforge] several tools get periods of connection refused (104) when connecting to wikis - https://phabricator.wikimedia.org/T356164 (10dcaro) Unfortunately, I was not able to capture that issue :/ (the script did not fail), thinking on how to debug more... [14:17:14] 10Toolforge, 10cloud-services-team, 10Patch-For-Review, 10User-aborrero: kubelet: cannot work with docker >= v25 - https://phabricator.wikimedia.org/T356629 (10taavi) I don't think this will be a problem as long as we pin the current Buster packages to v23. Bookworm nodes are no longer using Docker at all. [14:27:52] (03CR) 10Majavah: "I would assume here at least until we can actually build images on GitLab?" [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/997333 (https://phabricator.wikimedia.org/T356565) (owner: 10Ladsgroup) [14:29:02] 10Grid-Engine-to-K8s-Migration: Migrate jembot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319828 (10dcaro) Hi @-jem-! Were you able to migrate? I still see one webservice running on the grid, let me know if you are finding any issues or want any guidance. If you share... [14:31:49] 10Grid-Engine-to-K8s-Migration: Migrate klexibot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319845 (10dcaro) @Tkarcher Hi! Have you been able to migrate your tool to the new platform? Let me know if you are having issues or want some help (if so, having the url to the... [14:34:13] 10Grid-Engine-to-K8s-Migration: Migrate krdbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319847 (10dcaro) Hi @Krd, can you share your code on any public repo? (wikimedia gitlab/...), I can try giving it a go helping setup the environment. [14:36:17] (OpenstackAPIResponse) firing: (3) Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [14:36:23] 10Grid-Engine-to-K8s-Migration: Migrate legobot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319854 (10dcaro) Hi @Legoktm! How is the migration going? Is there anything blocking you/issue/bug that I can help with? [14:40:01] 10Grid-Engine-to-K8s-Migration: Migrate jembot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319828 (10-jem-) Hi, @dcaro, thanks for your interest. I'm afraid things got complicated; I asked in the IRC channel for guidance in December and I still have to find time to figu... [14:54:12] (03PS1) 10Majavah: Upgrade package-lock-lint to 0.2.4 [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/997440 (https://phabricator.wikimedia.org/T356517) [14:54:14] (03PS1) 10Majavah: Drop support for lockfileVersion 1 [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/997441 [14:54:16] (03PS1) 10Majavah: Update Runner to Bookworm / Python 3.11 / Node 18 / Npm 9 [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/997442 (https://phabricator.wikimedia.org/T356595) [14:54:21] (03CR) 10CI reject: [V: 04-1] Drop support for lockfileVersion 1 [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/997441 (owner: 10Majavah) [14:54:25] (03CR) 10CI reject: [V: 04-1] Update Runner to Bookworm / Python 3.11 / Node 18 / Npm 9 [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/997442 (https://phabricator.wikimedia.org/T356595) (owner: 10Majavah) [14:56:26] (03CR) 10CI reject: [V: 04-1] Upgrade package-lock-lint to 0.2.4 [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/997440 (https://phabricator.wikimedia.org/T356517) (owner: 10Majavah) [14:58:58] (03PS2) 10Majavah: Upgrade package-lock-lint to 0.2.4 [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/997440 (https://phabricator.wikimedia.org/T356517) [14:59:00] (03PS2) 10Majavah: Drop support for lockfileVersion 1 [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/997441 [14:59:02] (03PS2) 10Majavah: Update Runner to Bookworm / Python 3.11 / Node 18 / Npm 9 [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/997442 (https://phabricator.wikimedia.org/T356595) [14:59:04] (03PS1) 10Majavah: runner: fix type() usage [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/997448 [15:00:26] (03CR) 10CI reject: [V: 04-1] Drop support for lockfileVersion 1 [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/997441 (owner: 10Majavah) [15:00:28] (03CR) 10CI reject: [V: 04-1] Update Runner to Bookworm / Python 3.11 / Node 18 / Npm 9 [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/997442 (https://phabricator.wikimedia.org/T356595) (owner: 10Majavah) [15:00:30] (03CR) 10CI reject: [V: 04-1] runner: fix type() usage [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/997448 (owner: 10Majavah) [15:01:49] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [15:04:33] 10Grid-Engine-to-K8s-Migration: Migrate krdbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319847 (10Krd) Feel free to look into my code, but I don't like to publish that. [15:05:23] (03CR) 10Jforrester: "recheck" [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/997448 (owner: 10Majavah) [15:06:19] (03CR) 10Jforrester: "https://gerrit.wikimedia.org/r/admin/groups/d801734ba79f64ac9fb5976fdf2bbc48242a3300,members is a little empty. ;-(" [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/997448 (owner: 10Majavah) [15:08:13] (03CR) 10Majavah: "recheck" [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/997441 (owner: 10Majavah) [15:08:18] (03CR) 10Majavah: "recheck" [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/997442 (https://phabricator.wikimedia.org/T356595) (owner: 10Majavah) [15:30:55] (03PS1) 10Majavah: Add support for lockfileVersion v3 [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/997463 (https://phabricator.wikimedia.org/T356517) [15:31:03] (03CR) 10CI reject: [V: 04-1] Add support for lockfileVersion v3 [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/997463 (https://phabricator.wikimedia.org/T356517) (owner: 10Majavah) [15:32:53] (03PS2) 10Majavah: Add support for lockfileVersion v3 [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/997463 (https://phabricator.wikimedia.org/T356517) [15:32:55] (03PS3) 10Majavah: Update Runner to Bookworm / Python 3.11 / Node 18 / Npm 9 [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/997442 (https://phabricator.wikimedia.org/T356595) [15:33:00] (03CR) 10CI reject: [V: 04-1] Add support for lockfileVersion v3 [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/997463 (https://phabricator.wikimedia.org/T356517) (owner: 10Majavah) [15:33:02] (03CR) 10CI reject: [V: 04-1] Update Runner to Bookworm / Python 3.11 / Node 18 / Npm 9 [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/997442 (https://phabricator.wikimedia.org/T356595) (owner: 10Majavah) [15:46:53] 10Grid-Engine-to-K8s-Migration: Migrate map-search from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319875 (10dcaro) @Abbe98 Hi! Thanks to your code being online, I was able to send a PR (https://github.com/Abbe98/map-search/pull/4) to do some minimal changes needed to be ab... [15:58:22] (03CR) 10Jforrester: [C: 03+1] runner: fix type() usage [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/997448 (owner: 10Majavah) [15:58:38] (03CR) 10Jforrester: [C: 03+1] Upgrade package-lock-lint to 0.2.4 [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/997440 (https://phabricator.wikimedia.org/T356517) (owner: 10Majavah) [16:21:33] 10Grid-Engine-to-K8s-Migration: Migrate klexibot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319845 (10Tkarcher) 05Open→03Resolved Sorry! I migrated Klexibot some time ago already, but forgot to close this task here. I see the bot running on Kubernetes: ` tools.kl... [16:51:39] (03PS1) 10Majavah: runner: Stop trying to migrate to es5 eslint profile [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/997512 (https://phabricator.wikimedia.org/T356656) [16:54:05] (03CR) 10Lucas Werkmeister (WMDE): [C: 03+1] runner: Stop trying to migrate to es5 eslint profile [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/997512 (https://phabricator.wikimedia.org/T356656) (owner: 10Majavah) [17:10:58] (03CR) 10CI reject: [V: 04-1] runner: Stop trying to migrate to es5 eslint profile [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/997512 (https://phabricator.wikimedia.org/T356656) (owner: 10Majavah) [17:14:28] (03CR) 10Jforrester: "Ideally we should actually do the reverse – client-es5 should be renamed to client (so we can kill off client-es5), and maybe the same wit" [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/997512 (https://phabricator.wikimedia.org/T356656) (owner: 10Majavah) [17:15:06] 10Tool-gitlab-account-approval, 10Patch-For-Review, 10User-bd808: gitlab-account-approval bot stalled on 2024-01-09 - https://phabricator.wikimedia.org/T356097 (10bd808) >>! In T356097#9512092, @Legoktm wrote: > last log entry was: > ` > [pod/approve-28448319-kgrlk/job] 2024-02-02T18:39:17Z glaab.utils INFO:... [17:17:41] (03CR) 10Majavah: "Hmm, maybe - the migration to es6 is a bit disruptive here as `var` needs to be changed to `let`/`const` everywhere. I'd prefer to have th" [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/997512 (https://phabricator.wikimedia.org/T356656) (owner: 10Majavah) [17:20:08] (03CR) 10Jforrester: "LibUp will auto-disable failing rules like that, though, except that there's an (inadequate) upstream fixer that'd run if we make my sugge" [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/997512 (https://phabricator.wikimedia.org/T356656) (owner: 10Majavah) [17:22:14] (03CR) 10Majavah: [C: 03+2] runner: fix type() usage [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/997448 (owner: 10Majavah) [17:22:21] (03CR) 10Majavah: [C: 03+2] Upgrade package-lock-lint to 0.2.4 [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/997440 (https://phabricator.wikimedia.org/T356517) (owner: 10Majavah) [17:22:38] (03CR) 10CI reject: [V: 04-1] runner: fix type() usage [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/997448 (owner: 10Majavah) [17:22:40] (03CR) 10CI reject: [V: 04-1] Upgrade package-lock-lint to 0.2.4 [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/997440 (https://phabricator.wikimedia.org/T356517) (owner: 10Majavah) [17:36:14] (03CR) 10Majavah: runner: fix type() usage [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/997448 (owner: 10Majavah) [17:36:20] (03CR) 10Majavah: [C: 03+2] "..." [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/997448 (owner: 10Majavah) [17:36:28] (03CR) 10CI reject: [V: 04-1] runner: fix type() usage [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/997448 (owner: 10Majavah) [17:36:30] (03CR) 10CI reject: [V: 04-1] Upgrade package-lock-lint to 0.2.4 [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/997440 (https://phabricator.wikimedia.org/T356517) (owner: 10Majavah) [17:42:53] 10Tool-gitlab-account-approval, 10Patch-For-Review, 10User-bd808: gitlab-account-approval bot stalled on 2024-01-09 - https://phabricator.wikimedia.org/T356097 (10CodeReviewBot) bd808 merged https://gitlab.wikimedia.org/toolforge-repos/gitlab-account-approval/-/merge_requests/11 Run `approve` under `timeout... [17:58:55] 10cloud-services-team, 10superset.wmcloud.org, 10Infrastructure Security, 10Security-Team, and 4 others: 2 CVE's in SuperSet - https://phabricator.wikimedia.org/T345928 (10sbassett) p:05Triage→03Medium [18:01:49] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [18:36:17] (OpenstackAPIResponse) firing: (3) Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [18:49:18] 10Tool-gitlab-account-approval, 10Patch-For-Review, 10User-bd808: gitlab-account-approval bot stalled on 2024-01-09 - https://phabricator.wikimedia.org/T356097 (10bd808) Looking into places where timeouts may be missing: >>! In T356097#9504524, @bd808 wrote: > The "glaab.utils INFO: Checking monx94" log line... [18:53:13] (03Abandoned) 10Majavah: runner: fix type() usage [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/997448 (owner: 10Majavah) [18:53:28] (03Abandoned) 10Majavah: Upgrade package-lock-lint to 0.2.4 [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/997440 (https://phabricator.wikimedia.org/T356517) (owner: 10Majavah) [18:53:32] (03Abandoned) 10Majavah: Drop support for lockfileVersion 1 [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/997441 (owner: 10Majavah) [18:53:35] (03Abandoned) 10Majavah: Add support for lockfileVersion v3 [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/997463 (https://phabricator.wikimedia.org/T356517) (owner: 10Majavah) [19:18:41] (03Abandoned) 10Jforrester: stylelint: Run auto-fixer, it's quite good nowadays [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/897375 (owner: 10Jforrester) [20:01:02] (OpenstackAPIResponse) firing: (3) Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [20:01:18] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack [20:01:25] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) [20:03:37] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack [20:03:43] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) [20:37:41] (CloudVPSDesignateLeaks) firing: Detected 16 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [20:42:41] (CloudVPSDesignateLeaks) firing: (2) Detected 16 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [20:44:18] PROBLEM - Memcached on cloudservices1005 is CRITICAL: connect to address 10.64.150.5 and port 11000: Connection refused https://wikitech.wikimedia.org/wiki/Memcached [20:45:01] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack [20:45:08] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) [20:46:32] PROBLEM - Memcached on cloudservices1006 is CRITICAL: connect to address 10.64.151.4 and port 11000: Connection refused https://wikitech.wikimedia.org/wiki/Memcached [21:01:49] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [21:05:42] RECOVERY - Memcached on cloudservices1005 is OK: TCP OK - 0.010 second response time on 10.64.150.5 port 11211 https://wikitech.wikimedia.org/wiki/Memcached [21:05:42] RECOVERY - Memcached on cloudservices1006 is OK: TCP OK - 0.006 second response time on 10.64.151.4 port 11211 https://wikitech.wikimedia.org/wiki/Memcached [21:11:02] (OpenstackAPIResponse) firing: (2) Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [21:29:34] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack [21:29:42] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) [21:35:12] (CloudVPSDesignateLeaks) resolved: (2) Detected 16 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [21:37:02] 10Tool-gitlab-account-approval, 10Patch-For-Review, 10User-bd808: gitlab-account-approval bot stalled on 2024-01-09 - https://phabricator.wikimedia.org/T356097 (10bd808) In local testing, the LDAP connection is **not** raising errors when no server is available. Instead things just seem to hang indefinitely... [22:22:48] (03PS1) 10Jforrester: releases: Upgrade eslint-config-wikimedia to 0.26.0 [labs/libraryupgrader/config] - 10https://gerrit.wikimedia.org/r/997595 [22:22:54] (03CR) 10Jforrester: [C: 03+2] releases: Upgrade eslint-config-wikimedia to 0.26.0 [labs/libraryupgrader/config] - 10https://gerrit.wikimedia.org/r/997595 (owner: 10Jforrester) [22:24:30] (03Merged) 10jenkins-bot: releases: Upgrade eslint-config-wikimedia to 0.26.0 [labs/libraryupgrader/config] - 10https://gerrit.wikimedia.org/r/997595 (owner: 10Jforrester) [22:30:23] 10Tool-gitlab-account-approval, 10Patch-For-Review, 10User-bd808: gitlab-account-approval bot stalled on 2024-01-09 - https://phabricator.wikimedia.org/T356097 (10bd808) >>! In T356097#9515296, @bd808 wrote: > In local testing, the LDAP connection is **not** raising errors when no server is available. Instea... [22:42:26] 10Tool-gitlab-account-approval, 10Patch-For-Review, 10User-bd808: gitlab-account-approval bot stalled on 2024-01-09 - https://phabricator.wikimedia.org/T356097 (10CodeReviewBot) bd808 opened https://gitlab.wikimedia.org/toolforge-repos/gitlab-account-approval/-/merge_requests/12 Add network timeouts and oth... [22:51:27] 10Tool-gitlab-account-approval, 10Patch-For-Review, 10User-bd808: gitlab-account-approval bot stalled on 2024-01-09 - https://phabricator.wikimedia.org/T356097 (10CodeReviewBot) bd808 merged https://gitlab.wikimedia.org/toolforge-repos/gitlab-account-approval/-/merge_requests/12 Add network timeouts and oth... [22:59:39] 10Tool-gitlab-account-approval, 10Patch-For-Review, 10User-bd808: gitlab-account-approval bot stalled on 2024-01-09 - https://phabricator.wikimedia.org/T356097 (10bd808) 05Open→03Resolved I think that between the `timeout 3m ...` protection; fixing the unintuitive LDAPserver pool infinite search loop (T3... [23:22:12] 10Grid-Engine-to-K8s-Migration: Migrate fiwiki-tools from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319745 (10Zache) Hi, thanks for the notification. I think that I have converted all of my tools to kubernetes by now. Or least I do not see any of my tools any more in the l... [23:22:36] 10Grid-Engine-to-K8s-Migration, 10Wiki-Loves-Monuments-Database: Migrate heritage from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319787 (10Slaporte) a:05Slaporte→03JeanFred [23:25:21] 10Grid-Engine-to-K8s-Migration: Migrate fiwiki-tools from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319745 (10komla) @Zache, thanks. You can go ahead and mark the ticket as resolved. I can also do this with your permission. [23:27:23] 10Grid-Engine-to-K8s-Migration: Migrate fiwiki-tools from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319745 (10Zache) 05Open→03Resolved