[02:18:50] FIRING: ProbeDown: Service tools-static-15:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-15:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [02:23:50] RESOLVED: ProbeDown: Service tools-static-15:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-15:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:53:44] 06cloud-services-team, 10Toolforge, 07Documentation, 07good first task: Find and fix inaccuracies in Toolforge Django tutorial - https://phabricator.wikimedia.org/T245683#11218108 (103MindedScholar) Is this gitlab project? [06:45:03] !log tools.cluebotng-trainer Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18030092297 (https://github.com/cluebotng/component-configs/commits/2516589b458034b7c2540c998b29debeb63e82df) [06:45:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-trainer/SAL [06:47:36] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18030114498 (https://github.com/cluebotng/component-configs/commits/ea47ef95beb4cf8a1b7d439a83af7b2d4cf168ce) [06:47:38] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [07:06:34] 06cloud-services-team, 10Toolforge, 07Documentation, 07good first task: Find and fix inaccuracies in Toolforge Django tutorial - https://phabricator.wikimedia.org/T245683#11218173 (10Dzahn) No, not a gitlab project, It's about writing documentation on a wiki: https://wikitech.wikimedia.org/w/index.php?t... [07:19:55] 06cloud-services-team, 10Toolforge, 07Documentation, 07good first task: Find and fix inaccuracies in Toolforge Django tutorial - https://phabricator.wikimedia.org/T245683#11218192 (10VRATEJDWIVEDI) a:03VRATEJDWIVEDI Hi, I’d like to work on this task as a newcomer. I’ll go through the tutorial, check for... [07:53:21] 10Cloud-VPS (Quota-requests): Disk quota increase for project catalyst - https://phabricator.wikimedia.org/T405584#11218230 (10jnuche) > +1, will the snapshots and such be temporary? (curious, would be nice to clean them up when not needed) Yeah, in theory we will only need to create them a few times for sp... [07:58:08] (03open) 10filippo: Stop managing nfs records [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/87 [08:07:03] !log godog@r5 toolsbeta START - Cookbook wmcs.nfs.add_server [08:07:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [08:17:27] !log godog@r5 toolsbeta END (PASS) - Cookbook wmcs.nfs.add_server (exit_code=0) [08:17:31] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [08:52:10] 06cloud-services-team (FY2025/26-Q1), 10Toolforge (Toolforge iteration 24): [infra,haproxy,ingress] 2025-09-23 Ingress hitting the backend session limit and started replying with 5xxs - https://phabricator.wikimedia.org/T405280#11218399 (10fnegri) 05Resolved→03In progress Reopening as last night @DamianZar... [08:53:14] 06cloud-services-team (FY2025/26-Q1), 10Toolforge (Toolforge iteration 24): [infra,haproxy,ingress] 2025-09-23 Ingress hitting the backend session limit and started replying with 5xxs - https://phabricator.wikimedia.org/T405280#11218405 (10fnegri) p:05High→03Medium Lowering priority to medium as after the... [09:07:42] (03CR) 10FNegri: global: minor cleanups (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1186483 (owner: 10David Caro) [09:17:15] !log dcaro@acme paws END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) [09:17:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL [09:41:23] (03update) 10filippo: Stop managing nfs records [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/87 [10:29:28] FIRING: [2x] NodeTextfileStale: Stale textfile for cloudcephosd1022:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [10:34:28] RESOLVED: [9x] NodeTextfileStale: Stale textfile for cloudcephosd1018:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [10:40:03] 06cloud-services-team, 10Toolforge: phaste on toolforge doesn't work "ERR-INVALID-CERTIFICATE: Your authentication certificate for this server is invalid." - https://phabricator.wikimedia.org/T252304#11218752 (10taavi) 05Open→03Invalid phaste is no longer installed on the bastions: `lang=shell-session... [10:45:29] 06cloud-services-team, 10Toolforge: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T405171#11218767 (10taavi) a:03taavi [10:47:33] 06cloud-services-team, 10Striker: Unread alerts should be much more visible - https://phabricator.wikimedia.org/T405526#11218768 (10taavi) p:05Triage→03Low [10:48:47] 06cloud-services-team, 10Striker, 07Design: Striker should use Codex - https://phabricator.wikimedia.org/T380114#11218771 (10taavi) p:05Triage→03Low [10:49:06] 06cloud-services-team: SystemdUnitDown The systemd unit wmf_auto_restart_smartd.service on node cloudvirtlocal1003 has been failing for more than two hours. - https://phabricator.wikimedia.org/T405349#11218773 (10taavi) 05Open→03Invalid This has sorted itself [10:49:38] 06cloud-services-team, 10Toolforge: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T405171#11218775 (10taavi) 05Open→03Resolved p:05Triage→03Medium [10:54:19] 06cloud-services-team, 10PAWS, 06Commons, 10OpenRefine: New upstream release for Wikimedia Commons Extension for OpenRefine - https://phabricator.wikimedia.org/T403780#11218792 (10taavi) p:05Unbreak!→03High Please note that setting a priority is a very effective way to hide a task from our triage proce... [10:54:36] 10PAWS, 10OpenRefine: New upstream release for OpenRefine - https://phabricator.wikimedia.org/T388928#11218797 (10taavi) p:05Unbreak!→03High [10:54:42] 10PAWS, 10OpenRefine: New upstream release for OpenRefine - https://phabricator.wikimedia.org/T388928#11218798 (10taavi) [10:54:46] 06cloud-services-team, 10PAWS, 06Commons, 10OpenRefine: New upstream release for Wikimedia Commons Extension for OpenRefine - https://phabricator.wikimedia.org/T403780#11218799 (10taavi) [10:56:21] 06cloud-services-team (FY2025/26-Q1), 10Toolforge (Toolforge iteration 24): [infra,haproxy,ingress] 2025-09-23 Ingress hitting the backend session limit and started replying with 5xxs - https://phabricator.wikimedia.org/T405280#11218803 (10DamianZaremba) I would imagine it's because the backend /server/ has a... [10:58:47] 06cloud-services-team (FY2025/26-Q1), 10Toolforge (Toolforge iteration 24): [infra,haproxy,ingress] 2025-09-23 Ingress hitting the backend session limit and started replying with 5xxs - https://phabricator.wikimedia.org/T405280#11218829 (10DamianZaremba) https://gerrit.wikimedia.org/r/plugins/gitiles/operation... [11:00:09] supertassu opened https://github.com/toolforge/paws/pull/501 [11:30:33] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: Update Toolforge client packages to build on Trixie only - https://phabricator.wikimedia.org/T404733#11218928 (10taavi) p:05Triage→03Low [11:33:01] 06cloud-services-team, 10Toolforge: [jobs-cli] `images` returns images that do not exist - https://phabricator.wikimedia.org/T405642#11218935 (10taavi) These images do exist on the registry, for example: `lang=shell-session taavi@runko:~ $ docker pull tools-harbor.wmcloud.org/tool-cluebotng-review/export-stati... [11:33:13] 06cloud-services-team, 10Toolforge: Toolforge jobs for milhistbot not running - https://phabricator.wikimedia.org/T405310#11218936 (10taavi) 05Open→03Resolved a:03dcaro [11:34:47] 10Cloud-VPS (Debian Bullseye Deprecation), 10CFSSL-PKI, 06Infrastructure-Foundations: Rebuild VMs in PKI cloud-vps project - https://phabricator.wikimedia.org/T405017#11218940 (10taavi) [11:35:28] 10Cloud-VPS (Debian Bullseye Deprecation), 10CFSSL-PKI, 06Infrastructure-Foundations: Rebuild VMs in PKI cloud-vps project - https://phabricator.wikimedia.org/T405017#11218945 (10taavi) Repurposing this task for that, please close or adjust if you want to track that somewhere else! [11:36:09] 06cloud-services-team, 10Toolforge: disable_tool fails to archive mpic-alpha-demo - https://phabricator.wikimedia.org/T404072#11218948 (10taavi) p:05Triage→03Medium a:03taavi [11:38:31] 06cloud-services-team, 10Toolforge: disable_tool fails to archive mpic-alpha-demo - https://phabricator.wikimedia.org/T404072#11218961 (10taavi) I manually checked that tool has no databases and then deleted the config fiel: `lang=shell-session root@tools-nfs-2:/srv/tools/project/mpic-alpha-demo# chattr -i rep... [11:39:11] 06cloud-services-team, 10Toolforge: Improve detection of failing ssh to toolforge bastions - https://phabricator.wikimedia.org/T404054#11218962 (10taavi) p:05Triage→03Medium [11:39:21] 06cloud-services-team, 10Cloud-VPS: Add fqdn input to instance-related wmcs cookbooks - https://phabricator.wikimedia.org/T404052#11218963 (10taavi) p:05Triage→03Low [11:42:24] 06cloud-services-team, 10Toolforge: disable_tool fails to archive mpic-alpha-demo - https://phabricator.wikimedia.org/T404072#11218971 (10taavi) 05Open→03Resolved [11:47:17] FIRING: JobUnavailable: Reduced availability for job maintain_dbusers_eqiad in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [11:47:27] 06cloud-services-team: JobUnavailable Reduced availability for job maintain_dbusers_eqiad in cloud@eqiad - https://phabricator.wikimedia.org/T405728 (10phaultfinder) 03NEW [11:51:56] FIRING: SystemdUnitDown: The service unit maintain-dbusers.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [11:52:26] 06cloud-services-team, 10Toolforge: [jobs-cli] `images` returns images that do not exist - https://phabricator.wikimedia.org/T405642#11218984 (10DamianZaremba) > So is there some process that's supposed to delete them from there but is failing? Or is this more or less a feature request to add that? I'm 90% sur... [11:52:39] 06cloud-services-team, 10Toolforge: [jobs-cli] `images` returns images that do not exist - https://phabricator.wikimedia.org/T405642#11218985 (10DamianZaremba) 05Open→03Declined [11:53:52] 06cloud-services-team, 10Data-Services: JobUnavailable Reduced availability for job maintain_dbusers_eqiad in cloud@eqiad - https://phabricator.wikimedia.org/T405728#11218986 (10taavi) a:03taavi [11:55:30] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18036958606 (https://github.com/cluebotng/component-configs/commits/a51fe109bfad3e2df5aa8e89b837a951bf8ad2cf) [11:55:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [11:56:56] RESOLVED: SystemdUnitDown: The service unit maintain-dbusers.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [11:57:26] FIRING: SystemdUnitDown: The service unit maintain-dbusers.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [12:01:26] 10Toolforge (Quota-requests): Request increased build quota for cluebotng Toolforge tool - https://phabricator.wikimedia.org/T405645#11219007 (10DamianZaremba) Hit again today ` tools.cluebotng@tools-bastion-15:~$ toolforge build quota Registry =================== Storage ----------- Available 2.09Mi Capacity... [12:09:59] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18037156332 (https://github.com/cluebotng/component-configs/commits/a51fe109bfad3e2df5aa8e89b837a951bf8ad2cf) [12:10:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [12:22:52] (03update) 10filippo: Stop managing nfs records [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/87 [12:37:06] 06cloud-services-team, 10Cloud-VPS, 10Ceph: Review RAM allocation for cloudceph OSDs - https://phabricator.wikimedia.org/T404747#11219062 (10taavi) p:05Triage→03Medium [12:39:39] !log tools.cluebotng-staging Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18037926539 (https://github.com/cluebotng/component-configs/commits/4950150f14c22c0a7d3df1739fa5537aeba4157d) [12:39:41] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-staging/SAL [12:40:24] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18037926507 (https://github.com/cluebotng/component-configs/commits/4950150f14c22c0a7d3df1739fa5537aeba4157d) [12:40:25] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [12:46:13] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18038096319 (https://github.com/cluebotng/component-configs/commits/e10e601b4fb06b3fd97856ef86a30e5391fb4f17) [12:46:15] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [12:57:11] RESOLVED: SystemdUnitDown: The service unit maintain-dbusers.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [12:57:17] RESOLVED: JobUnavailable: Reduced availability for job maintain_dbusers_eqiad in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [13:00:44] 06cloud-services-team, 10Data-Services, 13Patch-For-Review: JobUnavailable Reduced availability for job maintain_dbusers_eqiad in cloud@eqiad - https://phabricator.wikimedia.org/T405728#11219265 (10taavi) 05Open→03Resolved [13:11:20] 06cloud-services-team (FY2025/26-Q1), 10Toolforge (Toolforge iteration 24): [infra,haproxy,ingress] 2025-09-23 Ingress hitting the backend session limit and started replying with 5xxs - https://phabricator.wikimedia.org/T405280#11219365 (10fnegri) > https://gerrit.wikimedia.org/r/plugins/gitiles/operations/pup... [13:14:08] 06cloud-services-team (FY2025/26-Q1), 10Toolforge (Toolforge iteration 24): [infra,haproxy,ingress] 2025-09-23 Ingress hitting the backend session limit and started replying with 5xxs - https://phabricator.wikimedia.org/T405280#11219372 (10DamianZaremba) > The graph above shows it goes slightly above 6000 thou... [13:46:18] (03update) 10fnegri: Stop managing nfs records [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/87 (owner: 10filippo) [13:46:30] (03approved) 10fnegri: Stop managing nfs records [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/87 (owner: 10filippo) [13:47:59] (03merge) 10filippo: Stop managing nfs records [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/87 [13:50:40] 10Cloud Services Proposals, 06cloud-services-team (FY2025/26-Q1), 10Data-Services, 06Data-Persistence, 06Data-Platform-SRE (2025.09.26 - 2025.10.17): Decision request - Who runs wikireplicas cookbooks - https://phabricator.wikimedia.org/T382607#11219516 (10Gehel) [13:51:01] 06cloud-services-team, 10Data-Services, 06Data-Engineering, 06Data-Engineering-Radar, and 3 others: Create wiki replicas views for globaljsonlinks tables - https://phabricator.wikimedia.org/T387419#11219522 (10Gehel) [13:57:17] FIRING: JobUnavailable: Reduced availability for job maintain_dbusers_eqiad in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [13:59:28] 06cloud-services-team: JobUnavailable Reduced availability for job maintain_dbusers_eqiad in cloud@eqiad - https://phabricator.wikimedia.org/T405740 (10phaultfinder) 03NEW [14:00:30] 10VPS-project-Wikistats, 06Product Safety and Integrity: Add a column for the blocks in WikiStats - https://phabricator.wikimedia.org/T264814#11219763 (10OKryva-WMF) [14:02:56] FIRING: SystemdUnitDown: The service unit maintain-dbusers.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [14:03:01] 06cloud-services-team, 10Data-Services, 06Community-Tech, 10Multiblocks: Unexpected error "Subquery returns more than 1 row" on wiki replicas - https://phabricator.wikimedia.org/T404473#11219807 (10Gehel) [14:03:46] 06cloud-services-team, 10Toolforge: tofu-provisioning: Failed to install provider - https://phabricator.wikimedia.org/T405742 (10fnegri) 03NEW [14:03:51] 06cloud-services-team, 10Toolforge: tofu-provisioning: Failed to install provider - https://phabricator.wikimedia.org/T405742#11219841 (10fnegri) p:05Triage→03High [14:08:14] 06cloud-services-team, 10Toolforge: tofu-provisioning: Failed to install provider - https://phabricator.wikimedia.org/T405742#11219873 (10fnegri) Maybe this new OpenTofu feature could help (haven't read the details yet): > OCI Registry Support - Distribute providers and modules through container registries, p... [14:27:17] RESOLVED: JobUnavailable: Reduced availability for job maintain_dbusers_eqiad in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [14:27:56] RESOLVED: SystemdUnitDown: The service unit maintain-dbusers.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [14:35:17] FIRING: JobUnavailable: Reduced availability for job maintain_dbusers_eqiad in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [14:40:56] FIRING: SystemdUnitDown: The service unit maintain-dbusers.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [15:00:17] RESOLVED: JobUnavailable: Reduced availability for job maintain_dbusers_eqiad in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [15:00:56] RESOLVED: SystemdUnitDown: The service unit maintain-dbusers.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [15:24:24] 10VPS-project-Wikistats, 06Trust-and-Safety: Add a column for the blocks in WikiStats - https://phabricator.wikimedia.org/T264814#11220337 (10OKryva-WMF) [15:44:46] (03CR) 10David Caro: global: minor cleanups (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1186483 (owner: 10David Caro) [16:02:48] 10VPS-project-Phabricator, 06collaboration-services, 06Release-Engineering-Team (Radar): 'Fulltext' searches fail on test Phab instance due to ElasticSearch default config (PhutilAggregateException: All Fulltext Search hosts failed / CURLE_COULDNT_CONNECT) - https://phabricator.wikimedia.org/T403948#11220474 (... [16:30:28] 10VPS-project-Phabricator, 06collaboration-services, 06Release-Engineering-Team (Radar): 'Fulltext' searches fail on test Phab instance due to ElasticSearch default config (PhutilAggregateException: All Fulltext Search hosts failed / CURLE_COULDNT_CONNECT) - https://phabricator.wikimedia.org/T403948#11220568 (... [16:56:12] 10VPS-project-Phabricator, 06collaboration-services, 06Release-Engineering-Team (Radar): 'Fulltext' searches fail on test Phab instance due to ElasticSearch default config (PhutilAggregateException: All Fulltext Search hosts failed / CURLE_COULDNT_CONNECT) - https://phabricator.wikimedia.org/T403948#11220641 (... [17:11:25] 10VPS-project-Phabricator, 06collaboration-services, 06Release-Engineering-Team (Radar): 'Fulltext' searches fail on test Phab instance due to ElasticSearch default config (PhutilAggregateException: All Fulltext Search hosts failed / CURLE_COULDNT_CONNECT) - https://phabricator.wikimedia.org/T403948#11220739 (... [17:13:16] 10VPS-project-Phabricator, 06collaboration-services, 06Release-Engineering-Team (Radar): 'Fulltext' searches fail on test Phab instance due to ElasticSearch default config (PhutilAggregateException: All Fulltext Search hosts failed / CURLE_COULDNT_CONNECT) - https://phabricator.wikimedia.org/T403948#11220745 (... [18:16:37] 10VPS-project-Phabricator, 06collaboration-services, 06Release-Engineering-Team (Radar): 'Fulltext' searches fail on test Phab instance due to ElasticSearch default config (PhutilAggregateException: All Fulltext Search hosts failed / CURLE_COULDNT_CONNECT) - https://phabricator.wikimedia.org/T403948#11220920 (... [18:22:41] 06cloud-services-team, 10Toolforge: [builds-api] Gitlab maintenance should not cause an outage for builds - https://phabricator.wikimedia.org/T405782#11220946 (10DamianZaremba) [18:26:50] 06cloud-services-team, 10Toolforge: [builds-api] Gitlab maintenance should not cause an outage for builds - https://phabricator.wikimedia.org/T405782#11220993 (10DamianZaremba) [18:34:10] 06cloud-services-team, 10Toolforge: [jobs-cli] `job_type` not handled in `dump`, generates warning per job - https://phabricator.wikimedia.org/T405786 (10DamianZaremba) 03NEW [18:54:06] 06cloud-services-team, 10Toolforge: [jobs-cli] `job_type` not handled in `dump`, generates warning per job - https://phabricator.wikimedia.org/T405786#11221062 (10DamianZaremba) Appears to come from https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/commit/75109675bfd56a2b36242923331794aaed252773 so... [18:54:47] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18046649036 (https://github.com/cluebotng/component-configs/commits/07b907ff75f0289f350549bae5e75bf4e91c91ca) [18:54:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [18:55:39] !log tools.cluebotng-staging Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18046649035 (https://github.com/cluebotng/component-configs/commits/07b907ff75f0289f350549bae5e75bf4e91c91ca) [18:55:40] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-staging/SAL [19:00:34] 06cloud-services-team, 10Toolforge: toolforge logs appears to suffer from intermittent latency - https://phabricator.wikimedia.org/T402736#11221072 (10DamianZaremba) Today a job managed to get partial logs... not the start, not the end, but something in the middle =\ ` tools.cluebotng@tools-bastion-15:~$ tool... [19:03:55] 10VPS-project-Phabricator, 06collaboration-services, 06Release-Engineering-Team (Radar): 'Fulltext' searches fail on test Phab instance due to ElasticSearch default config (PhutilAggregateException: All Fulltext Search hosts failed / CURLE_COULDNT_CONNECT) - https://phabricator.wikimedia.org/T403948#11221083 (... [19:04:44] 10VPS-project-Phabricator, 06collaboration-services, 06Release-Engineering-Team (Radar): 'Fulltext' searches fail on test Phab instance due to ElasticSearch default config (PhutilAggregateException: All Fulltext Search hosts failed / CURLE_COULDNT_CONNECT) - https://phabricator.wikimedia.org/T403948#11221084 (... [22:18:56] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18050560602 (https://github.com/cluebotng/component-configs/commits/886ded0824a9ce7b27c852949f3530bda15bef14) [22:18:59] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL