[00:37:29] (03open) 10raymond-ndibe: Draft: [jobs-api] periodically refresh image-config data [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/160 (https://phabricator.wikimedia.org/T357112) [00:37:36] (03update) 10raymond-ndibe: Draft: [jobs-api] periodically refresh image-config data [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/160 (https://phabricator.wikimedia.org/T357112) [06:33:27] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/28 [06:33:30] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/69 [06:33:38] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/69 [06:33:38] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: pre-commit: Autoupdate [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/65 (owner: 10ghost) [06:33:39] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/100 [06:33:44] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/100 [06:33:50] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: pre-commit: Autoupdate [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/65 (owner: 10ghost) [06:33:51] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/24 [06:33:56] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/20 [06:34:02] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/20 [06:34:07] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/24 [06:34:15] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/20 [06:34:18] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/20 [06:34:31] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/161 [06:34:35] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/161 [06:34:41] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/21 [06:34:51] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/29 [06:34:52] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/21 [06:34:57] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/29 [06:34:59] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/55 [06:35:03] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/130 [06:35:05] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/55 [06:35:09] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/130 [06:35:09] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/63 [06:35:17] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/63 [06:35:18] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/239 [06:35:19] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/239 [07:54:38] 06cloud-services-team, 10Toolforge, 03Wikimedia-Hackathon-2025: [Session] Introducing and exploring Toolforge UI with prospective users - https://phabricator.wikimedia.org/T383149#10782995 (10CBogen) Hi! This session will be 45 minutes from 4pm-4:45pm on Saturday, May 3rd. Please refer to the program for [[... [09:31:50] FIRING: ProbeDown: Service tools-static-15:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-15:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:36:50] RESOLVED: ProbeDown: Service tools-static-15:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-15:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [10:22:50] FIRING: ProbeDown: Service tools-static-15:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-15:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [10:24:27] 10Cloud Services Proposals, 06cloud-services-team, 10Toolforge: [DRAFT] Decision Request - Initial product approach to integrate Toolforge UI functionality with Toolsadmin - https://phabricator.wikimedia.org/T393010#10783150 (10Sarai-WMF) [10:27:50] RESOLVED: ProbeDown: Service tools-static-15:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-15:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [11:12:27] (03CR) 10Chlod Alejandro: "recheck" [labs/striker] - 10https://gerrit.wikimedia.org/r/1134724 (https://phabricator.wikimedia.org/T364605) (owner: 10Arendpieter) [11:14:33] 10wikitech.wikimedia.org, 10Bitu, 10Gerrit, 06Infrastructure-Foundations: Gerrit SSH authentication fails with new Wikimedia developer account - https://phabricator.wikimedia.org/T393092 (10Douginamug) 03NEW [11:14:44] (03CR) 10CI reject: [V:04-1] Switch username validation to Bitu API [labs/striker] - 10https://gerrit.wikimedia.org/r/1134724 (https://phabricator.wikimedia.org/T364605) (owner: 10Arendpieter) [11:16:50] 10wikitech.wikimedia.org, 10Bitu, 10Gerrit, 06Infrastructure-Foundations: Gerrit SSH authentication fails with new Wikimedia developer account - https://phabricator.wikimedia.org/T393092#10783222 (10Pintoch) I have replicated this issue by creating a new Wikitech account "PintochTestDeleteMe", associated i... [11:18:08] 10wikitech.wikimedia.org, 10Bitu, 06collaboration-services, 10Gerrit, and 2 others: Gerrit SSH authentication fails with new Wikimedia developer account - https://phabricator.wikimedia.org/T393092#10783223 (10RhinosF1) [11:33:50] FIRING: ProbeDown: Service tools-static-15:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-15:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [11:38:50] RESOLVED: ProbeDown: Service tools-static-15:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-15:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:26:33] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [labs/tools/weapon-of-mass-description] - 10https://gerrit.wikimedia.org/r/1140495 (owner: 10L10n-bot) [13:26:50] FIRING: ProbeDown: Service tools-static-15:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-15:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [13:31:50] RESOLVED: ProbeDown: Service tools-static-15:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-15:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [13:34:52] 10Tool-paulina: Editing interface for Paulina tool - https://phabricator.wikimedia.org/T392429#10783306 (10Pepe_piton) [13:51:50] FIRING: ProbeDown: Service tools-static-15:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-15:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [13:56:50] RESOLVED: ProbeDown: Service tools-static-15:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-15:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:01:30] 10Tool-paulina, 03Wikimedia-Hackathon-2025: Improve the Paulina tool through different tasks - https://phabricator.wikimedia.org/T393096 (10Pepe_piton) 03NEW [15:19:22] 10wikitech.wikimedia.org, 10Bitu, 06collaboration-services, 10Gerrit, and 2 others: Gerrit SSH authentication fails with new Wikimedia developer account - https://phabricator.wikimedia.org/T393092#10783501 (10thcipriani) hrm. Here's what I see in the ssh logs on gerrit: ` [2025-05-01T09:39:20.636Z] c5c5d... [15:22:46] 10wikitech.wikimedia.org, 10Bitu, 06collaboration-services, 10Gerrit, and 2 others: Gerrit SSH authentication fails with new Wikimedia developer account - https://phabricator.wikimedia.org/T393092#10783508 (10thcipriani) Confirming Gerrit can connect to ldap ` thcipriani@gerrit1003:/srv$ nc -vz ldap-ro.eq... [15:40:28] 10wikitech.wikimedia.org, 10Bitu, 06collaboration-services, 10Gerrit, and 2 others: Gerrit authentication fails with new Wikimedia developer account - https://phabricator.wikimedia.org/T393092#10783587 (10thcipriani) [15:46:58] 10wikitech.wikimedia.org, 10Bitu, 06collaboration-services, 10Gerrit, and 2 others: Gerrit authentication fails with new Wikimedia developer account - https://phabricator.wikimedia.org/T393092#10783620 (10thcipriani) Not just ssh auth. Another hint, the last new account to successfully authenticate was at... [15:59:31] 10wikitech.wikimedia.org, 10Bitu, 06collaboration-services, 10Gerrit, and 2 others: Gerrit SSH authentication fails with new Wikimedia developer account - https://phabricator.wikimedia.org/T393092#10783687 (10thcipriani) [16:01:40] 10wikitech.wikimedia.org, 10Bitu, 06collaboration-services, 10Gerrit, and 2 others: Gerrit SSH authentication fails with new Wikimedia developer account - https://phabricator.wikimedia.org/T393092#10783695 (10thcipriani) Nevermind. I thought it was not just ssh, then remembered creating an account on wikit... [16:11:28] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-codfw: Q4:rack/setup/install cloudcontrol2010-dev - https://phabricator.wikimedia.org/T393102 (10RobH) 03NEW [16:12:06] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-codfw: Q4:rack/setup/install cloudcontrol2010-dev - https://phabricator.wikimedia.org/T393102#10783752 (10RobH) [16:12:28] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-codfw: Q4:rack/setup/install cloudcontrol2010-dev - https://phabricator.wikimedia.org/T393102#10783754 (10RobH) a:03Andrew @andrew, Please note the workflow for racking tasks has changed this fiscal year, and we now require the puppet updates from the sub-... [18:31:54] 10Tool-wosretbot: [WosretBot] Gazette stops working when subscribers become inactive - https://phabricator.wikimedia.org/T393114 (10Tkarcher) 03NEW [19:09:54] 10Tool-wosretbot: [WosretBot] Gazette stops working when Kurier section has no date in signature - https://phabricator.wikimedia.org/T393116 (10Tkarcher) 03NEW [20:34:06] 10Tool-refill: Use full month name in date - https://phabricator.wikimedia.org/T393124 (10Ponor) 03NEW [20:53:50] FIRING: ProbeDown: Service tools-static-15:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-15:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [20:58:50] RESOLVED: ProbeDown: Service tools-static-15:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-15:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [21:13:33] 10cloud-services-team (FY2024/2025-Q3-Q4), 06DC-Ops, 10ops-eqiad, 06SRE: Temperature Inlet Temp issue on clouddumps1001:9290 - https://phabricator.wikimedia.org/T383723#10784598 (10Jclark-ctr) Made some more adjustments will see how this effects cooling [21:28:37] 06cloud-services-team, 10Data-Services, 06DBA, 10Wikidata, 07User-notice: Set up x3 replication to wikireplicas - https://phabricator.wikimedia.org/T390954#10784617 (10Quiddity) For Tech News: Please could someone suggest how to phrase an entry? My best-guess would be to re-use parts of the summary from... [22:14:35] 10Tool-refill: Also fill the language= parameter with the reFill tool - https://phabricator.wikimedia.org/T392901#10784702 (10Curb_Safe_Charmer) 05Open→03Invalid The language parameter is not set by reFill. As you suspect, it is set by citoid, or more accurately, by Zotero. There are some open tasks rela... [22:16:30] 10Tool-refill: Use full month name in date - https://phabricator.wikimedia.org/T393124#10784709 (10Curb_Safe_Charmer) 05Open→03Declined This would need to be implemented in Citoid (https://phabricator.wikimedia.org/project/profile/62/) rather than reFill. [22:17:09] 10Tool-refill: Also fill the language= parameter with the reFill tool - https://phabricator.wikimedia.org/T392901#10784711 (10Curb_Safe_Charmer) 05Invalid→03Declined [22:19:00] 10Tool-refill: massive error.log file spam due to LabsDB.php PHP fatal error - https://phabricator.wikimedia.org/T389917#10784712 (10Curb_Safe_Charmer) p:05Triage→03Low [23:11:42] 10Tool-refill: Use full month name in date - https://phabricator.wikimedia.org/T393124#10784768 (10Ponor) >>! In T393124#10784709, @Curb_Safe_Charmer wrote: > This would need to be implemented in Citoid (https://phabricator.wikimedia.org/project/profile/62/) rather than reFill. @Curb_Safe_Charmer Hm, that's...