[00:01:06] (03approved) 10raymond-ndibe: [toolforge,tool_handlers] replace destination_name comparisons with image_name [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/115 (https://phabricator.wikimedia.org/T395076) [00:02:14] (03merge) 10raymond-ndibe: [toolforge,tool_handlers] replace destination_name comparisons with image_name [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/115 (https://phabricator.wikimedia.org/T395076) [00:05:54] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: components-api: bump to 0.0.170-20251021000227-485a8fbc [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1003 (https://phabricator.wikimedia.org/T395076) [00:06:18] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api [00:07:51] (03update) 10raymond-ndibe: [toolforge_deploy_mr.py] support deploy of MRs from external contributors [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/273 (https://phabricator.wikimedia.org/T394595) [00:11:18] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api [00:14:23] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component components-api [00:19:35] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api [00:29:39] (03PS4) 10Pppery: Don't trigger on reopens [labs/tools/github-pr-closer] - 10https://gerrit.wikimedia.org/r/1079048 (https://phabricator.wikimedia.org/T374157) [00:30:05] (03update) 10raymond-ndibe: components-api: bump to 0.0.170-20251021000227-485a8fbc [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1003 (https://phabricator.wikimedia.org/T395076) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:30:06] (03approved) 10raymond-ndibe: components-api: bump to 0.0.170-20251021000227-485a8fbc [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1003 (https://phabricator.wikimedia.org/T395076) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:30:13] (03merge) 10raymond-ndibe: components-api: bump to 0.0.170-20251021000227-485a8fbc [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1003 (https://phabricator.wikimedia.org/T395076) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [00:44:27] (03update) 10raymond-ndibe: [toolforge_deploy_mr.py] support deploy of MRs from external contributors [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/273 (https://phabricator.wikimedia.org/T394595) [00:45:06] (03update) 10raymond-ndibe: [toolforge_deploy_mr.py] support deploy of MRs from external contributors [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/273 (https://phabricator.wikimedia.org/T394595) [00:48:58] (03update) 10raymond-ndibe: [DO NOT MERGE] testing gitlab ci changes [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/213 [00:54:28] (03open) 10samwilson: Remove non-existant Vector config vars [toolforge-repos/wikispore-config] - 10https://gitlab.wikimedia.org/toolforge-repos/wikispore-config/-/merge_requests/4 [00:55:24] (03update) 10raymond-ndibe: [toolforge_deploy_mr.py] support deploy of MRs from external contributors [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/273 (https://phabricator.wikimedia.org/T394595) [00:57:39] (03merge) 10samwilson: Remove non-existant Vector config vars [toolforge-repos/wikispore-config] - 10https://gitlab.wikimedia.org/toolforge-repos/wikispore-config/-/merge_requests/4 [00:57:59] (03update) 10raymond-ndibe: [helm image publish]: publish to reggie repo if PR owner not repo owner [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/61 (https://phabricator.wikimedia.org/T394595) [01:04:19] (03update) 10raymond-ndibe: [helm image publish]: publish to reggie repo if PR owner not repo owner [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/61 (https://phabricator.wikimedia.org/T394595) [01:06:02] (03approved) 10raymond-ndibe: [helm image publish]: publish to reggie repo if PR owner not repo owner [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/61 (https://phabricator.wikimedia.org/T394595) [01:06:03] (03update) 10raymond-ndibe: [helm image publish]: publish to reggie repo if PR owner not repo owner [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/61 (https://phabricator.wikimedia.org/T394595) [01:07:41] (03update) 10raymond-ndibe: [helm image publish]: publish to reggie repo if PR owner not repo owner [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/61 (https://phabricator.wikimedia.org/T394595) [01:07:54] (03merge) 10raymond-ndibe: [helm image publish]: publish to reggie repo if PR owner not repo owner [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/61 (https://phabricator.wikimedia.org/T394595) [01:09:18] (03approved) 10raymond-ndibe: [toolforge_deploy_mr.py] support deploy of MRs from external contributors [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/273 (https://phabricator.wikimedia.org/T394595) [01:09:33] (03merge) 10raymond-ndibe: [toolforge_deploy_mr.py] support deploy of MRs from external contributors [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/273 (https://phabricator.wikimedia.org/T394595) [01:10:35] 06cloud-services-team (FY2025/26-Q1), 10Toolforge (Toolforge iteration 24), 13Patch-For-Review: [cicd] create cicd flow for non repo owners - https://phabricator.wikimedia.org/T394595#11292147 (10Raymond_Ndibe) [01:10:46] 06cloud-services-team (FY2025/26-Q1), 10Toolforge (Toolforge iteration 24), 13Patch-For-Review: [cicd] create cicd flow for non repo owners - https://phabricator.wikimedia.org/T394595#11292149 (10Raymond_Ndibe) 05In progress→03Resolved [01:11:41] 10Toolforge (Toolforge iteration 24), 13Patch-For-Review: [components-api] exclude defaults when getting deployment - https://phabricator.wikimedia.org/T401648#11292153 (10Raymond_Ndibe) 05In progress→03Resolved [01:14:12] (03close) 10raymond-ndibe: [DO NOT MERGE] testing gitlab ci changes [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/213 [01:26:16] 10Toolforge (Toolforge iteration 24): alloy failing in macos lima-kilo - https://phabricator.wikimedia.org/T407822 (10Raymond_Ndibe) 03NEW [01:31:04] (03update) 10raymond-ndibe: helpers: add a script to easily deploy from the code [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/285 (owner: 10dcaro) [01:43:55] (03update) 10raymond-ndibe: helpers: add a script to easily deploy from the code [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/285 (owner: 10dcaro) [02:38:35] PROBLEM - Wikitech-static main page has content on wikitech-static.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Wikitech-static [02:39:27] RECOVERY - Wikitech-static main page has content on wikitech-static.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 30031 bytes in 1.772 second response time https://wikitech.wikimedia.org/wiki/Wikitech-static [02:51:50] (03open) 10raymond-ndibe: [jobs.py] breakup get_tags_in_use_by_toolforge_deploy gitlab querying into chunks [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/54 (https://phabricator.wikimedia.org/T407496) [02:57:08] (03update) 10raymond-ndibe: [jobs.py] breakup get_tags_in_use_by_toolforge_deploy gitlab querying into chunks [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/54 (https://phabricator.wikimedia.org/T407496) [02:57:59] 10Toolforge (Toolforge iteration 24), 13Patch-For-Review: [maintain-harbor] Failing to cleanup stale artifacts - https://phabricator.wikimedia.org/T407496#11292222 (10Raymond_Ndibe) 05Open→03In progress [06:04:35] PROBLEM - Wikitech-static main page has content on wikitech-static.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Wikitech-static [06:09:33] RECOVERY - Wikitech-static main page has content on wikitech-static.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 30031 bytes in 8.613 second response time https://wikitech.wikimedia.org/wiki/Wikitech-static [06:13:35] PROBLEM - Wikitech-static main page has content on wikitech-static.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Wikitech-static [06:14:29] RECOVERY - Wikitech-static main page has content on wikitech-static.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 30031 bytes in 3.520 second response time https://wikitech.wikimedia.org/wiki/Wikitech-static [06:45:20] 06cloud-services-team, 10Toolforge: Intermittent redis connection timeouts in Toolforge - https://phabricator.wikimedia.org/T318479#11292468 (10fgiunchedi) Is this still happening? I suspect the problem was related to k8s workers periodically renewing their DHCP lease and briefly losing the network. See also {... [06:53:30] (03update) 10taavi: Allocate floating IP to HAProxy service VIP [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/94 (https://phabricator.wikimedia.org/T283948) [06:53:36] (03update) 10taavi: Allow inbound HTTPS traffic to HAProxy service VIP [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/95 (https://phabricator.wikimedia.org/T283948) [07:52:50] FIRING: [3x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_admin_toolforge_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [07:53:18] PROBLEM - toolschecker: NFS read/writeable on labs instances on checker.tools.wmflabs.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 504 Gateway Time-out - string OK not found on http://checker.tools.wmflabs.org:80/nfs/home - 324 bytes in 60.021 second response time https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolschecker [07:53:20] RECOVERY - toolschecker: NFS read/writeable on labs instances on checker.tools.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 158 bytes in 0.006 second response time https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolschecker [07:57:50] RESOLVED: [3x] ProbeDown: Service tools-k8s-haproxy-7:443 has failed probes (http_admin_toolforge_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [08:01:25] 10Cloud-VPS (Quota-requests), 10Catalyst: Quota increase request for project catalyst - https://phabricator.wikimedia.org/T407733#11292639 (10jnuche) Thank you! [08:18:23] (03update) 10dcaro: tofu-provisioning: introduce gitlab CI/CD based workflow [repos/cloud/metricsinfra/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/metricsinfra/tofu-provisioning/-/merge_requests/2 (owner: 10aborrero) [08:18:30] (03update) 10dcaro: maintain-kubeusers: add logic to drop access to deleted admins [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/71 (https://phabricator.wikimedia.org/T394786) (owner: 10aborrero) [08:22:29] 06cloud-services-team, 10Cloud-VPS: metricsinfra-thanos-fe-2 Puppet failure - https://phabricator.wikimedia.org/T407837 (10taavi) 03NEW [08:35:25] (03update) 10miiswom: Draft: Proof of concept: Add author form [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/131 [08:44:53] (03update) 10miiswom: Draft: Proof of concept: Add author form [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/131 [09:02:23] 06cloud-services-team, 10Toolforge: Intermittent redis connection timeouts in Toolforge - https://phabricator.wikimedia.org/T318479#11292839 (10fnegri) 05Open→03Resolved a:03fnegri Interesting theory. I checked the error logs for the same two tools I had checked back in March (https://phabricator.wik... [09:19:47] (03update) 10bahago: Feature/add author form [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/127 [10:59:59] 10VPS-project-Phabricator, 10Bitu, 06collaboration-services, 06Infrastructure-Foundations, 13Patch-For-Review: Allow Bitu to link Phabricator account - https://phabricator.wikimedia.org/T406495#11293245 (10SLyngshede-WMF) 05Open→03In progress [11:07:45] 10Tool-wsindex, 10Wikisource Reader App: Add an option to sort list of books alphabetically - https://phabricator.wikimedia.org/T407311#11293269 (10System625) >>! In T407311#11290047, @Bodhisattwa wrote: > @System625 , thanks for the PR. Its a festive season in India and we are celebrating [[ https://en.wikip... [11:21:23] 10Tool-paulina, 10Outreachy (Round 31): Outreachy 31: Features to edit author and work data on Wikidata directly from Paulina - https://phabricator.wikimedia.org/T392429#11293297 (10Kimbrene-kakande) @Nat_WDU . could you please look into the MR request i made on footer responsiveness. I know your prolly busy t... [11:22:23] (03open) 10volans: deployment: exclude loki-tracing namespace [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/31 [11:40:54] 10Toolforge (Toolforge iteration 24): alloy failing in macos lima-kilo - https://phabricator.wikimedia.org/T407822#11293350 (10dcaro) Yep, that's my fault, I deployed the loki stuff manually, not using the MR as it required tweaking, so I missed this, the comments should use `//` inestad of `#`, fixing [11:43:22] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component logging [11:44:25] (03open) 10dcaro: loki: use // for comments [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1004 [11:45:02] (03approved) 10dcaro: loki: use // for comments [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1004 [11:45:09] (03merge) 10dcaro: loki: use // for comments [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1004 [11:48:13] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging [11:48:24] 10Toolforge (Toolforge iteration 24): alloy failing in macos lima-kilo - https://phabricator.wikimedia.org/T407822#11293361 (10dcaro) 05Open→03Resolved a:03dcaro Fixed in https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1004 [12:30:46] (03close) 10volans: deployment: exclude loki-tracing namespace [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/31 [12:59:41] 10Toolforge (Toolforge iteration 24): [infra,logging] prometheus failing to fetch the metrics endpoint - https://phabricator.wikimedia.org/T407852 (10dcaro) 03NEW [13:05:40] 06cloud-services-team, 10Data-Services, 06Data-Engineering, 06Data-Persistence, and 3 others: Set up x1 replication to Wiki Replicas - https://phabricator.wikimedia.org/T395881#11293564 (10Gehel) p:05Triage→03Medium [13:32:37] (03update) 10miiswom: Draft: Proof of concept: Add author form [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/131 [14:01:08] (03approved) 10filippo: Allocate floating IP to HAProxy service VIP [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/94 (https://phabricator.wikimedia.org/T283948) (owner: 10taavi) [14:01:27] (03approved) 10filippo: Allow inbound HTTPS traffic to HAProxy service VIP [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/95 (https://phabricator.wikimedia.org/T283948) (owner: 10taavi) [14:06:30] (03update) 10miiswom: Draft: Proof of concept: Add author form [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/131 [14:08:09] (03merge) 10taavi: Allocate floating IP to HAProxy service VIP [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/94 (https://phabricator.wikimedia.org/T283948) [14:08:10] (03update) 10taavi: Allow inbound HTTPS traffic to HAProxy service VIP [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/95 (https://phabricator.wikimedia.org/T283948) [14:17:07] (03update) 10taavi: Allow inbound HTTPS traffic to HAProxy service VIP [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/95 (https://phabricator.wikimedia.org/T283948) [14:17:08] (03update) 10taavi: toolsbeta: Point web traffic to HAProxy directly [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/96 (https://phabricator.wikimedia.org/T283948) [14:18:44] (03merge) 10taavi: Allow inbound HTTPS traffic to HAProxy service VIP [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/95 (https://phabricator.wikimedia.org/T283948) [14:18:46] (03update) 10taavi: toolsbeta: Point web traffic to HAProxy directly [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/96 (https://phabricator.wikimedia.org/T283948) [14:27:17] (03update) 10taavi: toolsbeta: Point web traffic to HAProxy directly [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/96 (https://phabricator.wikimedia.org/T283948) [15:05:42] (03update) 10damian: Draft: Use the image name as provided by builds-api [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/143 (https://phabricator.wikimedia.org/T403322) [15:09:22] (03update) 10damian: Draft: Use the image name as provided by builds-api [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/143 (https://phabricator.wikimedia.org/T403322) [15:10:18] (03update) 10miiswom: Draft: Proof of concept: Add author form [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/131 [15:11:07] (03update) 10damian: Draft: Use the image name as provided by builds-api [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/143 (https://phabricator.wikimedia.org/T403322) [15:11:36] (03update) 10damian: Draft: Use the image name as provided by builds-api [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/143 (https://phabricator.wikimedia.org/T403322) [15:21:47] (03update) 10damian: Use the image name as provided by builds-api [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/143 (https://phabricator.wikimedia.org/T403322) [15:22:16] (03update) 10damian: Use the image name as provided by builds-api [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/143 (https://phabricator.wikimedia.org/T403322) [15:32:49] (03update) 10bahago: Feature/add author form [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/127 [15:37:26] (03update) 10bahago: Feature/add work [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/130 [15:37:35] (03update) 10dipanshu1223: T406989: Add dropdown to populate empty biography fields with input [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/50 [15:41:00] (03update) 10taavi: toolsbeta: Point web traffic to HAProxy directly [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/96 (https://phabricator.wikimedia.org/T283948) [15:41:01] (03update) 10taavi: tools: Point web traffic to HAProxy directly [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/97 (https://phabricator.wikimedia.org/T283948) [15:41:02] (03open) 10taavi: tools: Point web traffic to HAProxy directly [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/97 (https://phabricator.wikimedia.org/T283948) [15:41:06] (03update) 10taavi: tools: Point web traffic to HAProxy directly [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/97 (https://phabricator.wikimedia.org/T283948) [15:43:17] FIRING: JobUnavailable: Reduced availability for job maintain_dbusers_eqiad in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [15:43:23] 06cloud-services-team: JobUnavailable Reduced availability for job maintain_dbusers_eqiad in cloud@eqiad - https://phabricator.wikimedia.org/T407868 (10phaultfinder) 03NEW [15:47:44] !log filippo@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.undrain_node (T405478) [15:47:51] T405478: Experiment with cloudcephosd1050 and cloudcephosd1051 in single-nic configuration - https://phabricator.wikimedia.org/T405478 [15:48:56] FIRING: SystemdUnitDown: The service unit maintain-dbusers.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [15:51:24] (03update) 10aradhya: Proof of concept for adding new statements( T406989) [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/80 [15:58:17] RESOLVED: JobUnavailable: Reduced availability for job maintain_dbusers_eqiad in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [15:58:56] RESOLVED: SystemdUnitDown: The service unit maintain-dbusers.service is in failed status on host cloudcontrol1007. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1007 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [16:05:45] 10Tool-paulina: Proof of concept for adding new statements - https://phabricator.wikimedia.org/T406989#11294507 (10Aradhya05) Hi @Pepe_piton, Thank you so much for your feedback! I have fixed these issues and added to the same merge request: https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/... [16:23:15] (03open) 10olatundeawo: Add author form t407551 [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/133 [16:26:25] (03update) 10damian: _do_run - retry runtime errors [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/139 (https://phabricator.wikimedia.org/T403175) [16:26:47] (03update) 10damian: _do_run - retry runtime errors [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/139 (https://phabricator.wikimedia.org/T403175) [16:29:19] 10Tool-paulina: Proof of concept: Add author form - https://phabricator.wikimedia.org/T407551#11294733 (10Olatundeawo) Hello @Pepe_piton @Nat_WDU I worked on creating a proof-of-concept “Add Author” form for the Paulina tool. Structure. The new page (add-author.html) extends layout.html and includes fields th... [16:32:55] (03update) 10dcaro: _do_run - retry runtime errors [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/139 (https://phabricator.wikimedia.org/T403175) (owner: 10damian) [16:35:21] (03update) 10bahago: Feature/add author form [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/127 [16:37:34] (03open) 10damian: api - add job_changed to update response [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/228 (https://phabricator.wikimedia.org/T403321) [16:37:42] (03close) 10damian: allow restarting job during patch [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/221 (https://phabricator.wikimedia.org/T403321) [17:05:09] (03update) 10damian: offload job restart logic to jobs-api [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/133 (https://phabricator.wikimedia.org/T403321) [17:07:12] (03open) 10olatundeawo: Add work form autocomplete t407558 [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/134 [17:07:21] PROBLEM - Wikitech-static main page has content on wikitech-static.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Wikitech-static [17:08:11] RECOVERY - Wikitech-static main page has content on wikitech-static.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 30029 bytes in 0.235 second response time https://wikitech.wikimedia.org/wiki/Wikitech-static [17:11:02] (03update) 10damian: offload job restart logic to jobs-api [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/133 (https://phabricator.wikimedia.org/T403321) [17:17:41] 10Tool-paulina: Proof of concept: add work form, with autocomplete - https://phabricator.wikimedia.org/T407558#11295020 (10Olatundeawo) I’ve submitted a Merge Request for the implementation of the Add Work page. This feature introduces a simple, responsive form for creating new Work items in Paulina. What I Im... [17:21:18] (03open) 10aradhya: hide empty author fields (T406909) [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/135 [17:24:19] 10Tool-paulina: Hide author data fields when they have no value - https://phabricator.wikimedia.org/T406909#11295082 (10Aradhya05) Hi @Nat_WDU @Pepe_piton , i have completed this task and submitted merge request : https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/135 Kindly review! Thank you [17:24:48] (03update) 10aradhya: hide empty author fields (T406909) [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/135 [17:40:56] FIRING: SystemdUnitDown: The service unit drain_rabbitmq_notification_error.service is in failed status on host cloudrabbit1001. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudrabbit1001 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [18:00:56] RESOLVED: SystemdUnitDown: The service unit drain_rabbitmq_notification_error.service is in failed status on host cloudrabbit1001. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudrabbit1001 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [18:25:13] (03open) 10damian: images - expand name parsing to support digests [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/229 [18:29:06] (03update) 10damian: images - expand name parsing to support digests [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/229 (https://phabricator.wikimedia.org/T403322) [18:29:14] (03update) 10damian: images - expand name parsing to support digests [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/229 (https://phabricator.wikimedia.org/T403322) [18:30:50] 10VPS-project-Phabricator, 06collaboration-services: Test Phabricator is down - https://phabricator.wikimedia.org/T407891 (10A_smart_kitten) 03NEW [18:34:25] PROBLEM - Wikitech-static main page has content on wikitech-static.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Wikitech-static [18:35:21] RECOVERY - Wikitech-static main page has content on wikitech-static.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 30039 bytes in 4.753 second response time https://wikitech.wikimedia.org/wiki/Wikitech-static [18:49:27] (03open) 10dipanshu1223: T405621:Country dropdown [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/136 [18:52:57] 10Tool-paulina: Improve country dropdown for the "country of origin" filter in the advanced search for works. - https://phabricator.wikimedia.org/T405621#11295559 (10Dipanshu1223) Good evening @Pepe_piton @Nat_WDU @DidiCoronel i made a merge request related this issue:https://gitlab.wikimedia.org/toolforge-repos... [19:03:25] PROBLEM - Wikitech-static main page has content on wikitech-static.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Wikitech-static [19:07:17] RECOVERY - Wikitech-static main page has content on wikitech-static.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 30040 bytes in 0.931 second response time https://wikitech.wikimedia.org/wiki/Wikitech-static [20:10:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance metricsinfra-thanos-fe-2 on project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [20:12:08] 10VPS-project-Phabricator, 06collaboration-services: Test Phabricator is down - https://phabricator.wikimedia.org/T407891#11295907 (10Dzahn) Well, it has been used to test a change. This is likely because https://gerrit.wikimedia.org/r/c/operations/puppet/+/1192636 was deployed there. [20:22:15] 10VPS-project-Phabricator, 06collaboration-services: Test Phabricator is down - https://phabricator.wikimedia.org/T407891#11296003 (10Dzahn) ` /srv/deployment/phabricator/deployment-cache/revs/16c9739f449313e18a6d74e5c88eb517c644aec8/phabricator/conf/local/local.json: empty ` [20:24:00] 10VPS-project-Phabricator, 06collaboration-services: Test Phabricator is down - https://phabricator.wikimedia.org/T407891#11296020 (10A_smart_kitten) >>! In T407891#11295907, @Dzahn wrote: > Well, it has been used to test a change. Fair enough! :) Thought it was probably best to report in case it was unexpect... [20:43:23] (03open) 10adwivedii: T407551: Add author form [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/137 [20:49:15] 10VPS-project-Phabricator, 06collaboration-services: Test Phabricator is down - https://phabricator.wikimedia.org/T407891#11296110 (10Dzahn) https://phabricator.wmcloud.org/ is up again reverted to the previous commit on the local puppetmaster.. ran puppet on deployment server.. deployed from deployment serv... [20:56:55] 06cloud-services-team, 06SRE: latest Trixie image (as of 2025-10-16) grub failure - https://phabricator.wikimedia.org/T407586#11296123 (10Andrew) ` root@cloudcontrol2010-dev:~# facter efi false ` [20:59:47] 10Tool-paulina: Proof of concept: Add author form - https://phabricator.wikimedia.org/T407551#11296140 (10Adwivedii) Hello @Pepe_piton @Nat_WDU, I have created my version of this task, which can be seen at this link: [[ https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/137 ]] Features Impl... [20:59:53] 10VPS-project-Phabricator, 06collaboration-services: Test Phabricator is down - https://phabricator.wikimedia.org/T407891#11296141 (10Dzahn) @A_smart_kitten Hello! No, you don't have to do anything different in the future. I think this is just a good example of what I tried to articulate in the past. Technic... [21:00:29] 10VPS-project-Phabricator, 06collaboration-services: Test Phabricator is down - https://phabricator.wikimedia.org/T407891#11296143 (10Dzahn) 05Open→03Resolved a:03Dzahn [21:10:36] (03update) 10adwivedii: T407551: Add author form [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/137 [21:16:49] (03update) 10damian: images - expand name parsing to support digests [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/229 (https://phabricator.wikimedia.org/T403322) [21:30:02] 06cloud-services-team, 06SRE: latest Trixie image (as of 2025-10-16) grub failure - https://phabricator.wikimedia.org/T407586#11296297 (10Andrew) This is on the console while the debian isntaller says it's installing grub. ` Oct 21 21:23:11 debconf: <-- 0 Install the GRUB boot loader Oct... [21:34:28] 06Toolforge-standards-committee: Define a process for keeping the committee membership "fresh" - https://phabricator.wikimedia.org/T379844#11296340 (10bd808) @JJMC89 has called for a vote on the proposed wording the mailing list. The vote will end 31 October 2025 AoE. [21:34:58] 06Toolforge-standards-committee: Define a process for keeping the committee membership "fresh" - https://phabricator.wikimedia.org/T379844#11296341 (10bd808) [21:36:15] (03update) 10adwivedii: T407551: Add author form [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/137 [22:24:49] (03update) 10miiswom: Draft: Proof of concept: Add author form [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/131 [22:34:09] (03update) 10miiswom: Draft: Proof of concept: Add author form [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/131 [23:18:50] (03open) 10damian: toolforge components - add logs-api [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/286 (https://phabricator.wikimedia.org/T407930) [23:21:41] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: [api-gateway] openapi.json broken - https://phabricator.wikimedia.org/T407930#11296628 (10DamianZaremba) [23:22:03] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: [api-gateway] openapi.json broken - https://phabricator.wikimedia.org/T407930#11296632 (10DamianZaremba) Tagging @dcaro from https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/commit/ce2e7a2102d2bf4e5ba3bb66175689ef047476f9 [23:25:38] (03update) 10damian: toolforge components - add logs-api [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/286 (https://phabricator.wikimedia.org/T407930) [23:36:15] 06cloud-services-team, 10Toolforge: [jobs-api] local config broken? - https://phabricator.wikimedia.org/T407931 (10DamianZaremba) 03NEW [23:47:13] 06cloud-services-team, 10Toolforge: [jobs-api] local config broken? - https://phabricator.wikimedia.org/T407931#11296669 (10DamianZaremba) ` - containerID: containerd://894c560e807240592442af983fcdab6cd63736e528f28431b83e11dd8505d8c3 image: toolsbeta-harbor.wmcloud.org/toolforge/jobs-api:image-0.0.425-20... [23:48:09] !log filippo@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99) (T405478) [23:48:17] T405478: Experiment with cloudcephosd1050 and cloudcephosd1051 in single-nic configuration - https://phabricator.wikimedia.org/T405478 [23:48:36] 06cloud-services-team, 10Toolforge: [jobs-api] local config broken? - https://phabricator.wikimedia.org/T407931#11296673 (10DamianZaremba) File is mounted into the container ` damian@lima-kilo:~/jobs-api$ docker inspect toolforge-control-plane | jq .[0].HostConfig.Binds [ "/etc/ldap.conf:/etc/ldap.conf:rshar... [23:52:27] (03open) 10sopzy: T407551-proof-of-concept-add-author-form [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/138 [23:58:59] (03update) 10sopzy: T407551-proof-of-concept-add-author-form [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/138