[00:04:54] 10Tool-gawa: [Code] Conception de la page STATISTIQUES - https://phabricator.wikimedia.org/T401767#11223042 (10poro26) [00:05:26] 10Tool-gawa: [Code] Conception de la page EVENEMENTS - https://phabricator.wikimedia.org/T403584#11223044 (10poro26) [00:05:58] 10Tool-gawa: [Code] Conception de la page RESULTATS - https://phabricator.wikimedia.org/T403590#11223046 (10poro26) [00:06:38] 10Tool-gawa: [Code] Conception de la page ERREUR 404 - https://phabricator.wikimedia.org/T403592#11223048 (10poro26) [00:44:55] FIRING: PawsJupyterHubDown: PAWS JupyterHub is down https://wikitech.wikimedia.org/wiki/PAWS/Admin - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPawsJupyterHubDown [00:45:28] FIRING: TargetDown: Job jupyterhub is unreachable in project paws instance hub-paws.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [00:49:55] RESOLVED: PawsJupyterHubDown: PAWS JupyterHub is down https://wikitech.wikimedia.org/wiki/PAWS/Admin - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPawsJupyterHubDown [00:50:28] RESOLVED: TargetDown: Job jupyterhub is unreachable in project paws instance hub-paws.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [01:16:43] 10Tool-documentation, 06Wiki-Mentor-Africa, 05Goal: Review at least 100 Toolforge tools each month. - https://phabricator.wikimedia.org/T363664#11223065 (10komla) Revisiting this. Will transfer work done to wiki page. [01:17:24] 10Tool-documentation, 06Wiki-Mentor-Africa, 05Goal: Review at least 100 Toolforge tools each month. - https://phabricator.wikimedia.org/T363664#11223066 (10komla) [01:20:34] 10Cloud-Services: Cloud VPS requests review - https://phabricator.wikimedia.org/T405864 (10komla) 03NEW The #Cloud-Services project tag is not intended to have any tasks. Please check the list on https://phabricator.wikimedia.org/project/profile/832/ and replace it with a more specific project tag to this task... [02:42:45] 06cloud-services-team (FY2025/26-Q1), 10Toolforge (Toolforge iteration 24), 07Epic: [KR] WE6.3 Introduce a sustainability scoring system for the Toolforge platform - https://phabricator.wikimedia.org/T368600#11223087 (10komla) Emails out to admins [02:51:48] 10Tool-link-dispenser: Not finished after 5 hours running - https://phabricator.wikimedia.org/T402178#11223094 (10Soda) @Chidgk1, This should be fixed now? [03:00:47] 10Tool-link-dispenser: Not finished after 5 hours running - https://phabricator.wikimedia.org/T402178#11223099 (10Soda) Also noting that if you end up in that kind of a situation, a reload (or even a retry with `?nocache=yes` at the end of the URL) should submit a fresh try and fix it [05:34:24] 06cloud-services-team, 10Toolforge: Dotnet bots failing with no logs - https://phabricator.wikimedia.org/T403927#11223152 (10Hawkeye7) Part of the problem is the `Output log: logs/conflicts.stdout.log ` This does not work, and likely causes the job to error. [07:18:28] FIRING: [4x] PuppetAgentFailure: Puppet agent failure detected on instance tools-k8s-worker-nfs-19 in project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [07:23:28] FIRING: [4x] PuppetAgentFailure: Puppet agent failure detected on instance tools-k8s-worker-nfs-19 in project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [07:33:28] RESOLVED: [4x] PuppetAgentFailure: Puppet agent failure detected on instance tools-k8s-worker-nfs-19 in project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [07:35:21] (03update) 10taavi: shared: Manage Kubernetes HAProxy groups [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/84 [07:35:21] (03update) 10taavi: shared: Allocate a VIP for HAProxy Keepalived usage [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/86 (https://phabricator.wikimedia.org/T405078) [07:35:30] (03update) 10taavi: shared: Allocate a VIP for HAProxy Keepalived usage [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/86 (https://phabricator.wikimedia.org/T405078) [07:35:37] (03update) 10taavi: shared: Manage Kubernetes HAProxy groups [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/84 [08:06:53] (03PS1) 10David Caro: reboot_stuck_workers: allow specifying the max D proc limit [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1192052 [08:10:23] (03CR) 10CI reject: [V:04-1] reboot_stuck_workers: allow specifying the max D proc limit [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1192052 (owner: 10David Caro) [08:20:15] 06cloud-services-team, 10Toolforge: Dotnet bots failing with no logs - https://phabricator.wikimedia.org/T403927#11223452 (10dcaro) Hmm, given the command you wrote there, it should not configure the filelogs at all, testing in one of my tools it does not: ` tools.wm-lol@tools-bastion-15:~$ toolforge jobs run... [08:20:46] 06cloud-services-team, 10Toolforge: Dotnet bots failing with no logs - https://phabricator.wikimedia.org/T403927#11223454 (10dcaro) p:05Triage→03Medium [08:28:56] 06cloud-services-team, 10Toolforge: Build standard images under pack / support execution via components-api - https://phabricator.wikimedia.org/T405262#11223516 (10taavi) On a first glance this is a duplicate of {T362076}, although that claims this is already implemented but I don't see any mention of that on... [08:29:11] 06cloud-services-team, 10Toolforge: Support pre-built images on components-api - https://phabricator.wikimedia.org/T405262#11223518 (10taavi) [08:30:12] 06cloud-services-team, 10Toolforge: Support pre-built images on components-api - https://phabricator.wikimedia.org/T405262#11223533 (10dcaro) >>! In T405262#11223516, @taavi wrote: > On a first glance this is a duplicate of {T362076}, although that claims this is already implemented but I don't see any mention... [08:34:19] 06cloud-services-team, 10Toolforge: Support pre-built images on components-api - https://phabricator.wikimedia.org/T405262#11223562 (10DamianZaremba) For now I have https://github.com/cluebotng/external-utilities which basically covers `mariadb` (`curl` actually is done in the `report` container now, but was i... [08:41:04] 06cloud-services-team, 10Toolforge: [builds-api] does not correctly resolve `ref` - builds random things - https://phabricator.wikimedia.org/T405829#11223624 (10DamianZaremba) It appears this does not happen when using components-api, as it explicitly resolves the ref before passing into builds-api. So this is... [08:44:54] 06cloud-services-team (FY2025/26-Q1), 10Toolforge (Toolforge iteration 24): 2025-09-28 ToolforgeWebHighErrorRate: High 5xx rate on Toolforge web services - https://phabricator.wikimedia.org/T405850#11223631 (10dcaro) There were a bunch of tools that started giving 500s right before the first bump of denied res... [08:47:48] 06cloud-services-team (FY2025/26-Q1), 10Toolforge (Toolforge iteration 24): 2025-09-28 ToolforgeWebHighErrorRate: High 5xx rate on Toolforge web services - https://phabricator.wikimedia.org/T405850#11223669 (10dcaro) When stacking the processes in D state graph, there's a correlation for the sunday morning bum... [09:11:19] supertassu closed https://github.com/toolforge/paws/pull/501 [09:14:43] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18091898160 (https://github.com/cluebotng/component-configs/commits/ff3951fa5af87196929a9a864f8189b7a7436ac8) [09:14:46] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [09:15:43] 06cloud-services-team, 10PAWS, 10OpenRefine: New upstream release for OpenRefine - https://phabricator.wikimedia.org/T388928#11223814 (10taavi) 05Open→03Resolved a:03taavi [09:15:46] 06cloud-services-team, 10PAWS, 06Commons, 10OpenRefine: New upstream release for Wikimedia Commons Extension for OpenRefine - https://phabricator.wikimedia.org/T403780#11223817 (10taavi) 05Open→03Resolved a:03taavi [09:15:49] 06cloud-services-team, 10PAWS: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T405170#11223819 (10taavi) 05Open→03Resolved a:03taavi [09:16:44] supertassu opened https://github.com/toolforge/paws/pull/502 [09:18:05] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18091994310 (https://github.com/cluebotng/component-configs/commits/ff3951fa5af87196929a9a864f8189b7a7436ac8) [09:18:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [09:21:43] !log tools.cluebotng-staging Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18092072445 (https://github.com/cluebotng/component-configs/commits/a0d50b624a6cdfa221225a08b11c52ed85e54d0c) [09:21:45] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-staging/SAL [09:23:14] supertassu closed https://github.com/toolforge/paws/pull/502 [09:33:21] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18092350259 (https://github.com/cluebotng/component-configs/commits/283965c9240c0c5a72e0ea1203439583935295cb) [09:33:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [10:02:29] (03update) 10dcaro: [build, api] support build queueing beyond max_parallel build config [repos/cloud/toolforge/builds-api] (run_pipeline_cleanup_per_repo) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/143 (https://phabricator.wikimedia.org/T402568) (owner: 10raymond-ndibe) [10:06:32] (03update) 10dcaro: [deploy_task, tool_handlers] queue deployments to allow creation of multiple deployments at once [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/131 (https://phabricator.wikimedia.org/T402568) (owner: 10raymond-ndibe) [10:24:30] (03Abandoned) 10David Caro: reboot_stuck_workers: allow specifying the max D proc limit [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1192052 (owner: 10David Caro) [10:30:42] 06cloud-services-team: Cloud VPS requests review - https://phabricator.wikimedia.org/T405864#11224045 (10Aklapper) [10:32:44] (03open) 10dcaro: worker_stuck: use the new metric for stuck workers [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/41 [10:34:46] !log dcaro@acme tools START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-79 [10:34:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [10:34:53] !log dcaro@acme tools END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=99) for tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-79 [10:34:55] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [10:35:10] !log dcaro@acme tools START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-79 [10:35:12] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [10:35:16] !log dcaro@acme tools END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=99) for tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-79 [10:35:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [10:35:33] !log dcaro@acme tools START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-79 [10:35:36] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [10:35:38] !log dcaro@acme tools END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=99) for tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-79 [10:35:41] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [10:35:42] !log dcaro@acme toolsbeta START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers no stuck workers found [10:35:44] !log dcaro@acme toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) no stuck workers found [10:35:45] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [10:35:47] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [10:36:19] (03PS1) 10David Caro: reboot_stuck_workers: use the new metric [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1192090 [10:37:28] FIRING: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance toolsbeta-puppetserver-1 in project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [10:39:49] !log dcaro@acme tools START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-79 [10:39:52] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [10:40:46] (03CR) 10CI reject: [V:04-1] reboot_stuck_workers: use the new metric [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1192090 (owner: 10David Caro) [10:44:15] (03CR) 10David Caro: "Tested locally, toolsbeta gets nothing:" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1192090 (owner: 10David Caro) [10:45:04] (03PS2) 10David Caro: reboot_stuck_workers: use the new metric [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1192090 [10:46:17] (03approved) 10fnegri: worker_stuck: use the new metric for stuck workers [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/41 (owner: 10dcaro) [10:48:21] (03CR) 10FNegri: global: minor cleanups (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1186483 (owner: 10David Caro) [10:48:48] (03merge) 10dcaro: worker_stuck: use the new metric for stuck workers [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/41 [10:59:04] 06cloud-services-team, 10Toolforge (Toolforge iteration 24): [builds-api] does not correctly resolve `ref` - builds random things - https://phabricator.wikimedia.org/T405829#11224131 (10dcaro) p:05Triage→03Low a:03dcaro [11:04:58] RESOLVED: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance toolsbeta-puppetserver-1 in project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [11:05:06] !log dcaro@acme tools END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-79 [11:05:10] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [11:07:10] (03open) 10dcaro: build: return error when the given ref is not resolvable [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/145 [11:44:28] FIRING: PuppetAgentFailure: Puppet agent failure detected on instance tools-k8s-worker-nfs-50 in project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [11:54:28] FIRING: [3x] PuppetAgentFailure: Puppet agent failure detected on instance tools-k8s-worker-nfs-50 in project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [11:57:50] 10Tool-link-dispenser: Not finished after 5 hours running - https://phabricator.wikimedia.org/T402178#11224362 (10Chidgk1) I am trying on my iphone and it may be hung or looping on 74/94 urls processed. I don’t have a non-Apple device to try it on. When you followed the steps to replicate above did it run succes... [12:07:10] 10Tool-link-dispenser: Not finished after 5 hours running - https://phabricator.wikimedia.org/T402178#11224396 (10Chidgk1) I retried after turning my iphone off and on again and turning on the VPN but that made no difference [12:32:00] (03open) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/ranker] - 10https://gitlab.wikimedia.org/toolforge-repos/ranker/-/merge_requests/26 [12:32:01] (03open) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/45 [12:32:01] (03open) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/lexeme-forms] - 10https://gitlab.wikimedia.org/toolforge-repos/lexeme-forms/-/merge_requests/16 [12:33:40] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [labs/tools/massmailer] - 10https://gerrit.wikimedia.org/r/1192116 (owner: 10L10n-bot) [12:39:28] FIRING: [3x] PuppetAgentFailure: Puppet agent failure detected on instance tools-k8s-worker-nfs-50 in project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [12:44:19] (03PS1) 10Btullis: Add new dummy keytabs for an-launcher1003 [labs/private] - 10https://gerrit.wikimedia.org/r/1192120 (https://phabricator.wikimedia.org/T402943) [12:45:04] (03CR) 10Btullis: [V:03+2 C:03+2] Add new dummy keytabs for an-launcher1003 [labs/private] - 10https://gerrit.wikimedia.org/r/1192120 (https://phabricator.wikimedia.org/T402943) (owner: 10Btullis) [12:49:28] RESOLVED: [3x] PuppetAgentFailure: Puppet agent failure detected on instance tools-k8s-worker-nfs-50 in project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [13:13:00] (03CR) 10FNegri: [C:03+1] reboot_stuck_workers: use the new metric [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1192090 (owner: 10David Caro) [13:14:53] (03PS8) 10David Caro: global: minor cleanups [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1186483 [13:15:02] (03CR) 10David Caro: global: minor cleanups (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1186483 (owner: 10David Caro) [13:21:26] (03CR) 10David Caro: [C:03+2] global: minor cleanups [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1186483 (owner: 10David Caro) [13:22:40] (03CR) 10FNegri: [C:03+1] wmcs_libs: k8s: Support tofu-managed groups for HAProxy (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1191307 (https://phabricator.wikimedia.org/T405078) (owner: 10Majavah) [13:23:27] !log dcaro@acme tools START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers no stuck workers found [13:23:30] !log dcaro@acme tools END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) no stuck workers found [13:23:31] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [13:23:34] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [13:25:56] (03Merged) 10jenkins-bot: global: minor cleanups [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1186483 (owner: 10David Caro) [13:28:23] (03CR) 10Majavah: wmcs_libs: k8s: Support tofu-managed groups for HAProxy (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1191307 (https://phabricator.wikimedia.org/T405078) (owner: 10Majavah) [13:29:38] (03CR) 10David Caro: wmcs_libs: k8s: Support tofu-managed groups for HAProxy (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1191307 (https://phabricator.wikimedia.org/T405078) (owner: 10Majavah) [13:31:35] (03PS2) 10Majavah: wmcs_libs: k8s: Support tofu-managed groups for HAProxy [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1191307 (https://phabricator.wikimedia.org/T405078) [13:31:35] (03PS3) 10Majavah: toolforge: k8s: Resolve K8s HAProxy VIPs from Hiera [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1191308 (https://phabricator.wikimedia.org/T405078) [13:31:35] (03PS3) 10Majavah: wmcs_libs: k8s: Fix is_worker logic [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1191332 [13:31:46] (03CR) 10FNegri: wmcs_libs: k8s: Support tofu-managed groups for HAProxy (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1191307 (https://phabricator.wikimedia.org/T405078) (owner: 10Majavah) [13:31:59] (03CR) 10CI reject: [V:04-1] wmcs_libs: k8s: Support tofu-managed groups for HAProxy [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1191307 (https://phabricator.wikimedia.org/T405078) (owner: 10Majavah) [13:32:58] (03PS4) 10Majavah: toolforge: k8s: Resolve K8s HAProxy VIPs from Hiera [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1191308 (https://phabricator.wikimedia.org/T405078) [13:32:58] (03PS3) 10Majavah: wmcs_libs: k8s: Support tofu-managed groups for HAProxy [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1191307 (https://phabricator.wikimedia.org/T405078) [13:32:58] (03PS4) 10Majavah: wmcs_libs: k8s: Fix is_worker logic [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1191332 [13:38:23] (03CR) 10Majavah: [C:03+2] toolforge: k8s: Resolve K8s HAProxy VIPs from Hiera [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1191308 (https://phabricator.wikimedia.org/T405078) (owner: 10Majavah) [13:42:20] (03Merged) 10jenkins-bot: toolforge: k8s: Resolve K8s HAProxy VIPs from Hiera [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1191308 (https://phabricator.wikimedia.org/T405078) (owner: 10Majavah) [13:44:03] (03update) 10taavi: shared: Allocate a VIP for HAProxy Keepalived usage [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/86 (https://phabricator.wikimedia.org/T405078) [13:44:13] (03update) 10taavi: shared: Manage Kubernetes HAProxy groups [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/84 [13:46:51] (03update) 10taavi: shared: Allocate a VIP for HAProxy Keepalived usage [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/86 (https://phabricator.wikimedia.org/T405078) [13:46:56] (03update) 10taavi: shared: Manage Kubernetes HAProxy groups [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/84 [13:48:14] (03PS4) 10Majavah: wmcs_libs: k8s: Support tofu-managed groups for HAProxy [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1191307 (https://phabricator.wikimedia.org/T405078) [13:48:14] (03PS5) 10Majavah: wmcs_libs: k8s: Fix is_worker logic [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1191332 [13:51:58] (03CR) 10Majavah: wmcs_libs: k8s: Support tofu-managed groups for HAProxy (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1191307 (https://phabricator.wikimedia.org/T405078) (owner: 10Majavah) [13:53:16] (03update) 10taavi: shared: Manage Kubernetes HAProxy groups [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/84 [14:03:49] (03update) 10dcaro: build: return error when the given ref is not resolvable [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/145 [14:04:17] 06cloud-services-team, 10Toolforge (Toolforge iteration 24): [builds-api] does not correctly resolve `ref` - builds random things - https://phabricator.wikimedia.org/T405829#11224735 (10dcaro) 05Open→03In progress [14:06:11] 06cloud-services-team (FY2025/26-Q1), 10Toolforge (Toolforge iteration 24): [infra,haproxy,ingress] 2025-09-23 Ingress hitting the backend session limit and started replying with 5xxs - https://phabricator.wikimedia.org/T405280#11224759 (10dcaro) 05In progress→03Resolved [14:06:18] 10Toolforge (Toolforge iteration 24): [jobs-api] loki logs take really long to appear - https://phabricator.wikimedia.org/T404176#11224762 (10dcaro) 05In progress→03Resolved [14:12:20] 06cloud-services-team, 10Toolforge: toolforge logs appears to suffer from intermittent latency - https://phabricator.wikimedia.org/T402736#11224798 (10dcaro) I think that might be the rate limiting happening, as alloy did pick up the logfile, but it seems the logs did not reach the storage, maybe we have to lo... [14:14:08] 10Tool-gawa: [Code] Ajustement de l’affichage pour écrans mobiles - https://phabricator.wikimedia.org/T405863#11224803 (10poro26) [14:14:56] 06cloud-services-team, 10Toolforge: toolforge logs appears to suffer from intermittent latency - https://phabricator.wikimedia.org/T402736#11224805 (10dcaro) As of right now, running this returns the first 100 logs, then one extra for some reason: ` tools.wm-lol@tools-bastion-15:~$ cat test.sh #!/bin/bash fo... [14:15:48] (03PS3) 10David Caro: reboot_stuck_workers: use the new metric [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1192090 [14:16:51] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18099932067 (https://github.com/cluebotng/component-configs/commits/0de901e1203dd61656503ef2127efe360e9ed6cc) [14:16:54] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [14:21:39] (03CR) 10David Caro: [C:03+2] reboot_stuck_workers: use the new metric [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1192090 (owner: 10David Caro) [14:24:50] (03CR) 10FNegri: [C:03+1] wmcs_libs: k8s: Support tofu-managed groups for HAProxy [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1191307 (https://phabricator.wikimedia.org/T405078) (owner: 10Majavah) [14:25:37] (03approved) 10fnegri: shared: Manage Kubernetes HAProxy groups [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/84 (owner: 10taavi) [14:25:49] (03Merged) 10jenkins-bot: reboot_stuck_workers: use the new metric [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1192090 (owner: 10David Caro) [14:26:09] 06cloud-services-team, 10Toolforge: toolforge logs appears to suffer from intermittent latency - https://phabricator.wikimedia.org/T402736#11224868 (10dcaro) I can reproduce on lima-kilo too, pointing too to the limit being reached instead of being something deeployment-related (cluster load/network/etc.). [14:27:56] (03CR) 10Majavah: [C:03+2] wmcs_libs: k8s: Support tofu-managed groups for HAProxy [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1191307 (https://phabricator.wikimedia.org/T405078) (owner: 10Majavah) [14:28:06] (03merge) 10taavi: shared: Manage Kubernetes HAProxy groups [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/84 [14:28:10] (03update) 10taavi: shared: Allocate a VIP for HAProxy Keepalived usage [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/86 (https://phabricator.wikimedia.org/T405078) [14:29:24] 06cloud-services-team, 10Toolforge (Toolforge iteration 24): toolforge logs appears to suffer from intermittent latency - https://phabricator.wikimedia.org/T402736#11224879 (10dcaro) a:03dcaro [14:32:07] (03Merged) 10jenkins-bot: wmcs_libs: k8s: Support tofu-managed groups for HAProxy [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1191307 (https://phabricator.wikimedia.org/T405078) (owner: 10Majavah) [14:37:45] (03update) 10taavi: shared: Allocate a VIP for HAProxy Keepalived usage [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/86 (https://phabricator.wikimedia.org/T405078) [14:38:52] 06cloud-services-team, 10Toolforge (Toolforge iteration 24): toolforge logs appears to suffer from intermittent latency - https://phabricator.wikimedia.org/T402736#11224898 (10dcaro) From alloy logs for one of the runs: ` │ ts=2025-09-29T14:35:17.935750637Z level=info msg="tail routine: started" component_path... [14:52:04] 10Cloud-VPS (Debian Bullseye Deprecation), 10CFSSL-PKI, 06Infrastructure-Foundations: Rebuild VMs in PKI cloud-vps project - https://phabricator.wikimedia.org/T405017#11224973 (10elukey) p:05Triage→03Medium [15:05:48] !log tools.cluebotng-trainer Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18101460176 (https://github.com/cluebotng/component-configs/commits/f43490cf3ca4913763b07a84c7ac0aa4281e96b4) [15:05:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-trainer/SAL [15:06:03] !log tools.cluebotng-staging Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18101460177 (https://github.com/cluebotng/component-configs/commits/f43490cf3ca4913763b07a84c7ac0aa4281e96b4) [15:06:04] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-staging/SAL [15:21:42] !log tools.cluebotng-trainer Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18101913411 (https://github.com/cluebotng/component-configs/commits/1b6389ae74b8974d6f49591c0abf14a8da974c4b) [15:21:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-trainer/SAL [15:45:53] (03CR) 10FNegri: [C:03+1] wmcs_libs: k8s: Fix is_worker logic [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1191332 (owner: 10Majavah) [15:49:30] (03approved) 10fnegri: shared: Allocate a VIP for HAProxy Keepalived usage [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/86 (https://phabricator.wikimedia.org/T405078) (owner: 10taavi) [15:50:37] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18102721922 (https://github.com/cluebotng/component-configs/commits/87ddcf2fce928fde2ba91ecdba3561b12b8de1d2) [15:50:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [15:58:49] 06cloud-services-team, 10Toolforge: [builds-api] Gitlab maintenance should not cause an outage for builds - https://phabricator.wikimedia.org/T405782#11225330 (10dcaro) p:05Triage→03Low [16:24:20] (03CR) 10Majavah: [C:03+2] wmcs_libs: k8s: Fix is_worker logic [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1191332 (owner: 10Majavah) [16:26:26] 10Toolforge (Quota-requests): Request increased build quota for cluebotng Toolforge tool - https://phabricator.wikimedia.org/T405645#11225575 (10DamianZaremba) Again today ` jobs/uploads/303eeb20-9905-43ad-a95a-a7d76cf67b22?_state=REDACTED&digest=sha256%3A2f21c1501782934e1b9df9e9828d2ff48a3c62949b43025229c601411... [16:28:44] (03Merged) 10jenkins-bot: wmcs_libs: k8s: Fix is_worker logic [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1191332 (owner: 10Majavah) [16:40:08] !log tools.cluebotng-trainer Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18104101421 (https://github.com/cluebotng/component-configs/commits/c49408a6e0285932adef0b5cc39e15d06c8742f5) [16:40:11] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-trainer/SAL [16:41:11] !log tools.cluebotng-staging Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18104101416 (https://github.com/cluebotng/component-configs/commits/c49408a6e0285932adef0b5cc39e15d06c8742f5) [16:41:12] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-staging/SAL [16:41:20] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18104101448 (https://github.com/cluebotng/component-configs/commits/c49408a6e0285932adef0b5cc39e15d06c8742f5) [16:41:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [16:41:26] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18104101417 (https://github.com/cluebotng/component-configs/commits/c49408a6e0285932adef0b5cc39e15d06c8742f5) [16:41:27] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [16:46:24] (03merge) 10taavi: shared: Allocate a VIP for HAProxy Keepalived usage [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/86 (https://phabricator.wikimedia.org/T405078) [16:48:45] 10VPS-project-Codesearch, 06collaboration-services: Graduate codesearch to production - https://phabricator.wikimedia.org/T268199#11225729 (10Dzahn) 05Open→03Stalled [18:20:34] (03approved) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/ranker] - 10https://gitlab.wikimedia.org/toolforge-repos/ranker/-/merge_requests/26 (owner: 10l10n-bot) [18:20:36] (03merge) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/ranker] - 10https://gitlab.wikimedia.org/toolforge-repos/ranker/-/merge_requests/26 (owner: 10l10n-bot) [18:21:50] (03approved) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/45 (owner: 10l10n-bot) [18:21:53] (03merge) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/45 (owner: 10l10n-bot) [18:22:39] (03approved) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/lexeme-forms] - 10https://gitlab.wikimedia.org/toolforge-repos/lexeme-forms/-/merge_requests/16 (owner: 10l10n-bot) [18:22:41] (03merge) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/lexeme-forms] - 10https://gitlab.wikimedia.org/toolforge-repos/lexeme-forms/-/merge_requests/16 (owner: 10l10n-bot) [18:46:49] 06cloud-services-team, 10Striker, 10CAS-SSO, 13Patch-For-Review: Use IDP for authentication in Striker - https://phabricator.wikimedia.org/T359554#11226188 (10Arendpieter) @SLyngshede-WMF, do you know how to create OIDC_KEY and OIDC_SECRET in idp.wikimedia.org in order to test this solution: https://gerrit... [18:47:10] (03PS1) 10Dzahn: move zuul nodepool to new location for I745f8c87b4c57f [labs/private] - 10https://gerrit.wikimedia.org/r/1192200 (https://phabricator.wikimedia.org/T395938) [18:48:02] (03PS2) 10Dzahn: move zuul nodepool to new location for I745f8c87b4c57f [labs/private] - 10https://gerrit.wikimedia.org/r/1192200 (https://phabricator.wikimedia.org/T395938) [18:48:08] (03CR) 10Dzahn: [C:03+2] move zuul nodepool to new location for I745f8c87b4c57f [labs/private] - 10https://gerrit.wikimedia.org/r/1192200 (https://phabricator.wikimedia.org/T395938) (owner: 10Dzahn) [18:48:32] (03PS3) 10Dzahn: move zuul nodepool user token to new location for I745f8c87b4c57f [labs/private] - 10https://gerrit.wikimedia.org/r/1192200 (https://phabricator.wikimedia.org/T395938) [18:56:16] (03CR) 10Dzahn: [V:03+2 C:03+2] move zuul nodepool user token to new location for I745f8c87b4c57f [labs/private] - 10https://gerrit.wikimedia.org/r/1192200 (https://phabricator.wikimedia.org/T395938) (owner: 10Dzahn) [19:50:57] 06cloud-services-team, 10Cloud-VPS: unable to "apt install helmfile" on CloudVPS debian 13 vm - https://phabricator.wikimedia.org/T405970 (10SDunlap) 03NEW [22:19:14] 06cloud-services-team, 10Toolforge: Access request for Toolforge - https://phabricator.wikimedia.org/T405984 (10Vincent_Vega) 03NEW [22:25:00] 06cloud-services-team, 10Toolforge: Access request for Toolforge - https://phabricator.wikimedia.org/T405984#11227297 (10JJMC89) 05Open→03Invalid See https://wikitech.wikimedia.org/wiki/Help:Toolforge/Quickstart for how to get access.