[00:35:01] (03update) 10bd808: Draw the rest of the owl [toolforge-repos/etherpad-backup] - 10https://gitlab.wikimedia.org/toolforge-repos/etherpad-backup/-/merge_requests/1 (https://phabricator.wikimedia.org/T422958 https://phabricator.wikimedia.org/T423014) [00:38:26] (03update) 10bd808: Draw the rest of the owl [toolforge-repos/etherpad-backup] - 10https://gitlab.wikimedia.org/toolforge-repos/etherpad-backup/-/merge_requests/1 (https://phabricator.wikimedia.org/T422958 https://phabricator.wikimedia.org/T423014) [02:57:45] (03update) 10davidcoronel: Draft: initial commit for pipeline [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/182 [03:04:49] (03update) 10davidcoronel: Draft: initial commit for pipeline [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/182 [03:08:30] (03update) 10davidcoronel: Draft: initial commit for pipeline [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/182 [03:21:49] (03update) 10davidcoronel: Draft: initial commit for pipeline [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/182 [03:25:15] (03update) 10davidcoronel: Draft: initial commit for pipeline [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/182 [03:27:08] (03update) 10davidcoronel: Testing pipeline [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/182 [03:31:28] 06cloud-services-team, 10Data-Services: [wikireplicas] Create views for new wiki urwikisource - https://phabricator.wikimedia.org/T415977#11849769 (10Pppery) 05Stalled→03Open Wiki was created a long time ago, resetting status [05:52:25] !log tools.cluebotng-staging Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24819177834 (https://github.com/cluebotng/component-configs/commits/50ebb073215919775512bff653c7d337c343c315) [05:52:27] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-staging/SAL [06:38:22] 06cloud-services-team, 10superset.wmcloud.org: Sunset superset.wmcloud.org - https://phabricator.wikimedia.org/T416373#11849914 (10Dev) Thanks @Andrew, that all sounds good. Happy to continue using the same Phabricator tag for coordination. We’ll start by setting up a test/dummy deployment over the next 2-3... [06:48:01] 10VPS-project-Phabricator, 10Bitu, 06collaboration-services, 06Infrastructure-Foundations, 10Phabricator: Allow Bitu to link Phabricator account - https://phabricator.wikimedia.org/T406495#11849932 (10SLyngshede-WMF) 05Open→03Resolved a:03SLyngshede-WMF @Dzahn Yes. We'll figure out if we need t... [06:48:18] 10VPS-project-Phabricator, 10Bitu, 06collaboration-services, 06Infrastructure-Foundations, 10Phabricator: Allow Bitu to link Phabricator account - https://phabricator.wikimedia.org/T406495#11849936 (10SLyngshede-WMF) [07:09:24] 06cloud-services-team, 10Toolforge: Buildservice Rust job won't start - https://phabricator.wikimedia.org/T424176 (10magnusmanske) 03NEW [07:32:48] 06cloud-services-team, 10Toolforge: Buildservice Rust job won't start - https://phabricator.wikimedia.org/T424176#11850075 (10magnusmanske) Update: Now it says "running" on `toolforge jobs list`. I didn't change anything. Might just have been a fluke. [08:09:33] (03update) 10taavi: Undeploy ingress-nginx [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1222 (https://phabricator.wikimedia.org/T392356) [08:14:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [08:19:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [08:57:33] 06cloud-services-team, 10Toolforge: [builds-api] `step-fix-procfile-args-passing` failing - https://phabricator.wikimedia.org/T424191 (10DamianZaremba) 03NEW [09:01:27] 10Cloud-VPS (Project-requests): Request creation of wolf-a Cloud VPS project - https://phabricator.wikimedia.org/T424192 (10Moh777tyjer3c) 03NEW [09:01:32] FIRING: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [09:01:43] FIRING: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [09:06:32] RESOLVED: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [09:06:43] RESOLVED: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [09:07:19] !log tools.cluebotng Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24826656498 (https://github.com/cluebotng/component-configs/commits/90aa39b3e81f19f42d08d5ec6131ba09c68bd786) [09:07:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [09:11:26] !log tools.cluebotng-staging Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/24826851188 (https://github.com/cluebotng/component-configs/commits/22ea2eb955d25f4a15e6b234e72a24bc01127a79) [09:11:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-staging/SAL [09:11:51] (03merge) 10taavi: Undeploy ingress-nginx [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1222 (https://phabricator.wikimedia.org/T392356) [09:12:04] !log tools.cluebotng Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24826859245 (https://github.com/cluebotng/component-configs/commits/22ea2eb955d25f4a15e6b234e72a24bc01127a79) [09:12:04] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [09:16:06] 10Cloud-VPS (Project-requests): Request creation of wolf-a Cloud VPS project - https://phabricator.wikimedia.org/T424192#11850439 (10fgiunchedi) Hello, thank you for your interest in Cloud VPS. From the description it seems the software can be containerized and run on Toolforge, which is the preferred and suppor... [09:17:10] (03open) 10taavi: kubernetes: Remove ingress-nginx specific alerts [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/61 (https://phabricator.wikimedia.org/T392356) [09:17:15] (03update) 10taavi: kubernetes: Remove ingress-nginx specific alerts [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/61 (https://phabricator.wikimedia.org/T392356) [09:18:09] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.remove_k8s_node [09:18:26] 10Cloud-VPS (Project-requests): Request creation of wolf-a Cloud VPS project - https://phabricator.wikimedia.org/T424192#11850447 (10Moh777tyjer3c) Thank you for your review. The project cannot be fully supported on Toolforge because it requires: persistent background services with full system-level control cu... [09:19:21] !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) [09:21:26] (03update) 10taavi: kubernetes: Remove ingress-nginx specific alerts [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/61 (https://phabricator.wikimedia.org/T392356) [09:22:12] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.remove_k8s_node [09:23:00] (03update) 10taavi: kubernetes: Remove ingress-nginx specific alerts [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/61 (https://phabricator.wikimedia.org/T392356) [09:23:16] !log tools.cluebotng Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24827343249 (https://github.com/cluebotng/component-configs/commits/218a7fa56222cf3e98b642eebbf6b1b2b273a92d) [09:23:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [09:23:25] !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) [09:23:38] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.remove_k8s_node [09:24:50] !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) [09:26:25] 06cloud-services-team, 10Toolforge: [builds-api] `step-fix-procfile-args-passing` failing - https://phabricator.wikimedia.org/T424191#11850469 (10DamianZaremba) I've tried this with with a bare executable (script) @ https://github.com/cluebotng/bot/blob/v2.0.4/Procfile rather than using args @ https://github.c... [09:27:03] 06cloud-services-team, 10Toolforge: [builds-api] `step-fix-procfile-args-passing` failing, blocking new builds - https://phabricator.wikimedia.org/T424191#11850470 (10DamianZaremba) [09:32:31] (03close) 10isaranto: Add per-watch event type filtering for project watches [toolforge-repos/phabricator-slack-bot] - 10https://gitlab.wikimedia.org/toolforge-repos/phabricator-slack-bot/-/merge_requests/1 [09:32:32] (03update) 10isaranto: Add per-watch event type filtering for project watches [toolforge-repos/phabricator-slack-bot] - 10https://gitlab.wikimedia.org/toolforge-repos/phabricator-slack-bot/-/merge_requests/1 [09:34:26] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.remove_k8s_node [09:35:50] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) [09:38:07] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.remove_k8s_node [09:39:26] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) [09:39:43] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.remove_k8s_node [09:40:27] !log taavi@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) [09:40:33] (03PS1) 10Majavah: toolforge: Remove support for ingress workers [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1276631 (https://phabricator.wikimedia.org/T392356) [09:41:02] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.remove_k8s_node [09:42:21] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) [09:51:32] FIRING: [2x] TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [09:51:43] FIRING: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [09:53:57] 10Cloud-VPS (Project-requests): Request creation of wolf-a Cloud VPS project - https://phabricator.wikimedia.org/T424192#11850556 (10fgiunchedi) I have listed the blockers below, and I'm having an hard time understanding why Toolforge would not work with a fully containerized deployment. Would you mind expanding... [09:56:32] RESOLVED: [2x] TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [09:56:43] RESOLVED: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [10:26:46] 06cloud-services-team, 10Cloud-VPS: Deleting ENC projects is broken - https://phabricator.wikimedia.org/T416588#11850708 (10taavi) After running the backfill script, we're left with the following stale items in the database: ` MariaDB [labspuppet]> select * from prefix where prefix_project_id is null; +-------... [10:38:42] 10Toolforge, 06tools-platform-team: maintain-dbusers should gracefully handle missing kubeconfig - https://phabricator.wikimedia.org/T424207 (10taavi) 03NEW [10:44:05] 10Toolforge, 06tools-platform-team: maintain-dbusers should not pass sensitive data in command line parameters - https://phabricator.wikimedia.org/T424209 (10taavi) 03NEW [11:04:01] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team, 07OKR-Work: Improve linting - enum descriptions - https://phabricator.wikimedia.org/T424210 (10KBach) 03NEW [11:06:51] 10Tool-wikimedia-attribution: Document referrer recommendations for Participation CTA - https://phabricator.wikimedia.org/T423567#11850836 (10Sarai-WMF) Status update: - Kadeem provided a revised version of the recommendation, which as been documented in the corresponding section of our framework's [[ https://d... [11:10:28] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team, 07OKR-Work: Improve linting - enum descriptions - https://phabricator.wikimedia.org/T424210#11850838 (10KBach) [11:15:39] 10Tool-wikimedia-attribution: Update data links to the Attribution API Beta - https://phabricator.wikimedia.org/T421385#11850877 (10Sarai-WMF) [11:26:57] 10Tool-wikimedia-attribution: Add a top-level “Technical implementation” section to the Wikimedia Attribution Framework - https://phabricator.wikimedia.org/T424212 (10Sarai-WMF) 03NEW [11:27:27] 10Tool-wikimedia-attribution: Add a top-level “Technical implementation” section to the Wikimedia Attribution Framework - https://phabricator.wikimedia.org/T424212#11850910 (10Sarai-WMF) a:03Sarai-WMF [11:36:02] (03open) 10vriaa: feat: add support for templates with CTAs [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/58 [11:44:03] (03CR) 10Filippo Giunchedi: [C:03+1] "LGTM, nice" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1276631 (https://phabricator.wikimedia.org/T392356) (owner: 10Majavah) [11:46:14] 10Cloud-VPS (Project-requests): Request creation of wolf-a Cloud VPS project - https://phabricator.wikimedia.org/T424192#11850952 (10Aklapper) 05Open→03Stalled @Moh777tyjer3c: Which Wikimedia communities are interested in this? Could you point to relevant discussions, please? Which other people will be (co-)... [11:57:43] 10Cloud-VPS (Project-requests): Request creation of wise VPS project - https://phabricator.wikimedia.org/T424068#11851033 (10fgiunchedi) Hello and thank you for reaching out. Wikimedia Foundation offers and supports a platform for running managed user workloads called Toolforge (https://wikitech.wikimedia.org/wi... [12:01:56] (03open) 10filippo: Remove ToM from notes template [repos/cloud/wmcs/utils] - 10https://gitlab.wikimedia.org/repos/cloud/wmcs/utils/-/merge_requests/2 [12:04:46] (03approved) 10volans: Remove ToM from notes template [repos/cloud/wmcs/utils] - 10https://gitlab.wikimedia.org/repos/cloud/wmcs/utils/-/merge_requests/2 (owner: 10filippo) [12:08:02] (03merge) 10filippo: Remove ToM from notes template [repos/cloud/wmcs/utils] - 10https://gitlab.wikimedia.org/repos/cloud/wmcs/utils/-/merge_requests/2 [12:09:45] (03open) 10filippo: k8s: adjust default memory requests [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/102 (https://phabricator.wikimedia.org/T420565) [12:10:14] (03update) 10filippo: k8s: adjust default memory requests [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/102 (https://phabricator.wikimedia.org/T420565) [12:10:38] (03CR) 10Majavah: [C:03+2] toolforge: Remove support for ingress workers [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1276631 (https://phabricator.wikimedia.org/T392356) (owner: 10Majavah) [12:11:02] (03merge) 10taavi: kubernetes: Remove ingress-nginx specific alerts [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/61 (https://phabricator.wikimedia.org/T392356) [12:14:41] (03Merged) 10jenkins-bot: toolforge: Remove support for ingress workers [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1276631 (https://phabricator.wikimedia.org/T392356) (owner: 10Majavah) [12:17:35] (03update) 10filippo: k8s: adjust default memory requests [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/102 (https://phabricator.wikimedia.org/T420565) [12:43:41] 06cloud-services-team, 10Toolforge: Buildservice Rust job won't start - https://phabricator.wikimedia.org/T424176#11851185 (10magnusmanske) 05Open→03Resolved a:03magnusmanske [13:11:16] 06cloud-services-team, 10Toolforge, 06tools-platform-team, 13Patch-For-Review: Audit tools memory requests vs actual usage - https://phabricator.wikimedia.org/T420565#11851288 (10fgiunchedi) I started from webservice-cli limits, and was thinking of the following deployment plan: 1. New package is availabl... [13:17:01] (03open) 10fnegri: builds-api: update heroku builder and runner images [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1225 (https://phabricator.wikimedia.org/T423417) [13:25:42] (03open) 10vriaa: Draft: fix: spacing overflow bug [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/59 [13:47:50] (03update) 10filippo: k8s: adjust default memory requests [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/102 (https://phabricator.wikimedia.org/T420565) [13:52:06] 06cloud-services-team, 10superset.wmcloud.org: Sunset superset.wmcloud.org - https://phabricator.wikimedia.org/T416373#11851454 (10Andrew) note to self, also need to sort out github access [14:02:18] (03open) 10marostegui: Add new delete to the decommissioning process [toolforge-repos/switchmaster] - 10https://gitlab.wikimedia.org/toolforge-repos/switchmaster/-/merge_requests/18 [14:02:34] (03close) 10marostegui: Add new delete to the decommissioning process [toolforge-repos/switchmaster] - 10https://gitlab.wikimedia.org/toolforge-repos/switchmaster/-/merge_requests/18 [14:02:34] (03update) 10marostegui: Add new delete to the decommissioning process [toolforge-repos/switchmaster] - 10https://gitlab.wikimedia.org/toolforge-repos/switchmaster/-/merge_requests/18 [14:06:10] (03open) 10marostegui: decommission.py: Add new delete [toolforge-repos/switchmaster] - 10https://gitlab.wikimedia.org/toolforge-repos/switchmaster/-/merge_requests/19 [14:39:13] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 07Essential-Work, 06MW-Interfaces-Team (MWI-Sprint-32 (2026-04-21 to 2026-05-05)): Linter: Split rules into more logical groupings. - https://phabricator.wikimedia.org/T415914#11851619 (10KBach) 05Open→03In progress [14:39:33] (03approved) 10ladsgroup: decommission.py: Add new delete [toolforge-repos/switchmaster] - 10https://gitlab.wikimedia.org/toolforge-repos/switchmaster/-/merge_requests/19 (owner: 10marostegui) [14:41:14] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MW-Interfaces-Team (MWI-Sprint-32 (2026-04-21 to 2026-05-05)), 07OKR-Work: Add linting rules for operations and paths - https://phabricator.wikimedia.org/T422504#11851629 (10KBach) a:03KineticPelagic [14:42:12] (03merge) 10marostegui: decommission.py: Add new delete [toolforge-repos/switchmaster] - 10https://gitlab.wikimedia.org/toolforge-repos/switchmaster/-/merge_requests/19 [15:15:32] FIRING: InstanceDown: Project tools instance tools-puppetserver-01 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [15:27:49] (03open) 10raymond-ndibe: tests: fix wrong test data [repos/cloud/toolforge/jobs-api] (fix_jobs_load_bug) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/291 (https://phabricator.wikimedia.org/T423880 https://phabricator.wikimedia.org/T423891) [15:28:28] (03update) 10raymond-ndibe: tests: fix wrong test data [repos/cloud/toolforge/jobs-api] (fix_jobs_load_bug) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/291 (https://phabricator.wikimedia.org/T423880 https://phabricator.wikimedia.org/T423891) [15:44:32] (03approved) 10fnegri: k8s: adjust default memory requests [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/102 (https://phabricator.wikimedia.org/T420565) (owner: 10filippo) [15:50:42] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MW-Interfaces-Team (MWI-Sprint-32 (2026-04-21 to 2026-05-05)), 13Patch-For-Review: OpenAPI linting: Add missing OpenAPI spec elements to Response Components - https://phabricator.wikimedia.org/T422739#11851988 (10HCoplin-WMF) [16:05:32] RESOLVED: InstanceDown: Project tools instance tools-puppetserver-01 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [16:15:32] FIRING: PuppetAgentNoResources: No Puppet resources found on instance tools-k8s-worker-103 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [16:19:24] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MW-Interfaces-Team (MWI-Sprint-32 (2026-04-21 to 2026-05-05)), 07OKR-Work: Add linting rules for operations and paths - https://phabricator.wikimedia.org/T422504#11852134 (10KineticPelagic) Depends on https://phabricator.wikimedia.org/T415914. [16:20:39] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MW-Interfaces-Team (MWI-Sprint-32 (2026-04-21 to 2026-05-05)), 07OKR-Work: Add linting rules for operations and paths - https://phabricator.wikimedia.org/T422504#11852142 (10KineticPelagic) >>! In T422504#11797157, @HCoplin-WMF wrote: > Flagging that I bel... [16:30:32] RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance tools-k8s-worker-103 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [16:46:40] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MW-Interfaces-Team (MWI-Sprint-32 (2026-04-21 to 2026-05-05)), 07OKR-Work: Add linting rules for operations and paths - https://phabricator.wikimedia.org/T422504#11852268 (10HCoplin-WMF) This spreadsheet! :D https://docs.google.com/spreadsheets/d/1nxGsWvpu... [17:08:59] (03approved) 10aaron: fix(linter): fix five false positives on OAD example [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/15 (owner: 10aghirelli) [17:09:06] (03merge) 10aaron: fix(linter): fix five false positives on OAD example [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/15 (owner: 10aghirelli) [17:15:38] 06cloud-services-team, 10Toolforge: [builds-api] `step-fix-procfile-args-passing` failing, blocking new builds - https://phabricator.wikimedia.org/T424191#11852377 (10Raymond_Ndibe) Hello Damian what does your config look like? [18:33:26] (03approved) 10pepepiton: Testing pipeline [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/182 (owner: 10davidcoronel) [18:33:42] (03merge) 10pepepiton: Testing pipeline [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/182 (owner: 10davidcoronel) [19:44:22] 06cloud-services-team, 10Openstack-Magnum: Investigate new Magnum drivers - https://phabricator.wikimedia.org/T393782#11852853 (10Andrew) ...and, it turns out upstream has better support for the capi driver (not the capi-helm driver) so now I'm setting that up instead. https://stackhpc-kayobe-config.readthedo... [20:03:18] 10Tool-humaniki-2: "unknown" gender QID gets saved to database - https://phabricator.wikimedia.org/T424150#11852926 (10Danya) 05Open→03In progress [20:03:23] 10Tool-humaniki-2: "unknown" gender QID gets saved to database - https://phabricator.wikimedia.org/T424150#11852927 (10Danya) a:03Danya [20:04:10] (03open) 10danyya: Draft: Fix unknown qid in db [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/13 [20:13:54] (03update) 10danyya: Draft: Fix unknown qid in db [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/13 [20:20:25] (03update) 10danyya: Fix unknown qid in db [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/13 [20:20:25] (03open) 10aghirelli: chore(tests): split rule tests by validation domain (T415914) [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/16 [20:20:34] (03merge) 10danyya: Fix unknown qid in db [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/13 [20:22:10] 10Tool-humaniki-2: "unknown" gender QID gets saved to database - https://phabricator.wikimedia.org/T424150#11852965 (10Danya) 05In progress→03Resolved [20:23:29] (03update) 10aghirelli: chore(tests): split rule tests by validation domain (T415914) [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/16 [20:24:22] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 07Essential-Work, 06MW-Interfaces-Team (MWI-Sprint-32 (2026-04-21 to 2026-05-05)), 13Patch-For-Review: Linter: Split rules into more logical groupings. - https://phabricator.wikimedia.org/T415914#11852979 (10AGhirelli-WMF) [20:34:45] (03open) 10danyya: Draft: Fix SQL CLI [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/14 [20:36:23] (03CR) 10Akoopal: [C:03+2] "Useful test, tested and working." [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1275515 (owner: 10Lokal Profil) [20:38:37] (03Merged) 10jenkins-bot: Catch duplicate field dest values in monuments_config [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1275515 (owner: 10Lokal Profil) [20:45:02] (03CR) 10Akoopal: [C:03+2] "Although at the moment not really needed, good practice to do this. Looks good overal." [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1275522 (owner: 10Lokal Profil) [20:47:10] (03Merged) 10jenkins-bot: Escape field defaults when generating CREATE TABLE SQL [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1275522 (owner: 10Lokal Profil) [20:59:00] 10Tool-paulina: Create functional tests for the basic flows - https://phabricator.wikimedia.org/T401015#11853082 (10Pepe_piton) First iteration of tests: https://gitlab.wikimedia.org/toolforge-repos/paulina/-/commit/fb222b8c07a0cd0c21f4e7b87b0ddd50056a9995 Thanks @DidiCoronel! [21:07:01] (03CR) 10Lokal Profil: Tighten monuments_config schema and validation tests (031 comment) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1275532 (owner: 10Lokal Profil) [21:16:34] (03open) 10danyya: Draft: Fix scheduler data fetching [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/15 [21:19:10] (03update) 10danyya: Draft: Fix SQL CLI [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/14 [21:24:44] 10Tool-humaniki-2: Use dumps instead of live QLever/MediaWiki queries? - https://phabricator.wikimedia.org/T424147#11853198 (10Danya) 05In progress→03Declined Wikidata dumps are way too big to be usable (100+ GB when compressed). [21:34:27] 10Tool-humaniki-2: Reduce snapshots frequency - https://phabricator.wikimedia.org/T424277 (10Danya) 03NEW [21:45:59] (03update) 10danyya: Draft: Fix scheduler data fetching [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/15 [21:50:48] 10Tool-humaniki-2: Notify tool’s devs when snapshot capture fails - https://phabricator.wikimedia.org/T424279 (10Danya) 03NEW [21:51:50] 10Tool-humaniki-2: Notify tool’s devs when snapshot capture fails - https://phabricator.wikimedia.org/T424279#11853290 (10Danya) p:05Triage→03Medium [21:51:53] 10Tool-humaniki-2: Reduce snapshots frequency - https://phabricator.wikimedia.org/T424277#11853291 (10Danya) p:05Triage→03Medium [21:52:14] 10Tool-humaniki-2: Localize table labels - https://phabricator.wikimedia.org/T423787#11853294 (10Danya) p:05Low→03Medium [21:54:57] (03update) 10danyya: Fix scheduler data fetching [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/15 [21:55:15] (03merge) 10danyya: Fix scheduler data fetching [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/15 [22:26:28] (03update) 10danyya: Draft: Fix SQL CLI [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/14 [22:27:18] (03update) 10danyya: Fix SQL CLI [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/14 [22:28:20] (03merge) 10danyya: Fix SQL CLI [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/14 [22:32:09] 10Tool-humaniki-2: Scheduler does not take birth year snapshots - https://phabricator.wikimedia.org/T423025#11853382 (10Danya) A potential fix has been deployed, waiting for next Sunday before closing this ticket in order to check if it has actually fixed the issue. [22:50:57] 10Tool-humaniki-2: Button to download the whole database? - https://phabricator.wikimedia.org/T424282 (10Danya) 03NEW [22:51:10] 10Tool-humaniki-2: Button to download the whole database? - https://phabricator.wikimedia.org/T424282#11853417 (10Danya) p:05Triage→03Low [23:19:48] 06cloud-services-team, 10Toolforge: [builds-api] `step-fix-procfile-args-passing` failing, blocking new builds - https://phabricator.wikimedia.org/T424191#11853468 (10DamianZaremba) >>! In T424191#11852377, @Raymond_Ndibe wrote: > Hello Damian what does your config look like? The component config is https://g... [23:26:55] 06cloud-services-team, 10Toolforge: [builds-api] `step-fix-procfile-args-passing` failing, blocking new builds - https://phabricator.wikimedia.org/T424191#11853474 (10DamianZaremba) It appears that if you use `-` in the command name the script explodes. I have a successful build after ` cbng: ./scripts/run-cb... [23:28:31] !log tools.cluebotng Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/24863962598 (https://github.com/cluebotng/component-configs/commits/07b9ad8616853af2ed49f96de6e55c14e3faabe0) [23:28:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL