[00:33:18] (03open) 10damian: Use image name from builds-api [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/134 (https://phabricator.wikimedia.org/T403322) [00:43:25] (03update) 10raymond-ndibe: [jobs-cli] quota refactor [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/97 (https://phabricator.wikimedia.org/T389118) [00:47:23] (03update) 10damian: Draft: getBuild - add digest to image [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/144 (https://phabricator.wikimedia.org/T403322) [00:57:32] (03update) 10damian: Use image name from builds-api [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/134 (https://phabricator.wikimedia.org/T403322) [01:19:33] (03update) 10damian: Draft: getBuild - add digest to image [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/144 (https://phabricator.wikimedia.org/T403322) [01:20:49] (03update) 10damian: Draft: getBuild - add digest to image [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/144 (https://phabricator.wikimedia.org/T403322) [01:21:21] (03update) 10damian: Draft: getBuild - add digest to image [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/144 (https://phabricator.wikimedia.org/T403322) [01:22:33] (03update) 10damian: Draft: getBuild - add digest to image [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/144 (https://phabricator.wikimedia.org/T403322) [01:24:07] (03update) 10damian: Draft: getBuild - add digest to image [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/144 (https://phabricator.wikimedia.org/T403322) [01:30:30] (03update) 10damian: Draft: getBuild - add digest to image [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/144 (https://phabricator.wikimedia.org/T403322) [01:39:14] (03update) 10damian: Draft: getBuild - add digest to image [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/144 (https://phabricator.wikimedia.org/T403322) [01:42:51] (03update) 10damian: Draft: getBuild - add digest to image [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/144 (https://phabricator.wikimedia.org/T403322) [01:57:49] (03update) 10damian: Draft: getBuild - add digest to image [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/144 (https://phabricator.wikimedia.org/T403322) [01:58:37] (03update) 10damian: getBuild - add digest to image [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/144 (https://phabricator.wikimedia.org/T403322) [01:59:07] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: [builds-api] return image digest - https://phabricator.wikimedia.org/T403322#11212650 (10DamianZaremba) https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/134 - use the value returned from builds-api, rather than construc... [01:59:53] (03update) 10damian: getBuild - add digest to image [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/144 (https://phabricator.wikimedia.org/T403322) [02:05:44] (03update) 10damian: getBuild - add digest to image [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/144 (https://phabricator.wikimedia.org/T403322) [03:53:38] 06cloud-services-team, 10Toolforge: Toolforge jobs for milhistbot not running - https://phabricator.wikimedia.org/T405310#11212670 (10Hawkeye7) I tried running the one-off job (aclass3) today, and it went from "Pending" to "Pod in 'Pending' phase. State 'waiting'. Reason 'ContainerCreating'" in less than 20 s... [04:22:36] 06cloud-services-team, 10Toolforge: toolforge buildservice based tool is not starting - https://phabricator.wikimedia.org/T405319#11212687 (10santhosh) Thanks. It works now. Tool is up https://api-explorer.toolforge.org/ [04:23:00] 06cloud-services-team, 10Toolforge: toolforge buildservice based tool is not starting - https://phabricator.wikimedia.org/T405319#11212688 (10santhosh) 05Open→03Resolved a:03santhosh [07:30:41] (03open) 10taavi: kubernetes: Page on web service global error rate [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/40 (https://phabricator.wikimedia.org/T399870) [07:30:44] (03update) 10taavi: kubernetes: Page on web service global error rate [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/40 (https://phabricator.wikimedia.org/T399870) [07:47:22] (03update) 10taavi: kubernetes: Page on web service global error rate [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/40 (https://phabricator.wikimedia.org/T399870) [07:51:41] (03approved) 10dcaro: d/changelog: bump to 16.1.22 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/132 (https://phabricator.wikimedia.org/T390136) [07:51:46] (03merge) 10dcaro: d/changelog: bump to 16.1.22 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/132 (https://phabricator.wikimedia.org/T390136) [07:53:25] (03merge) 10dcaro: jobs: lower the default cpu limit to 1cpu [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/217 [07:53:37] (03update) 10dcaro: [jobs-api] split job models to oneoff, scheduled and continuous [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/154 (https://phabricator.wikimedia.org/T389118 https://phabricator.wikimedia.org/T390136) (owner: 10raymond-ndibe) [07:54:48] (03update) 10dcaro: [status] make job status an enum, with clearly defined states [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/208 (https://phabricator.wikimedia.org/T401172) (owner: 10raymond-ndibe) [07:55:36] (03update) 10taavi: kubernetes: Page on web service global error rate [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/40 (https://phabricator.wikimedia.org/T399870) [07:55:45] 06cloud-services-team, 10Cloud-VPS, 06SRE-OnFire, 05Cloud-Services-Origin-Team, and 2 others: Cloud VPS: NFS servers: the current setup requires a puppet run after a reboot to get address right - https://phabricator.wikimedia.org/T347681#11212920 (10fgiunchedi) a:05dcaro→03fgiunchedi [07:55:57] (03update) 10taavi: service: Split security group logic to new file [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/83 [07:55:58] (03update) 10taavi: shared: Manage Kubernetes HAProxy groups [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/84 [07:56:06] (03open) 10taavi: service: Split security group logic to new file [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/83 [07:56:07] (03open) 10taavi: shared: Manage Kubernetes HAProxy groups [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/84 [07:56:13] (03update) 10taavi: service: Split security group logic to new file [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/83 [07:56:27] (03update) 10taavi: shared: Manage Kubernetes HAProxy groups [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/84 [07:56:30] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-api: bump to 0.0.417-20250925075336-32d4be5a [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/979 [07:56:32] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-api: bump to 0.0.417-20250925075336-32d4be5a [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/979 [08:00:47] (03update) 10taavi: shared: Manage Kubernetes HAProxy groups [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/84 [08:03:05] (03update) 10taavi: service: Split security group logic to new file [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/83 [08:03:05] (03update) 10taavi: service: Set a default for security_group_rules [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/85 [08:03:05] (03update) 10taavi: shared: Manage Kubernetes HAProxy groups [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/84 [08:03:07] (03open) 10taavi: service: Set a default for security_group_rules [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/85 [08:03:14] (03update) 10taavi: shared: Manage Kubernetes HAProxy groups [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/84 [08:03:19] (03update) 10taavi: service: Set a default for security_group_rules [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/85 [08:04:44] (03update) 10taavi: service: Split security group logic to new file [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/83 [08:06:42] (03CR) 10Brouberol: [C:03+1] idp: Add dummy data for airflow-wikidata [labs/private] - 10https://gerrit.wikimedia.org/r/1191190 (https://phabricator.wikimedia.org/T404073) (owner: 10Stevemunene) [08:08:04] (03update) 10taavi: shared: Manage Kubernetes HAProxy groups [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/84 [08:10:25] (03approved) 10fnegri: kubernetes: Page on web service global error rate [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/40 (https://phabricator.wikimedia.org/T399870) (owner: 10taavi) [08:11:32] (03update) 10taavi: shared: Manage Kubernetes HAProxy groups [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/84 [08:26:33] (03CR) 10Stevemunene: [V:03+2 C:03+2] idp: Add dummy data for airflow-wikidata [labs/private] - 10https://gerrit.wikimedia.org/r/1191190 (https://phabricator.wikimedia.org/T404073) (owner: 10Stevemunene) [08:28:58] (03update) 10taavi: shared: Manage Kubernetes HAProxy groups [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/84 [08:29:11] (03update) 10taavi: service: Set a default for security_group_rules [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/85 [08:41:17] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [08:43:13] 06cloud-services-team, 10Data-Services, 06Community-Tech, 06Data-Platform-SRE, 10Multiblocks: Unexpected error "Subquery returns more than 1 row" on wiki replicas - https://phabricator.wikimedia.org/T404473#11213094 (10fnegri) I found the issue, I did run `maintain-views` too soon, before Puppet did sync... [08:54:18] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [09:14:33] (03PS1) 10Majavah: wmcs_libs: k8s: Support tofu-managed groups for HAProxy [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1191307 (https://phabricator.wikimedia.org/T405078) [09:14:35] (03PS1) 10Majavah: toolforge: k8s: Resolve K8s HAProxy VIPs from Hiera [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1191308 (https://phabricator.wikimedia.org/T405078) [09:14:38] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [09:17:13] 10Toolforge, 06tools-infrastructure-team: Merge Toolforge Nginx front proxy into the existing K8s HAProxy setup - https://phabricator.wikimedia.org/T283948#11213214 (10taavi) [09:18:21] (03CR) 10CI reject: [V:04-1] toolforge: k8s: Resolve K8s HAProxy VIPs from Hiera [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1191308 (https://phabricator.wikimedia.org/T405078) (owner: 10Majavah) [09:25:00] (03merge) 10dcaro: loki.alloy: decrease frequency for fetching logs [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/962 [09:26:23] (03PS2) 10Majavah: toolforge: k8s: Resolve K8s HAProxy VIPs from Hiera [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1191308 (https://phabricator.wikimedia.org/T405078) [09:27:44] (03update) 10taavi: service: Split security group logic to new file [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/83 [09:27:45] (03update) 10taavi: shared: Manage Kubernetes HAProxy groups [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/84 [09:27:45] (03update) 10taavi: service: Set a default for security_group_rules [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/85 [09:27:45] (03update) 10taavi: shared: Allocate a VIP for HAProxy Keepalived usage [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/86 (https://phabricator.wikimedia.org/T405078) [09:27:47] (03open) 10taavi: shared: Allocate a VIP for HAProxy Keepalived usage [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/86 (https://phabricator.wikimedia.org/T405078) [09:27:54] (03update) 10taavi: shared: Allocate a VIP for HAProxy Keepalived usage [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/86 (https://phabricator.wikimedia.org/T405078) [09:28:41] (03update) 10taavi: shared: Allocate a VIP for HAProxy Keepalived usage [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/86 (https://phabricator.wikimedia.org/T405078) [09:29:21] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [09:30:45] (03update) 10taavi: shared: Allocate a VIP for HAProxy Keepalived usage [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/86 (https://phabricator.wikimedia.org/T405078) [09:33:15] (03update) 10taavi: shared: Allocate a VIP for HAProxy Keepalived usage [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/86 (https://phabricator.wikimedia.org/T405078) [09:33:57] (03merge) 10taavi: kubernetes: Page on web service global error rate [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/40 (https://phabricator.wikimedia.org/T399870) [09:34:40] 06cloud-services-team, 10Toolforge, 06SRE-OnFire, 13Patch-For-Review, 10Sustainability (Incident Followup): [k8s,infra,o11y] Add paging alert when many tools are unreachable - https://phabricator.wikimedia.org/T399870#11213300 (10taavi) 05Open→03Resolved [09:59:09] (03approved) 10dcaro: jobs-api: bump to 0.0.417-20250925075336-32d4be5a [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/979 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [09:59:12] (03update) 10dcaro: jobs-api: bump to 0.0.417-20250925075336-32d4be5a [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/979 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [09:59:31] (03merge) 10dcaro: jobs-api: bump to 0.0.417-20250925075336-32d4be5a [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/979 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [10:18:24] (03CR) 10FNegri: global: minor cleanups (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1186483 (owner: 10David Caro) [10:20:36] (03merge) 10dcaro: [jobs-api] split job models to oneoff, scheduled and continuous [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/154 (https://phabricator.wikimedia.org/T389118 https://phabricator.wikimedia.org/T390136) (owner: 10raymond-ndibe) [10:23:21] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-api: bump to 0.0.418-20250925102045-75109675 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/980 (https://phabricator.wikimedia.org/T389118 https://phabricator.wikimedia.org/T390136) [10:23:45] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [10:35:11] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [10:49:26] (03CR) 10Filippo Giunchedi: "I don't have enough expertise yet on this part of the cookbooks, patch LGTM overall though" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1191307 (https://phabricator.wikimedia.org/T405078) (owner: 10Majavah) [10:51:01] (03CR) 10Filippo Giunchedi: [C:03+1] toolforge: k8s: Resolve K8s HAProxy VIPs from Hiera [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1191308 (https://phabricator.wikimedia.org/T405078) (owner: 10Majavah) [10:54:07] (03PS1) 10Majavah: wmcs_libs: k8s: Fix is_worker logic [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1191332 [10:58:17] (03CR) 10CI reject: [V:04-1] wmcs_libs: k8s: Fix is_worker logic [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1191332 (owner: 10Majavah) [10:59:10] (03PS2) 10Majavah: wmcs_libs: k8s: Fix is_worker logic [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1191332 [11:00:12] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: [builds-api] return image digest - https://phabricator.wikimedia.org/T403322#11213651 (10DamianZaremba) The results are already exposed to the `TaskRun`, so the above finds the `TaskRun` for the given `PipelineRun` and yanks the values out of the object... [11:21:53] (03update) 10damian: getBuild - add digest to image [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/144 (https://phabricator.wikimedia.org/T403322) [11:54:32] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [12:04:35] !log dcaro@cloudcumin1001 tools END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api [12:05:52] 06cloud-services-team, 10MediaWiki-extensions-TorBlock: TorBlock maintenance failures on labweb hosts - https://phabricator.wikimedia.org/T228465#11213984 (10Reedy) [12:10:00] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [12:22:19] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [12:29:44] 10Cloud-VPS (Quota-requests): Disk quota increase for project catalyst - https://phabricator.wikimedia.org/T405363#11214068 (10jnuche) 05Resolved→03Open We actually need an additional 250G to be able to create not just the snapshot but also a volume out of that snapshot before it can be transferred to a diff... [13:04:55] 10Cloud-VPS (Quota-requests): Disk quota increase for project catalyst - https://phabricator.wikimedia.org/T405363#11214197 (10taavi) 05Open→03Resolved Please open a new task instead of re-using an existing one [13:08:17] 10Cloud-VPS (Quota-requests): Disk quota increase for project catalyst - https://phabricator.wikimedia.org/T405584 (10jnuche) 03NEW [13:09:18] 10Cloud-VPS (Quota-requests): Disk quota increase for project catalyst - https://phabricator.wikimedia.org/T405584#11214229 (10jnuche) [13:12:16] (03approved) 10fnegri: service: Split security group logic to new file [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/83 (owner: 10taavi) [13:12:30] (03approved) 10fnegri: service: Set a default for security_group_rules [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/85 (owner: 10taavi) [13:14:42] (03merge) 10taavi: service: Split security group logic to new file [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/83 [13:14:45] (03update) 10taavi: service: Set a default for security_group_rules [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/85 [13:15:20] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [13:21:13] (03merge) 10taavi: service: Set a default for security_group_rules [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/85 [13:21:16] (03update) 10taavi: shared: Manage Kubernetes HAProxy groups [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/84 [13:28:27] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [13:41:42] (03open) 10dcaro: api: add back patch to create_job [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/222 [13:48:28] (03update) 10dcaro: api: add back patch to create_job [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/222 [13:56:59] (03update) 10dcaro: api: add back patch to create_job [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/222 [14:05:39] (03update) 10raymond-ndibe: [tool-config] handle unset and default arguments consistently [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/123 (https://phabricator.wikimedia.org/T401648 https://phabricator.wikimedia.org/T402572) [14:07:16] 06cloud-services-team, 10Openstack-Magnum: ssh to cloud-vps 'utility' nodes (magnum, trove, octavia) - https://phabricator.wikimedia.org/T402317#11214496 (10fgiunchedi) >>! In T402317#11210466, @Andrew wrote: >> - Install Trove root keys on cloudcontrols, as octavia root keys are already >> > > This is now... [14:09:13] 10Tool-paulina, 10Outreachy (Round 31): Outreachy 31: Features to edit author and work data on Wikidata directly from Paulina - https://phabricator.wikimedia.org/T392429#11214528 (10Nat_WDU) [14:10:19] 10Tool-paulina, 10Outreachy (Round 31): Outreachy 31: Features to edit author and work data on Wikidata directly from Paulina - https://phabricator.wikimedia.org/T392429#11214535 (10Nat_WDU) [14:17:13] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18008240418 (https://github.com/cluebotng/component-configs/commits/9949a4a5acff374c1edd7b6e21959a28721e02d0) [14:17:16] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [14:17:59] 10Tool-wsindex, 10Developer-Outreach, 10Outreachy (Round 31): Outreachy 31: Improve the Wikisource Reader App - https://phabricator.wikimedia.org/T405593#11214605 (10Saiphani02) [14:23:05] 10Tool-paulina: Improvements to the copyright term page: term description - https://phabricator.wikimedia.org/T405598 (10Nat_WDU) 03NEW [14:24:19] 10Tool-paulina, 10Outreachy (Round 31): Outreachy 31: Features to edit author and work data on Wikidata directly from Paulina - https://phabricator.wikimedia.org/T392429#11214646 (10Nat_WDU) [14:29:03] 06cloud-services-team, 10Toolforge: [jobs-cli] not including `port` in dump - https://phabricator.wikimedia.org/T405601 (10DamianZaremba) 03NEW [14:31:16] (03open) 10damian: job_prepare_for_dump - Do not remove `port` [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/133 [14:31:43] 06cloud-services-team, 10Toolforge: [jobs-cli] not including `port` in dump - https://phabricator.wikimedia.org/T405601#11214701 (10DamianZaremba) https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/133 [14:33:20] (03update) 10damian: job_prepare_for_dump - Do not remove `port` [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/133 [14:34:01] (03update) 10damian: job_prepare_for_dump - Do not remove `port` [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/133 [14:35:03] 10Tool-paulina: Improvements to the copyright term page: contry list styles - https://phabricator.wikimedia.org/T405603 (10Nat_WDU) 03NEW [14:35:21] (03update) 10dcaro: api: add back patch to create_job [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/222 [14:35:28] 10Tool-paulina: Improvements to the copyright term page: country list styles - https://phabricator.wikimedia.org/T405603#11214732 (10Nat_WDU) [14:36:01] !log tools.cluebotng-trainer Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18008148799 (https://github.com/cluebotng/component-configs/commits/0bb2ca770839346853b6a8440927a8afde70c94d) [14:36:04] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-trainer/SAL [14:36:17] 10Tool-paulina, 10Outreachy (Round 31): Outreachy 31: Features to edit author and work data on Wikidata directly from Paulina - https://phabricator.wikimedia.org/T392429#11214745 (10Nat_WDU) [14:37:15] (03update) 10dcaro: api: support getting `continuous` field [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/222 [14:47:25] 10Tool-paulina: Prefill nationality of author and country of origin of work at the country page - https://phabricator.wikimedia.org/T405604 (10Nat_WDU) 03NEW [14:48:00] 10Tool-paulina, 10Outreachy (Round 31): Outreachy 31: Features to edit author and work data on Wikidata directly from Paulina - https://phabricator.wikimedia.org/T392429#11214810 (10Nat_WDU) [14:50:09] 06cloud-services-team, 10Toolforge: [jobs-cli] erroneously removes `port` from `dump` when `port_protocol` is not set - https://phabricator.wikimedia.org/T405601#11214832 (10DamianZaremba) [14:50:47] 06cloud-services-team, 10Toolforge: [jobs-cli] during `dump` erroneously removes `port` from `job` when `port_protocol` is not set - https://phabricator.wikimedia.org/T405601#11214835 (10DamianZaremba) [14:53:26] 10Tool-paulina: Show link to SPARQL query at an author's list of works - https://phabricator.wikimedia.org/T397554#11214857 (10Nat_WDU) [14:56:55] 10Tool-paulina: At an author's list of works, show a link to the SPARQL query at the Wikidata Query Service - https://phabricator.wikimedia.org/T397554#11214884 (10Nat_WDU) [14:57:26] 10Tool-paulina, 10Outreachy (Round 31): Outreachy 31: Features to edit author and work data on Wikidata directly from Paulina - https://phabricator.wikimedia.org/T392429#11214890 (10Nat_WDU) [14:57:37] (03update) 10taavi: job_prepare_for_dump - Do not remove `port` [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/133 (https://phabricator.wikimedia.org/T405601) (owner: 10damian) [14:57:46] 10Tool-paulina: At an author's list of works, show a link to the SPARQL query at the Wikidata Query Service - https://phabricator.wikimedia.org/T397554#11214894 (10Nat_WDU) a:05Pepe_piton→03None [15:20:45] 06cloud-services-team, 10Data-Services, 06Community-Tech, 06Data-Platform-SRE, 10Multiblocks: Unexpected error "Subquery returns more than 1 row" on wiki replicas - https://phabricator.wikimedia.org/T404473#11214996 (10SD0001) 05In progress→03Resolved Working fine now. Thanks! [15:33:01] (03approved) 10dcaro: job_prepare_for_dump - Do not remove `port` [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/133 (https://phabricator.wikimedia.org/T405601) (owner: 10damian) [15:34:09] (03merge) 10dcaro: job_prepare_for_dump - Do not remove `port` [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/133 (https://phabricator.wikimedia.org/T405601) (owner: 10damian) [15:34:20] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18012641376 (https://github.com/cluebotng/component-configs/commits/87c176492b1f1fb18570dbb70687258843c5773c) [15:34:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [15:34:29] !log tools.cluebotng-staging Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18012641380 (https://github.com/cluebotng/component-configs/commits/87c176492b1f1fb18570dbb70687258843c5773c) [15:34:30] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-staging/SAL [15:36:57] (03open) 10dcaro: d/changelog: bump to 16.1.23 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/134 [15:39:00] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [15:39:00] !log dcaro@cloudcumin1001 toolsbeta END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api [15:40:01] !log taavi@cloudcumin1001 gitlab-runners-staging START - Cookbook wmcs.openstack.quota_increase (T404668) [15:40:06] T404668: Increase gitlab-runners-staging volumes to 12 - https://phabricator.wikimedia.org/T404668 [15:40:08] !log taavi@cloudcumin1001 gitlab-runners-staging END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) (T404668) [15:40:54] 10Cloud-VPS (Quota-requests): Increase iops for recommendation-api project - https://phabricator.wikimedia.org/T404254#11215139 (10taavi) 05Open→03Declined Unfortunately closing due to a lack of response to @fnegri's question. [15:41:01] 10Cloud-VPS (Quota-requests): Increase gitlab-runners-staging volumes to 12 - https://phabricator.wikimedia.org/T404668#11215141 (10taavi) 05Open→03Resolved a:03taavi [15:52:13] 10Cloud-VPS (Quota-requests): Disk quota increase for project catalyst - https://phabricator.wikimedia.org/T405584#11215214 (10dcaro) +1, will the snapshots and such be temporary? (curious, would be nice to clean them up when not needed) [15:52:53] 10Cloud-VPS (Quota-requests): Disk quota increase for project catalyst - https://phabricator.wikimedia.org/T405584#11215216 (10fnegri) +1 [15:54:29] !log taavi@cloudcumin1001 catalyst START - Cookbook wmcs.openstack.quota_increase (T405584) [15:54:34] T405584: Disk quota increase for project catalyst - https://phabricator.wikimedia.org/T405584 [15:54:36] !log taavi@cloudcumin1001 catalyst END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) (T405584) [15:54:53] 10Cloud-VPS (Quota-requests): Disk quota increase for project catalyst - https://phabricator.wikimedia.org/T405584#11215220 (10taavi) a:03taavi [15:54:57] 10Cloud-VPS (Quota-requests): Disk quota increase for project catalyst - https://phabricator.wikimedia.org/T405584#11215222 (10taavi) 05Open→03Resolved [15:57:59] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli [15:58:23] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli [15:58:36] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli [16:06:06] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18013531091 (https://github.com/cluebotng/component-configs/commits/9a6dc9f53f08ea206e75ad75ddddc3429e1e004f) [16:06:08] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [16:07:12] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli [16:07:23] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18013531088 (https://github.com/cluebotng/component-configs/commits/9a6dc9f53f08ea206e75ad75ddddc3429e1e004f) [16:07:24] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [16:07:33] !log tools.cluebotng-trainer Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18013531079 (https://github.com/cluebotng/component-configs/commits/9a6dc9f53f08ea206e75ad75ddddc3429e1e004f) [16:07:34] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-trainer/SAL [16:11:14] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18013677281 (https://github.com/cluebotng/component-configs/commits/371029d320611d8be6103da43ce9e0a91a2f8e1a) [16:11:16] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [16:12:25] !log tools.cluebotng-trainer Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18013677303 (https://github.com/cluebotng/component-configs/commits/371029d320611d8be6103da43ce9e0a91a2f8e1a) [16:12:26] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18013677351 (https://github.com/cluebotng/component-configs/commits/371029d320611d8be6103da43ce9e0a91a2f8e1a) [16:12:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-trainer/SAL [16:12:27] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [16:12:55] (03approved) 10dcaro: d/changelog: bump to 16.1.23 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/134 [16:12:58] (03merge) 10dcaro: d/changelog: bump to 16.1.23 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/134 [16:14:14] 06cloud-services-team, 10Toolforge (Toolforge iteration 24): [jobs-cli] during `dump` erroneously removes `port` from `job` when `port_protocol` is not set - https://phabricator.wikimedia.org/T405601#11215294 (10dcaro) 05Open→03Resolved a:03dcaro [16:15:16] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18013782359 (https://github.com/cluebotng/component-configs/commits/7e1eb9e3c9a52e0dd71cc58dc797183236a1c27e) [16:15:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [16:16:56] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18013782292 (https://github.com/cluebotng/component-configs/commits/7e1eb9e3c9a52e0dd71cc58dc797183236a1c27e) [16:16:58] !log tools.cluebotng-trainer Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18013782304 (https://github.com/cluebotng/component-configs/commits/7e1eb9e3c9a52e0dd71cc58dc797183236a1c27e) [16:16:58] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [16:16:59] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-trainer/SAL [16:17:02] !log tools.cluebotng-staging Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18013782309 (https://github.com/cluebotng/component-configs/commits/7e1eb9e3c9a52e0dd71cc58dc797183236a1c27e) [16:17:02] (03update) 10raymond-ndibe: [tool-config] handle unset and default arguments consistently [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/123 (https://phabricator.wikimedia.org/T401648 https://phabricator.wikimedia.org/T402572) [16:17:03] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-staging/SAL [16:27:06] 06cloud-services-team, 10Toolforge: [components-api] deployment is marked `successful` before finishing - https://phabricator.wikimedia.org/T405620 (10DamianZaremba) 03NEW [16:28:31] 06cloud-services-team, 10Toolforge: [components-api] deployment is marked `successful` before finishing - https://phabricator.wikimedia.org/T405620#11215376 (10DamianZaremba) [16:32:28] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18014221965 (https://github.com/cluebotng/component-configs/commits/b0737b89fc85c164c5a869aff21421ba21af2e4d) [16:32:31] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [16:33:51] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18014221959 (https://github.com/cluebotng/component-configs/commits/b0737b89fc85c164c5a869aff21421ba21af2e4d) [16:33:52] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [16:34:04] !log tools.cluebotng-trainer Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18014221974 (https://github.com/cluebotng/component-configs/commits/b0737b89fc85c164c5a869aff21421ba21af2e4d) [16:34:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-trainer/SAL [16:34:43] !log tools.cluebotng-staging Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18014221945 (https://github.com/cluebotng/component-configs/commits/b0737b89fc85c164c5a869aff21421ba21af2e4d) [16:34:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-staging/SAL [16:39:30] (03update) 10dcaro: jobs-api: bump to 0.0.418-20250925102045-75109675 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/980 (https://phabricator.wikimedia.org/T389118 https://phabricator.wikimedia.org/T390136) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [16:40:38] (03update) 10dcaro: jobs-api: bump to 0.0.418-20250925102045-75109675 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/980 (https://phabricator.wikimedia.org/T389118 https://phabricator.wikimedia.org/T390136) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [16:45:46] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18014528718 (https://github.com/cluebotng/component-configs/commits/96654b441f84901e1a607ced407eb9babb8fdbfc) [16:45:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [16:46:16] (03update) 10dcaro: jobs-api: bump to 0.0.418-20250925102045-75109675 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/980 (https://phabricator.wikimedia.org/T389118 https://phabricator.wikimedia.org/T390136) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [16:46:55] FIRING: PawsJupyterHubDown: PAWS JupyterHub is down https://wikitech.wikimedia.org/wiki/PAWS/Admin - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPawsJupyterHubDown [16:47:12] 10Tool-paulina: Improve country dropdown for the "country of origin" filter in the advanced search for works. - https://phabricator.wikimedia.org/T405621 (10Pepe_piton) 03NEW [16:47:28] FIRING: TargetDown: Job jupyterhub is unreachable in project paws instance hub-paws.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [16:50:47] !log dcaro@acme paws START - Cookbook wmcs.openstack.cloudvirt.vm_console [16:50:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL [16:51:56] RESOLVED: PawsJupyterHubDown: PAWS JupyterHub is down https://wikitech.wikimedia.org/wiki/PAWS/Admin - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPawsJupyterHubDown [16:52:28] RESOLVED: TargetDown: Job jupyterhub is unreachable in project paws instance hub-paws.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [16:52:41] (03approved) 10dcaro: api: support getting `continuous` field [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/222 [16:52:44] (03merge) 10dcaro: api: support getting `continuous` field [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/222 [16:54:06] 10Tool-paulina: Improve country dropdown - https://phabricator.wikimedia.org/T393278#11215519 (10Pepe_piton) 05In progress→03Resolved >>! In T393278#10924899, @DidiCoronel wrote: > Feature implemented on Authors search but not on Works search yet. I transformed the remainder of this task (the work's "Co... [16:55:19] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18014528718 (https://github.com/cluebotng/component-configs/commits/96654b441f84901e1a607ced407eb9babb8fdbfc) [16:55:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [16:55:55] 10Tool-paulina, 10Outreachy (Round 31): Outreachy 31: Features to edit author and work data on Wikidata directly from Paulina - https://phabricator.wikimedia.org/T392429#11215538 (10Pepe_piton) [16:56:06] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-api: bump to 0.0.420-20250925165253-31f66be8 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/980 (https://phabricator.wikimedia.org/T389118 https://phabricator.wikimedia.org/T390136) [16:56:08] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-api: bump to 0.0.420-20250925165253-31f66be8 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/980 (https://phabricator.wikimedia.org/T389118 https://phabricator.wikimedia.org/T390136) [16:56:16] !log dcaro@acme paws END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) [16:56:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL [16:56:24] !log dcaro@acme paws START - Cookbook wmcs.vps.instance.force_reboot vm paws-127c-uwce57bvcgrt-node-1 (cluster eqiad1, project paws) [16:56:25] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL [16:56:28] !log dcaro@acme paws END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm paws-127c-uwce57bvcgrt-node-1 (cluster eqiad1, project paws) [16:56:30] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL [16:56:36] !log dcaro@acme paws START - Cookbook wmcs.openstack.cloudvirt.vm_console [16:56:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL [16:56:46] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18014801881 (https://github.com/cluebotng/component-configs/commits/4f92189a79e68827f38e9a6a233b20c02529e77c) [16:56:46] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [16:56:54] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18014801858 (https://github.com/cluebotng/component-configs/commits/4f92189a79e68827f38e9a6a233b20c02529e77c) [16:56:55] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [16:56:59] !log tools.cluebotng-staging Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18014801883 (https://github.com/cluebotng/component-configs/commits/4f92189a79e68827f38e9a6a233b20c02529e77c) [16:57:00] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-staging/SAL [16:57:29] (03update) 10dcaro: jobs-api: bump to 0.0.420-20250925165253-31f66be8 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/980 (https://phabricator.wikimedia.org/T389118 https://phabricator.wikimedia.org/T390136) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [16:58:13] !log tools.cluebotng-trainer Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18014801857 (https://github.com/cluebotng/component-configs/commits/4f92189a79e68827f38e9a6a233b20c02529e77c) [16:58:14] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-trainer/SAL [17:01:34] 06cloud-services-team, 10Toolforge: [components-api] deployment is marked `successful` before finishing - https://phabricator.wikimedia.org/T405620#11215613 (10Raymond_Ndibe) a:03Raymond_Ndibe [17:01:48] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [17:02:35] 06cloud-services-team, 10Toolforge: [envvars] scope to jobs/components - https://phabricator.wikimedia.org/T405022#11215628 (10DamianZaremba) Today when removing an envvars used by `grafana-alloy` 2 unrelated containers got stuck in `CreateContainerConfigError` because the admission policy set their environmen... [17:04:11] 10Wikibugs, 10Phabricator (Upstream), 07Upstream: Wikibugs reports color of milestones wrong - https://phabricator.wikimedia.org/T395250#11215641 (10Aklapper) 05Stalled→03Resolved This issue should now be fixed on phabricator.wikimedia.org after today's software deployment in `T404134`. [17:07:55] (03open) 10raymond-ndibe: [do_deploy] update_deployment status only after successful jobs creation [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/135 (https://phabricator.wikimedia.org/T405620) [17:08:06] (03update) 10raymond-ndibe: [do_deploy] update_deployment status only after successful jobs creation [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/135 (https://phabricator.wikimedia.org/T405620) [17:14:15] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [17:14:25] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [17:16:43] (03approved) 10dcaro: [do_deploy] update_deployment status only after successful jobs creation [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/135 (https://phabricator.wikimedia.org/T405620) (owner: 10raymond-ndibe) [17:20:29] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18015392766 (https://github.com/cluebotng/component-configs/commits/61a7ceac210077c3d81bc064c37f8d8668cc2cfb) [17:20:32] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [17:22:13] (03CR) 10David Caro: global: minor cleanups (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1186483 (owner: 10David Caro) [17:27:41] (03update) 10dcaro: getBuild - add digest to image [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/144 (https://phabricator.wikimedia.org/T403322) (owner: 10damian) [17:27:50] !log dcaro@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api [17:30:18] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [17:32:19] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18015392766 (https://github.com/cluebotng/component-configs/commits/61a7ceac210077c3d81bc064c37f8d8668cc2cfb) [17:32:20] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [17:37:43] (03update) 10raymond-ndibe: [do_deploy] update_deployment status only after successful jobs creation [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/135 (https://phabricator.wikimedia.org/T405620) [17:37:44] (03approved) 10raymond-ndibe: [do_deploy] update_deployment status only after successful jobs creation [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/135 (https://phabricator.wikimedia.org/T405620) [17:42:13] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [17:44:35] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18015998409 (https://github.com/cluebotng/component-configs/commits/5592cdfcdc7e683a993c8e784d83fb1a71a0b04c) [17:44:36] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [17:44:40] !log tools.cluebotng-staging Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18015998408 (https://github.com/cluebotng/component-configs/commits/5592cdfcdc7e683a993c8e784d83fb1a71a0b04c) [17:44:41] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-staging/SAL [17:46:05] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18015998407 (https://github.com/cluebotng/component-configs/commits/5592cdfcdc7e683a993c8e784d83fb1a71a0b04c) [17:46:06] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [17:47:27] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18016018062 (https://github.com/cluebotng/component-configs/commits/150020d96f0c95173ba88c382221223a0c1f7a8d) [17:47:29] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [17:49:15] (03approved) 10dcaro: jobs-api: bump to 0.0.420-20250925165253-31f66be8 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/980 (https://phabricator.wikimedia.org/T389118 https://phabricator.wikimedia.org/T390136) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:49:23] (03merge) 10dcaro: jobs-api: bump to 0.0.420-20250925165253-31f66be8 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/980 (https://phabricator.wikimedia.org/T389118 https://phabricator.wikimedia.org/T390136) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:50:30] (03merge) 10raymond-ndibe: [do_deploy] update_deployment status only after successful jobs creation [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/135 (https://phabricator.wikimedia.org/T405620) [17:53:18] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: components-api: bump to 0.0.163-20250925175045-84c86f6b [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/981 (https://phabricator.wikimedia.org/T405620) [17:53:19] 10Tool-paulina: Create API call examples to create and edit Wikidata works and authors items. - https://phabricator.wikimedia.org/T405635 (10Pepe_piton) 03NEW [17:53:48] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api [17:54:02] 10Tool-paulina, 10Outreachy (Round 31): Outreachy 31: Features to edit author and work data on Wikidata directly from Paulina - https://phabricator.wikimedia.org/T392429#11215991 (10Pepe_piton) [17:57:02] 10Tool-paulina: Create API call examples to create and edit Wikidata works and authors items. - https://phabricator.wikimedia.org/T405635#11216039 (10Pepe_piton) [17:58:52] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api [17:59:24] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component components-api [18:04:27] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api [18:05:48] (03approved) 10raymond-ndibe: components-api: bump to 0.0.163-20250925175045-84c86f6b [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/981 (https://phabricator.wikimedia.org/T405620) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [18:05:50] (03update) 10raymond-ndibe: components-api: bump to 0.0.163-20250925175045-84c86f6b [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/981 (https://phabricator.wikimedia.org/T405620) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [18:05:56] (03merge) 10raymond-ndibe: components-api: bump to 0.0.163-20250925175045-84c86f6b [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/981 (https://phabricator.wikimedia.org/T405620) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [18:06:51] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: [components-api] deployment is marked `successful` before finishing - https://phabricator.wikimedia.org/T405620#11216246 (10Raymond_Ndibe) 05Open→03Resolved [18:15:33] 06cloud-services-team, 10Toolforge: [jobs-cli] `images` returns images that do not exist - https://phabricator.wikimedia.org/T405642 (10DamianZaremba) 03NEW [18:17:00] 10Toolforge (Quota-requests): Request increased build quota for cluebotng-review Toolforge tool - https://phabricator.wikimedia.org/T405643 (10DamianZaremba) 03NEW [18:19:02] 10Toolforge (Quota-requests): Request increased build quota for cluebotng-monitoring Toolforge tool - https://phabricator.wikimedia.org/T405644 (10DamianZaremba) 03NEW [18:19:30] 10Toolforge (Quota-requests): Request increased build quota for cluebotng Toolforge tool - https://phabricator.wikimedia.org/T405645 (10DamianZaremba) 03NEW [18:25:18] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18016894528 (https://github.com/cluebotng/component-configs/commits/cfc9adc9516df0f11c8b6d1df68232d0a46cb4eb) [18:25:20] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [18:29:39] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18017003184 (https://github.com/cluebotng/component-configs/commits/e9fc8d46ac1a0ff0ac6203458fa171c6430492ce) [18:29:40] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [18:43:18] 10Toolforge (Toolforge iteration 24), 13Patch-For-Review: [components-api] support port protocol in config - https://phabricator.wikimedia.org/T401994#11216684 (10DamianZaremba) What is the timeline for this? I have 2 tools which have 1 job stuck in `jobs.yaml` rather than `components` due to needing UDP port... [18:56:19] (03open) 10swayamagrahari: Add tool stats page [toolforge-repos/isa] - 10https://gitlab.wikimedia.org/toolforge-repos/isa/-/merge_requests/19 [21:38:56] FIRING: ProbeDown: Service tools-k8s-haproxy-5:443 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [21:43:56] FIRING: [2x] ProbeDown: Service tools-k8s-haproxy-5:443 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [21:55:43] FIRING: ToolforgeWebHighErrorRate: High 5xx rate on Toolforge web services #page - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeWebHighErrorRate - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/infra-k8s-haproxy?var-frontend=k8s-ingress-https - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeWebHighErrorRate [21:58:56] FIRING: [2x] ProbeDown: Service tools-k8s-haproxy-5:443 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [22:00:43] RESOLVED: ToolforgeWebHighErrorRate: High 5xx rate on Toolforge web services #page - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeWebHighErrorRate - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/infra-k8s-haproxy?var-frontend=k8s-ingress-https - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeWebHighErrorRate [22:03:56] RESOLVED: [2x] ProbeDown: Service tools-k8s-haproxy-5:443 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [22:04:56] FIRING: ProbeDown: Service tools-k8s-haproxy-5:443 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [22:09:56] RESOLVED: [2x] ProbeDown: Service tools-k8s-haproxy-5:443 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [22:16:56] FIRING: ProbeDown: Service tools-k8s-haproxy-5:443 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [22:21:56] RESOLVED: [2x] ProbeDown: Service tools-k8s-haproxy-5:443 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [23:07:16] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) [23:13:53] 10Tool-gawa: [Documentation] Mise à jour du contenu de la page Méta-Wiki de l'outil GAWA - https://phabricator.wikimedia.org/T405148#11217695 (10poro26) [23:14:41] 10Tool-gawa: [Documentation] Mise à jour du contenu de la page Méta-Wiki de l'outil GAWA - https://phabricator.wikimedia.org/T405148#11217696 (10poro26) 05Open→03Resolved Tâche effectuée : https://meta.wikimedia.org/wiki/G%C3%A9n%C3%A9rateur_d%27Articles_Wikip%C3%A9dia_%C3%A0_Am%C3%A9liorer_(GAWA)