[00:14:26] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/24 [00:14:29] (03open) 10renovatebot: Lock file maintenance [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/24 [00:14:40] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/35 [00:14:43] (03open) 10renovatebot: Lock file maintenance [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/35 [00:14:54] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/31 [00:14:58] (03open) 10renovatebot: Lock file maintenance [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/31 [00:16:46] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/63 [00:16:49] (03open) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/63 [00:17:11] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/84 [00:17:13] (03open) 10renovatebot: Lock file maintenance [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/84 [00:17:59] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/49 [00:18:02] (03open) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/49 [00:18:49] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/45 [00:18:53] (03open) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/45 [00:19:18] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/30 [00:19:21] (03open) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/30 [00:19:32] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/34 [00:19:35] (03open) 10renovatebot: Lock file maintenance [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/34 [00:19:41] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewikisignbot] - 10https://gitlab.wikimedia.org/toolforge-repos/dewikisignbot/-/merge_requests/35 [00:19:45] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewikisignbot] - 10https://gitlab.wikimedia.org/toolforge-repos/dewikisignbot/-/merge_requests/35 [00:19:45] (03open) 10renovatebot: Lock file maintenance [toolforge-repos/dewikisignbot] - 10https://gitlab.wikimedia.org/toolforge-repos/dewikisignbot/-/merge_requests/35 [01:52:12] (03open) 10andrew: maintain-kubeusers: Bump spacemedia envvars quota [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1445 (https://phabricator.wikimedia.org/T439237) [03:38:38] RESOLVED: ToolsDBHistoryLengthGrowing: ToolsDB History Length is above the desired threshold - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBHistoryLengthGrowing - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsDBHistoryLengthGrowing [03:46:08] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.depool_and_destroy (T429387) [03:46:13] T429387: cloudceph HEALTH_WARN, multiple OSD(s) experiencing slow operations in BlueStore - https://phabricator.wikimedia.org/T429387 [03:46:53] 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: cloudceph HEALTH_WARN, multiple OSD(s) experiencing slow operations in BlueStore - https://phabricator.wikimedia.org/T429387#12394141 (10Andrew) Moving 1043 to from 32k to 16k allocation since it doesn't seem much happier than the others. [03:50:24] 10Toolforge (Push-to-Deploy), 06tools-platform-team: Create continuous job deployment tutorial - https://phabricator.wikimedia.org/T437903#12394142 (10komla) 05In progress→03Invalid [07:20:34] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) (T429387) [07:20:40] T429387: cloudceph HEALTH_WARN, multiple OSD(s) experiencing slow operations in BlueStore - https://phabricator.wikimedia.org/T429387 [08:01:17] (03merge) 10dcaro: cli: add --sources option [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/5 (https://phabricator.wikimedia.org/T432572) [08:01:25] (03update) 10dcaro: cli: add --since and --until options [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/6 (https://phabricator.wikimedia.org/T432572) [08:06:02] (03update) 10dcaro: cli: add --since and --until options [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/6 (https://phabricator.wikimedia.org/T432572) [08:06:08] (03update) 10dcaro: cli: add --since and --until options [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/6 (https://phabricator.wikimedia.org/T432572) [08:06:11] (03update) 10dcaro: cli: add --since and --until options [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/6 (https://phabricator.wikimedia.org/T432572) [08:06:27] (03update) 10dcaro: cli: add --lines option [repos/cloud/toolforge/logs-cli] (add_since_until_options) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/7 (https://phabricator.wikimedia.org/T432572) [08:11:45] 10Toolforge, 06tools-platform-team, 07OKR-Work, 13Patch-For-Review: [logs-cli,toolforge-cli] Add subcommand toolforge logs to query and retrieve the relevant logs, will have to design a minimal interface to filter by event/subsystem/time/etc. - https://phabricator.wikimedia.org/T432572#12394409 (10dcaro) [08:12:14] 10Toolforge, 06tools-platform-team, 07OKR-Work: [logs-cli] Add `--direction` flag to choose the direction of the logs shown - https://phabricator.wikimedia.org/T433869#12394412 (10dcaro) [08:12:15] 10Toolforge, 06tools-platform-team, 07OKR-Work, 13Patch-For-Review: [logs-cli,toolforge-cli] Add subcommand toolforge logs to query and retrieve the relevant logs, will have to design a minimal interface to filter by event/subsystem/time/etc. - https://phabricator.wikimedia.org/T432572#12394413 (10dcaro) [08:12:47] 10Toolforge, 06tools-platform-team, 07OKR-Work: [logs-cli] add a `--first` flag, or replace `--latest` with `--lines` - https://phabricator.wikimedia.org/T433870#12394416 (10dcaro) [08:12:48] 10Toolforge, 06tools-platform-team, 07OKR-Work, 13Patch-For-Review: [logs-cli,toolforge-cli] Add subcommand toolforge logs to query and retrieve the relevant logs, will have to design a minimal interface to filter by event/subsystem/time/etc. - https://phabricator.wikimedia.org/T432572#12394415 (10dcaro) [08:13:22] 10Toolforge, 06tools-platform-team, 07OKR-Work: [logs-cli] add a `--first` flag, or replace `--latest` with `--lines` - https://phabricator.wikimedia.org/T433870#12394418 (10dcaro) >>! In T433870#12318902, @dcaro wrote: > We decided for API parity: https://phabricator.wikimedia.org/T432572#12318897 Given th... [08:13:53] 10Toolforge, 06tools-platform-team, 07OKR-Work: [logs-cli] add a `--first` flag, or replace `--latest` with `--lines` - https://phabricator.wikimedia.org/T433870#12394419 (10dcaro) 05Open→03Invalid [08:21:57] 06tools-infrastructure-team: Upgrade cloudcumin hosts to Trixie - https://phabricator.wikimedia.org/T436667#12394445 (10Volans) 05Open→03In progress [08:29:44] (03update) 10dcaro: cli: add --lines option [repos/cloud/toolforge/logs-cli] (add_since_until_options) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/7 (https://phabricator.wikimedia.org/T432572) [08:29:47] (03update) 10dcaro: cli: add --lines option [repos/cloud/toolforge/logs-cli] (add_since_until_options) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/7 (https://phabricator.wikimedia.org/T432572) [08:31:18] (03update) 10dcaro: cli: add the --job-names flag [repos/cloud/toolforge/logs-cli] (add_lines_option) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/10 (https://phabricator.wikimedia.org/T432572) [08:37:20] (03update) 10dcaro: cli: add --deployment-ids flag [repos/cloud/toolforge/logs-cli] (add_jobs_option) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/11 (https://phabricator.wikimedia.org/T432572) [08:37:24] FIRING: SystemdUnitFailed: user@499.service on cloudcumin2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:39:29] (03update) 10dcaro: cli: add --build-ids flag [repos/cloud/toolforge/logs-cli] (add_deployments_option) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/12 (https://phabricator.wikimedia.org/T432572) [08:39:48] (03update) 10dcaro: cli: add the --job-names flag [repos/cloud/toolforge/logs-cli] (add_lines_option) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/10 (https://phabricator.wikimedia.org/T432572) [08:40:14] (03merge) 10dcaro: cli: add warning message for beta status [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/14 (https://phabricator.wikimedia.org/T439581) [08:40:32] (03update) 10dcaro: cli: add --lines option [repos/cloud/toolforge/logs-cli] (add_since_until_options) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/7 (https://phabricator.wikimedia.org/T432572) [08:41:55] (03open) 10dcaro: d/changelog: bump to 0.0.4 [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/15 (https://phabricator.wikimedia.org/T432572 https://phabricator.wikimedia.org/T439581) [08:54:28] FIRING: KeyholderUnarmed: 2 unarmed Keyholder key(s) on cloudcumin2001:9100 - https://wikitech.wikimedia.org/wiki/Keyholder - TODO - https://alerts.wikimedia.org/?q=alertname%3DKeyholderUnarmed [08:57:24] RESOLVED: SystemdUnitFailed: user@499.service on cloudcumin2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:58:36] 10Toolforge, 06tools-platform-team, 07OKR-Work: [components-api] send deployment events to the logs-api system-event log endpoint - https://phabricator.wikimedia.org/T432571#12394553 (10TLepage-WMF) 05Open→03In progress a:05dcaro→03TLepage-WMF [08:59:28] RESOLVED: KeyholderUnarmed: 2 unarmed Keyholder key(s) on cloudcumin2001:9100 - https://wikitech.wikimedia.org/wiki/Keyholder - TODO - https://alerts.wikimedia.org/?q=alertname%3DKeyholderUnarmed [09:07:47] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component logs-cli [09:08:19] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-cli [09:08:31] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component logs-cli [09:09:06] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-cli [09:10:34] (03approved) 10dcaro: d/changelog: bump to 0.0.4 [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/15 (https://phabricator.wikimedia.org/T432572 https://phabricator.wikimedia.org/T439581) [09:10:41] (03merge) 10dcaro: d/changelog: bump to 0.0.4 [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/15 (https://phabricator.wikimedia.org/T432572 https://phabricator.wikimedia.org/T439581) [09:11:42] 10Toolforge, 06tools-platform-team, 07OKR-Work, 13Patch-For-Review: [logs-cli] show a warning message stating it's in beta/unstable - https://phabricator.wikimedia.org/T439581#12394581 (10dcaro) 05In progress→03Resolved [09:12:20] 10Toolforge, 06tools-platform-team: [logs-cli,logs-api] add functional tests for all the new "finished" features - https://phabricator.wikimedia.org/T439884#12394582 (10dcaro) p:05Triage→03Medium [09:12:37] 10Toolforge, 06tools-platform-team: [logs-cli,logs-api] add functional tests for all the new "finished" features - https://phabricator.wikimedia.org/T439884#12394584 (10dcaro) p:05Medium→03High [09:23:56] (03approved) 10fnegri: cli: add --since and --until options [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/6 (https://phabricator.wikimedia.org/T432572) (owner: 10dcaro) [09:24:56] 10Toolforge, 06tools-platform-team, 07OKR-Work: [logs-cli] Add `--direction` flag to choose the direction of the logs shown - https://phabricator.wikimedia.org/T433869#12394606 (10dcaro) 05Open→03In progress a:03dcaro [09:26:32] (03open) 10dcaro: cli: add --direction flag [repos/cloud/toolforge/logs-cli] (add_builds_option) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/16 (https://phabricator.wikimedia.org/T433869) [09:27:11] (03update) 10fnegri: cli: add --lines option [repos/cloud/toolforge/logs-cli] (add_since_until_options) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/7 (https://phabricator.wikimedia.org/T432572) (owner: 10dcaro) [09:27:16] (03approved) 10fnegri: cli: add --lines option [repos/cloud/toolforge/logs-cli] (add_since_until_options) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/7 (https://phabricator.wikimedia.org/T432572) (owner: 10dcaro) [09:35:41] (03update) 10dcaro: cli: add --direction flag [repos/cloud/toolforge/logs-cli] (add_builds_option) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/16 (https://phabricator.wikimedia.org/T433869) [09:36:26] (03merge) 10dcaro: cli: add --since and --until options [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/6 (https://phabricator.wikimedia.org/T432572) [09:36:32] (03update) 10dcaro: cli: add --lines option [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/7 (https://phabricator.wikimedia.org/T432572) [09:37:09] (03update) 10dcaro: cli: add --lines option [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/7 (https://phabricator.wikimedia.org/T432572) [09:37:12] (03update) 10dcaro: cli: add --lines option [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/7 (https://phabricator.wikimedia.org/T432572) [09:38:31] (03merge) 10dcaro: cli: add --lines option [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/7 (https://phabricator.wikimedia.org/T432572) [09:38:33] (03update) 10dcaro: cli: add the --job-names flag [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/10 (https://phabricator.wikimedia.org/T432572) [09:42:58] (03approved) 10volans: eqiad1: cloudinfra: Add security group for WAF (requestctl) nodes [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/365 (https://phabricator.wikimedia.org/T439418) (owner: 10taavi) [09:43:52] 10Toolforge, 06tools-platform-team, 07OKR-Work: [logs-cli] make 'get' the default action if none passed - https://phabricator.wikimedia.org/T440166 (10dcaro) 03NEW [09:44:04] 10Toolforge, 06tools-platform-team, 07OKR-Work: [logs-cli] make 'get' the default action if none passed - https://phabricator.wikimedia.org/T440166#12394721 (10dcaro) p:05Triage→03Medium [09:47:11] (03approved) 10fnegri: cli: add --build-ids flag [repos/cloud/toolforge/logs-cli] (add_deployments_option) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/12 (https://phabricator.wikimedia.org/T432572) (owner: 10dcaro) [09:47:29] (03approved) 10fnegri: cli: add --direction flag [repos/cloud/toolforge/logs-cli] (add_builds_option) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/16 (https://phabricator.wikimedia.org/T433869) (owner: 10dcaro) [09:48:45] 10Toolforge, 06tools-platform-team, 07OKR-Work, 13Patch-For-Review: [logs-cli] Add `--direction` flag to choose the direction of the logs shown - https://phabricator.wikimedia.org/T433869#12394739 (10fnegri) I think adding the `--direction` option is useful, but is not covering a common use case: printing... [09:49:37] (03update) 10dcaro: cli: add the --job-names flag [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/10 (https://phabricator.wikimedia.org/T432572) [09:49:46] (03update) 10dcaro: cli: add the --job-names flag [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/10 (https://phabricator.wikimedia.org/T432572) [09:51:06] (03merge) 10dcaro: cli: add the --job-names flag [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/10 (https://phabricator.wikimedia.org/T432572) [09:51:11] (03update) 10dcaro: cli: add --deployment-ids flag [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/11 (https://phabricator.wikimedia.org/T432572) [09:57:36] 10Toolforge, 06tools-platform-team, 07OKR-Work, 13Patch-For-Review: [logs-cli] Add `--direction` flag to choose the direction of the logs shown - https://phabricator.wikimedia.org/T433869#12394817 (10dcaro) > I think adding the --direction option is useful, but is not covering a common use case: printing t... [09:59:46] (03update) 10dcaro: cli: add --deployment-ids flag [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/11 (https://phabricator.wikimedia.org/T432572) [10:00:26] (03open) 10dcaro: cli: make `get` the default action [repos/cloud/toolforge/logs-cli] (add_direction_option) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/17 (https://phabricator.wikimedia.org/T433869) [10:01:57] (03update) 10dcaro: cli: make `get` the default action [repos/cloud/toolforge/logs-cli] (add_direction_option) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/17 (https://phabricator.wikimedia.org/T433869 https://phabricator.wikimedia.org/T440166) [10:02:00] 10Toolforge, 06tools-platform-team, 07OKR-Work: [logs-cli] make 'get' the default action if none passed - https://phabricator.wikimedia.org/T440166#12394852 (10dcaro) 05Open→03In progress [10:02:20] (03update) 10dcaro: cli: make `get` the default action [repos/cloud/toolforge/logs-cli] (add_direction_option) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/17 (https://phabricator.wikimedia.org/T440166) [10:06:30] (03update) 10dcaro: cli: make `get` the default action [repos/cloud/toolforge/logs-cli] (add_direction_option) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/17 (https://phabricator.wikimedia.org/T440166) [10:07:04] (03approved) 10fnegri: cli: add --deployment-ids flag [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/11 (https://phabricator.wikimedia.org/T432572) (owner: 10dcaro) [10:07:37] (03approved) 10fnegri: cli: make `get` the default action [repos/cloud/toolforge/logs-cli] (add_direction_option) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/17 (https://phabricator.wikimedia.org/T440166) (owner: 10dcaro) [10:10:08] 10Toolforge, 06tools-platform-team, 07OKR-Work, 13Patch-For-Review: [logs-cli] Add `--direction` flag to choose the direction of the logs shown - https://phabricator.wikimedia.org/T433869#12394889 (10fnegri) > this is mostly a UX issue that would fit there. Agreed, this is a UX issue and should not block... [10:13:33] 10Toolforge, 06tools-platform-team, 07OKR-Work, 13Patch-For-Review: [logs-cli] Add `--direction` flag to choose the direction of the logs shown - https://phabricator.wikimedia.org/T433869#12394899 (10fnegri) Oh we already have one, just found it: {T433870} [10:15:43] (03update) 10dcaro: cli: make `get` the default action [repos/cloud/toolforge/logs-cli] (add_direction_option) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/17 (https://phabricator.wikimedia.org/T440166) [10:15:59] (03update) 10dcaro: cli: make `get` the default action [repos/cloud/toolforge/logs-cli] (add_direction_option) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/17 (https://phabricator.wikimedia.org/T440166) [10:16:25] (03merge) 10dcaro: cli: add --deployment-ids flag [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/11 (https://phabricator.wikimedia.org/T432572) [10:16:26] (03update) 10dcaro: cli: add --build-ids flag [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/12 (https://phabricator.wikimedia.org/T432572) [10:16:35] (03update) 10dcaro: cli: add --build-ids flag [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/12 (https://phabricator.wikimedia.org/T432572) [10:17:17] 10Toolforge, 06tools-platform-team, 07OKR-Work: [logs-cli] add a `--first` flag, or replace `--latest` with `--lines` - https://phabricator.wikimedia.org/T433870#12394913 (10fnegri) > there's no task needed anymore I think this would still be a nice feature to have, although it shouldn't block the curre... [10:20:47] (03merge) 10dcaro: cli: add --build-ids flag [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/12 (https://phabricator.wikimedia.org/T432572) [10:20:49] (03update) 10dcaro: cli: add --direction flag [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/16 (https://phabricator.wikimedia.org/T433869) [10:20:58] (03update) 10dcaro: cli: add --direction flag [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/16 (https://phabricator.wikimedia.org/T433869) [10:21:11] (03update) 10dcaro: cli: add --direction flag [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/16 (https://phabricator.wikimedia.org/T433869) [10:22:38] (03merge) 10dcaro: cli: add --direction flag [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/16 (https://phabricator.wikimedia.org/T433869) [10:22:43] (03update) 10dcaro: cli: make `get` the default action [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/17 (https://phabricator.wikimedia.org/T440166) [10:54:42] 10Tool-refill: reFill should be more inclusive in what it fixes - https://phabricator.wikimedia.org/T440175 (10Pigsonthewing) 03NEW [10:58:18] (03update) 10dcaro: cli: make `get` the default action [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/17 (https://phabricator.wikimedia.org/T440166) [10:58:25] (03update) 10dcaro: cli: make `get` the default action [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/17 (https://phabricator.wikimedia.org/T440166) [10:59:51] (03merge) 10dcaro: cli: make `get` the default action [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/17 (https://phabricator.wikimedia.org/T440166) [11:39:21] (03merge) 10samwilson: Use dir=auto for the OCR output text field [toolforge-repos/ocr] - 10https://gitlab.wikimedia.org/toolforge-repos/ocr/-/merge_requests/19 (owner: 10sweil) [11:39:49] (03merge) 10samwilson: Update Guzzle to fix security vulnerabilities [toolforge-repos/ocr] - 10https://gitlab.wikimedia.org/toolforge-repos/ocr/-/merge_requests/20 (owner: 10sweil) [11:40:05] (03merge) 10samwilson: Add robots.txt [toolforge-repos/ocr] - 10https://gitlab.wikimedia.org/toolforge-repos/ocr/-/merge_requests/21 (owner: 10sweil) [11:45:07] FIRING: ToolsDBHistoryLengthGrowing: ToolsDB History Length is above the desired threshold - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBHistoryLengthGrowing - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsDBHistoryLengthGrowing [11:46:51] (03open) 10dcaro: d/changelog: bump to 0.0.5 [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/18 (https://phabricator.wikimedia.org/T432572 https://phabricator.wikimedia.org/T433869 https://phabricator.wikimedia.org/T440166) [11:46:59] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component logs-cli [11:48:30] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-cli [11:48:54] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component logs-cli [11:49:23] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-cli [11:50:07] RESOLVED: ToolsDBHistoryLengthGrowing: ToolsDB History Length is above the desired threshold - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBHistoryLengthGrowing - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsDBHistoryLengthGrowing [12:00:30] 10Toolforge, 06tools-platform-team, 07OKR-Work, 13Patch-For-Review: [logs-cli] make 'get' the default action if none passed - https://phabricator.wikimedia.org/T440166#12395205 (10dcaro) 05In progress→03Resolved [12:00:33] 10Toolforge, 06tools-platform-team, 07OKR-Work, 13Patch-For-Review: [logs-cli,toolforge-cli] Add subcommand toolforge logs to query and retrieve the relevant logs, will have to design a minimal interface to filter by event/subsystem/time/etc. - https://phabricator.wikimedia.org/T432572#12395208 (10dcaro... [12:00:41] 10Toolforge, 06tools-platform-team, 07OKR-Work, 13Patch-For-Review: [logs-cli] Add `--direction` flag to choose the direction of the logs shown - https://phabricator.wikimedia.org/T433869#12395222 (10dcaro) 05In progress→03Resolved [12:05:23] (03open) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/lexeme-forms] - 10https://gitlab.wikimedia.org/toolforge-repos/lexeme-forms/-/merge_requests/51 [12:06:00] (03approved) 10dcaro: d/changelog: bump to 0.0.5 [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/18 (https://phabricator.wikimedia.org/T432572 https://phabricator.wikimedia.org/T433869 https://phabricator.wikimedia.org/T440166) [12:06:09] (03merge) 10dcaro: d/changelog: bump to 0.0.5 [repos/cloud/toolforge/logs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-cli/-/merge_requests/18 (https://phabricator.wikimedia.org/T432572 https://phabricator.wikimedia.org/T433869 https://phabricator.wikimedia.org/T440166) [12:23:02] 06cloud-services-team, 10Quarry, 06tools-platform-team, 06RoadToWiki, and 3 others: "1 rows" should be "1 row" - https://phabricator.wikimedia.org/T419564#12395284 (10dcaro) 05Open→03Resolved This was already deployed \o/ [12:25:33] 10Toolforge, 06tools-platform-team: [all] first of the month dependency upgrade MRs - https://phabricator.wikimedia.org/T440180 (10dcaro) 03NEW [12:25:37] 10Toolforge, 06tools-platform-team: [all] first of the month dependency upgrade MRs - https://phabricator.wikimedia.org/T440180#12395300 (10dcaro) p:05Triage→03High [12:25:44] 10Toolforge, 06tools-platform-team: [all] first of the month dependency upgrade MRs - https://phabricator.wikimedia.org/T440180#12395302 (10dcaro) 05Open→03In progress [12:25:46] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/78 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:25:50] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/78 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:26:32] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/70 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:26:36] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/70 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:26:43] (03update) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/77 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:27:00] (03approved) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/77 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:27:01] (03update) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/77 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:27:08] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/343 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:27:16] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/343 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:27:17] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/95 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:27:24] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/95 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:27:40] (03approved) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/108 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:27:45] (03merge) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/108 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:27:53] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/107 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:27:56] (03update) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/107 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:27:58] (03update) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/107 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:28:16] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/168 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:28:22] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/168 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:28:34] (03merge) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/77 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:28:34] (03approved) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/121 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:28:43] (03merge) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/121 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:28:47] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/79 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:28:55] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/79 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:29:10] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/53 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:29:15] (03update) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/382 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:29:19] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/36 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:29:20] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/53 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:29:28] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/36 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:29:38] (03approved) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/154 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:29:54] (03merge) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/154 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:29:55] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: builds-builder: bump to 0.0.154-20261005122738-4cc80ba9 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1446 [12:30:02] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/120 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:30:14] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: builds-builder: bump to 0.0.154-20261005122738-4cc80ba9 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1446 [12:30:21] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: maintain-harbor: bump to 0.0.75-20261005122650-9b0ddd06 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1447 [12:30:38] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: maintain-harbor: bump to 0.0.75-20261005122650-9b0ddd06 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1447 [12:30:45] (03update) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/120 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:30:54] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/120 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:31:06] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/107 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:31:21] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: api-gateway: bump to 0.0.108-20261005122800-4a69b40f [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1448 [12:31:29] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: api-gateway: bump to 0.0.108-20261005122800-4a69b40f [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1448 [12:31:47] (03approved) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/62 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:31:50] (03merge) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/62 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:31:57] (03update) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/381 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:32:18] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/122 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:32:28] (03approved) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/197 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:32:29] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/122 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:32:35] (03merge) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/197 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:32:50] (03update) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/153 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:33:00] (03update) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/61 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:33:02] (03update) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/61 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:33:13] (03approved) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/101 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:33:21] (03merge) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/101 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:33:24] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/44 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:33:31] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/44 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:33:38] (03approved) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/49 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:33:47] (03merge) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/49 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:33:51] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/44 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:34:02] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/44 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:34:04] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/94 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:34:13] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/94 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:34:14] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/183 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:34:21] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/183 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:34:52] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: api-gateway: bump to 0.0.109-20261005123136-5bb89794 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1448 [12:34:59] (03update) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/153 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:35:00] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/misctools-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/misctools-cli/-/merge_requests/25 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:35:12] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/misctools-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/misctools-cli/-/merge_requests/25 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:35:16] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/153 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:35:28] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/196 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:35:37] (03update) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/196 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:35:42] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/153 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:35:48] (03update) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/196 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:35:57] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: api-gateway: bump to 0.0.109-20261005123136-5bb89794 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1448 [12:35:57] (03update) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/100 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:36:07] (03update) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/48 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:36:19] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/61 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:36:23] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-emailer: bump to 0.0.96-20261005123203-a8a31f7c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1449 [12:36:30] (03update) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/381 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:36:34] (03approved) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/382 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:36:41] (03merge) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/382 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:36:45] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/381 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:36:50] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-builder [12:36:58] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-emailer: bump to 0.0.96-20261005123203-a8a31f7c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1449 [12:37:28] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builder [12:37:30] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: volume-admission: bump to 0.0.89-20261005123003-652db2ba [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1450 [12:38:01] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: components-api: bump to 0.0.220-20261005123256-d7da109d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1451 [12:38:07] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: volume-admission: bump to 0.0.89-20261005123003-652db2ba [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1450 [12:38:28] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: components-api: bump to 0.0.220-20261005123256-d7da109d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1451 [12:39:04] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: maintain-kubeusers: bump to 0.0.203-20261005123458-89615cdc [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1452 [12:39:16] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: envvars-api: bump to 0.0.90-20261005122930-8b672ed5 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1453 [12:39:21] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: maintain-kubeusers: bump to 0.0.203-20261005123458-89615cdc [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1452 [12:39:34] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: envvars-api: bump to 0.0.90-20261005122930-8b672ed5 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1453 [12:40:12] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: registry-admission: bump to 0.0.80-20261005123439-5be87520 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1454 [12:40:18] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: registry-admission: bump to 0.0.80-20261005123439-5be87520 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1454 [12:42:07] FIRING: ToolsDBHistoryLengthGrowing: ToolsDB History Length is above the desired threshold - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBHistoryLengthGrowing - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsDBHistoryLengthGrowing [12:45:00] (03update) 10andrew: maintain-kubeusers: Bump spacemedia envvars quota [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1445 (https://phabricator.wikimedia.org/T439237) [12:45:43] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-builder [12:47:16] (03PS1) 10Brouberol: Add fixtures for the codfw cephosd admin [labs/private] - 10https://gerrit.wikimedia.org/r/1351299 [12:47:29] (03CR) 10Brouberol: [C:03+2] Add fixtures for the codfw cephosd admin [labs/private] - 10https://gerrit.wikimedia.org/r/1351299 (owner: 10Brouberol) [12:47:33] (03CR) 10Brouberol: [V:03+2 C:03+2] Add fixtures for the codfw cephosd admin [labs/private] - 10https://gerrit.wikimedia.org/r/1351299 (owner: 10Brouberol) [12:48:37] 10Toolforge, 06tools-platform-team: [toolsbeta,harbor] Got out of quota for CI images (toolforge project) - https://phabricator.wikimedia.org/T440182 (10dcaro) 03NEW [12:50:00] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder [12:51:10] 10Toolforge (Quota-requests): Request increased quota for mna Toolforge tool - https://phabricator.wikimedia.org/T439975#12395391 (10dcaro) +1 [12:51:19] 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops: Set cloudvirt fleet to 'performance' governor - https://phabricator.wikimedia.org/T439729#12395392 (10fgiunchedi) [12:51:53] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component builds-builder [12:56:01] 10Toolforge, 06tools-platform-team: [toolsbeta,harbor] Got out of quota for CI images (toolforge project) - https://phabricator.wikimedia.org/T440182#12395431 (10dcaro) p:05Triage→03High [12:56:18] 10Toolforge, 06tools-platform-team: [toolsbeta,harbor] Got out of quota for CI images (toolforge project) - https://phabricator.wikimedia.org/T440182#12395434 (10dcaro) The cleanup should have been done by maintain-harbor, looking [12:56:40] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder [12:58:29] (03approved) 10dcaro: builds-builder: bump to 0.0.154-20261005122738-4cc80ba9 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1446 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:58:32] (03merge) 10dcaro: builds-builder: bump to 0.0.154-20261005122738-4cc80ba9 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1446 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:58:47] (03update) 10dcaro: registry-admission: bump to 0.0.80-20261005123439-5be87520 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1454 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:58:58] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component registry-admission [13:03:15] 10Toolforge, 06tools-platform-team: [toolsbeta,harbor] Got out of quota for CI images (toolforge project) - https://phabricator.wikimedia.org/T440182#12395451 (10dcaro) Something weird is going on, harbor reports that there's repositories with >200 artifacts, with many with >40 {F106434520} But when running... [13:05:21] 10Toolforge, 06tools-platform-team: [toolsbeta,harbor] Got out of quota for CI images (toolforge project) - https://phabricator.wikimedia.org/T440182#12395462 (10dcaro) Running in in toolsbeta freed some space, ~2GB (now we have 68.44GiB used), still not quite enough. [13:06:36] (03PS1) 10Brouberol: Add fixtures for the codfw cephosd admin [labs/private] - 10https://gerrit.wikimedia.org/r/1351305 [13:06:52] (03CR) 10Brouberol: [C:03+2] Add fixtures for the codfw cephosd admin [labs/private] - 10https://gerrit.wikimedia.org/r/1351305 (owner: 10Brouberol) [13:06:54] (03CR) 10Brouberol: [V:03+2 C:03+2] Add fixtures for the codfw cephosd admin [labs/private] - 10https://gerrit.wikimedia.org/r/1351305 (owner: 10Brouberol) [13:07:26] (03open) 10dcaro: ci: delete outdated run_local_ci.sh [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/71 [13:07:42] (03approved) 10dcaro: ci: delete outdated run_local_ci.sh [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/71 [13:07:44] (03update) 10dcaro: ci: delete outdated run_local_ci.sh [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/71 [13:09:37] (03merge) 10dcaro: ci: delete outdated run_local_ci.sh [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/71 [13:10:41] (03PS1) 10Andrew Bogott: create_project: encourage -db naming for trove-only projects [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1351309 [13:11:13] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission [13:11:41] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: maintain-harbor: bump to 0.0.76-20261005130951-59c1e0e2 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1447 [13:11:50] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: maintain-harbor: bump to 0.0.76-20261005130951-59c1e0e2 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1447 [13:13:24] PROBLEM - Host cloudcephosd1043 is DOWN: PING CRITICAL - Packet loss = 100% [13:13:48] (03CR) 10CI reject: [V:04-1] create_project: encourage -db naming for trove-only projects [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1351309 (owner: 10Andrew Bogott) [13:14:27] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component registry-admission [13:15:43] 10Toolforge, 06tools-platform-team: [maintain-harbor] package with poetry/etc. like the rest - https://phabricator.wikimedia.org/T440184 (10dcaro) 03NEW [13:15:54] RECOVERY - Host cloudcephosd1043 is UP: PING OK - Packet loss = 0%, RTA = 0.46 ms [13:16:05] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add [13:17:55] (03PS2) 10Andrew Bogott: create_project: encourage -db naming for trove-only projects [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1351309 [13:21:46] 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops: Set cloudvirt fleet to 'performance' governor - https://phabricator.wikimedia.org/T439729#12395552 (10fgiunchedi) Preliminary results, C8 added about 500w to host-reported power consumption (https://w.wiki/Wmhh) {F106437854} [13:23:09] FIRING: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [13:24:24] 10Cloud-VPS (Project-requests), 06tools-infrastructure-team: Request creation of mna-db VPS project - https://phabricator.wikimedia.org/T440186 (10Andrew) 03NEW [13:27:24] FIRING: SystemdUnitFailed: ceph-osd@94.service on cloudcephosd1043:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:28:24] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission [13:35:02] !log andrew@cloudcumin1001 mna-database START - Cookbook wmcs.vps.create_project for trove-only project mna-database in eqiad1 [13:35:04] andrew@cloudcumin1001: Unknown project "mna-database" [13:35:06] !log andrew@cloudcumin1001 mna-database END (FAIL) - Cookbook wmcs.vps.create_project (exit_code=99) for trove-only project mna-database in eqiad1 [13:35:06] andrew@cloudcumin1001: Unknown project "mna-database" [13:35:12] !log andrew@cloudcumin1001 mna-db START - Cookbook wmcs.vps.create_project for trove-only project mna-db in eqiad1 [13:35:12] andrew@cloudcumin1001: Unknown project "mna-db" [13:36:00] (03update) 10group_199_bot_f98be072172e323ae6d1441939d3e461: projects: added project mna-db [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/366 [13:36:00] (03open) 10group_199_bot_f98be072172e323ae6d1441939d3e461: projects: added project mna-db [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/366 [13:37:24] RESOLVED: SystemdUnitFailed: ceph-osd@94.service on cloudcephosd1043:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:37:45] (03merge) 10andrew: projects: added project mna-db [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/366 (owner: 10group_199_bot_f98be072172e323ae6d1441939d3e461) [13:38:13] (03merge) 10dcaro: registry-admission: bump to 0.0.80-20261005123439-5be87520 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1454 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [13:38:33] (03update) 10dcaro: envvars-api: bump to 0.0.90-20261005122930-8b672ed5 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1453 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [13:41:27] !log andrew@cloudcumin1001 mna-db END (PASS) - Cookbook wmcs.vps.create_project (exit_code=0) for trove-only project mna-db in eqiad1 [13:48:45] 10Toolforge (Quota-requests): Request increased quota for mna Toolforge tool - https://phabricator.wikimedia.org/T439975#12395706 (10Andrew) 05Open→03Resolved a:03Andrew You should be all set. You can access/configure things via https://horizon.wikimedia.org/ [13:51:42] FIRING: [9x] AlertLintProblem: Linting problems found for BackupCollectorError - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [13:55:43] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) [13:56:26] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.depool_and_destroy (T429387) [13:56:32] T429387: cloudceph HEALTH_WARN, multiple OSD(s) experiencing slow operations in BlueStore - https://phabricator.wikimedia.org/T429387 [13:56:35] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) (T429387) [13:56:42] FIRING: [18x] AlertLintProblem: Linting problems found for BackupCollectorError - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [13:56:56] (03CR) 10David Caro: [C:03+1] "LGTM" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1351309 (owner: 10Andrew Bogott) [13:56:58] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.depool_and_destroy (T429387) [14:00:56] 10Toolforge, 06tools-platform-team, 07OKR-Work: [logs-cli] add a `--first` flag, or replace `--latest` with `--lines` - https://phabricator.wikimedia.org/T433870#12395774 (10dcaro) >>! In T433870#12394913, @fnegri wrote: >> there's no task needed anymore > > I think this would still be a nice feature to... [14:06:18] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component envvars-api [14:07:18] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api [14:13:11] (03CR) 10Andrew Bogott: [C:03+2] create_project: encourage -db naming for trove-only projects [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1351309 (owner: 10Andrew Bogott) [14:16:10] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component envvars-api [14:16:39] RESOLVED: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [14:16:41] (03Merged) 10jenkins-bot: create_project: encourage -db naming for trove-only projects [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1351309 (owner: 10Andrew Bogott) [14:17:24] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api [14:24:29] (03approved) 10dcaro: envvars-api: bump to 0.0.90-20261005122930-8b672ed5 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1453 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:24:35] (03merge) 10dcaro: envvars-api: bump to 0.0.90-20261005122930-8b672ed5 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1453 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:25:20] (03update) 10dcaro: maintain-kubeusers: bump to 0.0.203-20261005123458-89615cdc [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1452 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:26:42] FIRING: [18x] AlertLintProblem: Linting problems found for BackupCollectorError - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [14:27:25] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) (T429387) [14:27:32] T429387: cloudceph HEALTH_WARN, multiple OSD(s) experiencing slow operations in BlueStore - https://phabricator.wikimedia.org/T429387 [14:31:42] RESOLVED: [18x] AlertLintProblem: Linting problems found for BackupCollectorError - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [14:46:05] (03update) 10dcaro: maintain-kubeusers: Bump spacemedia envvars quota [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1445 (https://phabricator.wikimedia.org/T439237) (owner: 10andrew) [14:46:31] (03approved) 10dcaro: maintain-kubeusers: Bump spacemedia envvars quota [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1445 (https://phabricator.wikimedia.org/T439237) (owner: 10andrew) [14:52:23] (03merge) 10andrew: maintain-kubeusers: Bump spacemedia envvars quota [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1445 (https://phabricator.wikimedia.org/T439237) [14:52:40] (03update) 10dcaro: maintain-kubeusers: bump to 0.0.203-20261005123458-89615cdc [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1452 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:54:18] !log andrew@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers (T439237) [14:54:23] T439237: Request increased quota for spacemedia Toolforge tool - https://phabricator.wikimedia.org/T439237 [14:55:14] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 07Sustainability (Incident Followup): Add monitoring for backy2 openstack backups - https://phabricator.wikimedia.org/T428893#12396107 (10fgiunchedi) Alerts merged, next up is dashboards and runbooks [15:01:03] 10Cloud-VPS (Project-requests), 06tools-infrastructure-team: Request creation of mna-db VPS project - https://phabricator.wikimedia.org/T440186#12396186 (10Andrew) 05Open→03Resolved a:03Andrew [15:03:05] !log andrew@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers (T439237) [15:03:10] T439237: Request increased quota for spacemedia Toolforge tool - https://phabricator.wikimedia.org/T439237 [15:09:48] (03PS1) 10Muehlenhoff: Remove obsolete stub keytabs for apt/install [labs/private] - 10https://gerrit.wikimedia.org/r/1351370 (https://phabricator.wikimedia.org/T439522) [15:10:35] (03CR) 10Muehlenhoff: [V:03+2 C:03+2] Remove obsolete stub keytabs for apt/install [labs/private] - 10https://gerrit.wikimedia.org/r/1351370 (https://phabricator.wikimedia.org/T439522) (owner: 10Muehlenhoff) [15:11:58] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.depool_and_destroy (T429387) [15:11:59] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) (T429387) [15:12:04] T429387: cloudceph HEALTH_WARN, multiple OSD(s) experiencing slow operations in BlueStore - https://phabricator.wikimedia.org/T429387 [15:13:29] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add [15:17:25] PROBLEM - Host cloudcephosd1043 is DOWN: PING CRITICAL - Packet loss = 100% [15:19:54] RECOVERY - Host cloudcephosd1043 is UP: PING OK - Packet loss = 0%, RTA = 0.25 ms [15:21:09] FIRING: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [15:22:42] andrew@cloudcumin1001 bootstrap_and_add (PID 2796394) is awaiting input [15:33:10] 06cloud-services-team (Hardware), 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops, and 2 others: Ensure cloudvirt capacity is more evenly spread out among racks - https://phabricator.wikimedia.org/T424658#12396365 (10fgiunchedi) 05Resolved→03Open reopening, we'll be rebalancing hosts slated for deco... [16:03:33] 10Toolforge, 06tools-platform-team: [maintain-harbor] package with poetry/etc. like the rest - https://phabricator.wikimedia.org/T440184#12396513 (10dcaro) p:05Triage→03Medium [16:03:39] 10Toolforge, 06tools-platform-team: [maintain-harbor] package with poetry/etc. like the rest - https://phabricator.wikimedia.org/T440184#12396515 (10dcaro) 05Open→03In progress [16:07:02] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) [16:17:39] RESOLVED: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [16:20:12] 10Toolforge, 06tools-platform-team, 07Epic, 07OKR-Work: [hypothesis] ST5.4.1 Extend logging capabilities - https://phabricator.wikimedia.org/T432564#12396615 (10dcaro) Weekly asana report: ### Summary * Hypothesis text: ** If we make a subset of platform and system-level events available to tool maintai... [16:31:24] FIRING: [2x] SystemdUnitFailed: hdfs_rsync_mediawiki_content_history.service on clouddumps1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [16:37:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [16:42:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [16:42:51] 10Toolforge, 06tools-platform-team: toolforge jobs logs -f has an Internal server error - https://phabricator.wikimedia.org/T440103#12396740 (10dcaro) 05Open→03In progress p:05Triage→03High a:03dcaro The error on logs-api side is: ` INFO: 127.0.0.1:44404 - "GET /v1/tool/nervousbot/job/nerv... [16:43:48] 10Toolforge, 06tools-platform-team: toolforge jobs logs -f has an Internal server error - https://phabricator.wikimedia.org/T440103#12396748 (10dcaro) And in lima-kilo: ` local.tf-test2@toolslocal:~$ toolforge jobs run --image python3.13 --no-filelog --command date testoneoff1 local.tf-test2@toolslocal:... [16:49:27] 10Toolforge, 06tools-platform-team: toolforge jobs logs -f has an Internal server error - https://phabricator.wikimedia.org/T440103#12396760 (10dcaro) Version tools-harbor.wmcloud.org/toolforge/logs-api:image-0.0.48-20260911084153-e8e7be4a seems to work on lima-kilo, looking [16:52:40] 10Toolforge, 06tools-platform-team: toolforge jobs logs -f has an Internal server error - https://phabricator.wikimedia.org/T440103#12396764 (10dcaro) >>! In T440103#12396760, @dcaro wrote: > Version tools-harbor.wmcloud.org/toolforge/logs-api:image-0.0.48-20260911084153-e8e7be4a seems to work on lima-k... [16:57:29] !log andrew@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers (T439237) [16:57:34] T439237: Request increased quota for spacemedia Toolforge tool - https://phabricator.wikimedia.org/T439237 [17:19:24] !log andrew@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers (T439237) [17:19:29] T439237: Request increased quota for spacemedia Toolforge tool - https://phabricator.wikimedia.org/T439237 [17:21:11] 10Tool-spacemedia, 10Toolforge (Quota-requests): Request increased quota for spacemedia Toolforge tool - https://phabricator.wikimedia.org/T439237#12396933 (10Andrew) 05Open→03Resolved a:03Andrew Done. Please re-open if you still hit a limit at 60. [17:25:43] 10Toolforge, 06tools-platform-team: toolforge jobs logs -f has an Internal server error - https://phabricator.wikimedia.org/T440103#12396946 (10dcaro) The issue is the limit, we are sending 'limit=0' to loki webservice point and it does not like it, running this works: ` local.tf-test2@toolslocal:~$ too... [17:31:06] (03open) 10dcaro: loki: sending lines=0 to the tail endpoint crashes [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/50 (https://phabricator.wikimedia.org/T440103) [17:42:03] (03approved) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/lexeme-forms] - 10https://gitlab.wikimedia.org/toolforge-repos/lexeme-forms/-/merge_requests/51 (owner: 10l10n-bot) [17:42:11] (03merge) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/lexeme-forms] - 10https://gitlab.wikimedia.org/toolforge-repos/lexeme-forms/-/merge_requests/51 (owner: 10l10n-bot) [17:43:51] (03approved) 10dcaro: loki: sending lines=0 to the tail endpoint crashes [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/50 (https://phabricator.wikimedia.org/T440103) [17:43:57] (03merge) 10dcaro: loki: sending lines=0 to the tail endpoint crashes [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/50 (https://phabricator.wikimedia.org/T440103) [17:46:56] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: logs-api: bump to 0.0.58-20261005174410-1e88478d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1455 (https://phabricator.wikimedia.org/T440103) [17:46:57] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: logs-api: bump to 0.0.58-20261005174410-1e88478d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1455 (https://phabricator.wikimedia.org/T440103) [17:49:48] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component logs-api [17:50:27] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops: cloudvirt1071 crash, again - https://phabricator.wikimedia.org/T431374#12397040 (10Andrew) *bump* [17:50:36] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logs-api [17:51:05] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops: cloudvirt1071 crash, again - https://phabricator.wikimedia.org/T431374#12397041 (10Andrew) p:05Medium→03High [17:51:59] 06cloud-services-team, 10PAWS, 06tools-platform-team: Should we ban use of 'freeroot' on PAWS? - https://phabricator.wikimedia.org/T431340#12397043 (10Andrew) 05Open→03Declined I /think/ this is covered by the 'minesweeper' process which was accidentally not running when I opened this. [17:52:24] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team: Should we care about the efficiency of wmcs.openstack.cloudvirt.safe_reboot? - https://phabricator.wikimedia.org/T431337#12397047 (10Andrew) p:05Triage→03Low [17:52:35] 10Toolforge, 06tools-platform-team: [jobs-api,functional-test] Add a test for `--follow` - https://phabricator.wikimedia.org/T440222 (10dcaro) 03NEW [17:53:59] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component logs-api [17:54:18] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logs-api [17:54:50] 06cloud-services-team, 10PAWS, 06tools-platform-team: Should we ban use of 'freeroot' on PAWS? - https://phabricator.wikimedia.org/T431340#12397080 (10dcaro) >>! In T431340#12397043, @Andrew wrote: > I /think/ this is covered by the 'minesweeper' process which was accidentally not running when I opened t... [17:55:49] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component logs-api [18:00:03] 10Toolforge, 06tools-platform-team: [functional-tests,deploy] Lately got ~50% errors due to not finding the branch - https://phabricator.wikimedia.org/T440224 (10dcaro) 03NEW [18:01:09] (03open) 10dcaro: run_functional_tests: add some verbose output [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1456 (https://phabricator.wikimedia.org/T440224) [18:02:54] (03update) 10dcaro: run_functional_tests: add some verbose output [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1456 (https://phabricator.wikimedia.org/T440224) [18:03:04] (03approved) 10dcaro: run_functional_tests: add some verbose output [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1456 (https://phabricator.wikimedia.org/T440224) [18:03:04] (03update) 10dcaro: run_functional_tests: add some verbose output [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1456 (https://phabricator.wikimedia.org/T440224) [18:03:29] (03merge) 10dcaro: run_functional_tests: add some verbose output [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1456 (https://phabricator.wikimedia.org/T440224) [18:03:59] (03update) 10dcaro: logs-api: bump to 0.0.58-20261005174410-1e88478d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1455 (https://phabricator.wikimedia.org/T440103) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [18:06:10] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api [18:06:33] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component logs-api [18:13:39] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: toolforge jobs logs -f has an Internal server error - https://phabricator.wikimedia.org/T440103#12397189 (10dcaro) This has been fixed and deployed now :), let me know if you still find issues [18:13:47] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: toolforge jobs logs -f has an Internal server error - https://phabricator.wikimedia.org/T440103#12397192 (10dcaro) 05In progress→03Resolved [18:14:46] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [functional-tests,deploy] Lately got ~50% errors due to not finding the branch - https://phabricator.wikimedia.org/T440224#12397195 (10dcaro) I have enabled `set -x` right before the failing command, that should show some extra logs next time it happens. [18:18:14] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api [18:20:53] (03approved) 10dcaro: logs-api: bump to 0.0.58-20261005174410-1e88478d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1455 (https://phabricator.wikimedia.org/T440103) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [18:20:59] (03merge) 10dcaro: logs-api: bump to 0.0.58-20261005174410-1e88478d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1455 (https://phabricator.wikimedia.org/T440103) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [18:21:00] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team: Trove DB offline - https://phabricator.wikimedia.org/T430503#12397228 (10Andrew) 05Open→03Resolved a:03Andrew I've rebuilt this with a more modern base image and it looks like it's up and healthy now. Current disk usage for the db vo... [18:21:15] (03open) 10dcaro: global: adopt shared standards [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/72 (https://phabricator.wikimedia.org/T440184) [18:23:14] (03update) 10dcaro: global: adopt shared standards [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/72 (https://phabricator.wikimedia.org/T440184) [18:26:45] 10Tool-delintbot: Fix all sorts of bogus-image-options errors - https://phabricator.wikimedia.org/T421530#12397272 (10Redmin) [18:27:15] 10Tool-delintbot: Fix all sorts of bogus-image-options errors - https://phabricator.wikimedia.org/T421530#12397273 (10Redmin) [18:32:32] (03update) 10dcaro: global: adopt shared standards [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/72 (https://phabricator.wikimedia.org/T440184) [18:32:48] 10Tool-delintbot: Fix all sorts of bogus-image-options errors - https://phabricator.wikimedia.org/T421530#12397291 (10Redmin) [18:38:08] 10Tool-delintbot: Make lint category an optional argument - https://phabricator.wikimedia.org/T416792#12397311 (10Redmin) 05Open→03Declined When fixing bogus image options, removing extraneous colons, etc, the checks put in place in T415882 and T420294 can block legitimate edits. To avoid that problem, t... [18:41:59] (03update) 10dcaro: global: adopt shared standards [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/72 (https://phabricator.wikimedia.org/T440184) [18:44:36] (03update) 10dcaro: global: adopt shared standards [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/72 (https://phabricator.wikimedia.org/T440184) [18:53:07] (03open) 10nichesports: Reordered ainb_cases.py function order to improve readability [toolforge-repos/nichesportsbot] - 10https://gitlab.wikimedia.org/toolforge-repos/nichesportsbot/-/merge_requests/1 [18:54:07] (03merge) 10nichesports: Reordered ainb_cases.py function order to improve readability [toolforge-repos/nichesportsbot] - 10https://gitlab.wikimedia.org/toolforge-repos/nichesportsbot/-/merge_requests/1 [19:03:04] (03update) 10dcaro: global: adopt shared standards [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/72 (https://phabricator.wikimedia.org/T440184) [19:23:48] 10Cloud-VPS, 06tools-infrastructure-team, 06DC-Ops: Set cloudvirt fleet to 'performance' governor - https://phabricator.wikimedia.org/T439729#12397559 (10wiki_willy) Thanks for the heads up @fgiunchedi! If this helps performance and optimization, a half kW increase per cabinet isn't bad at all. Thanks for... [19:25:09] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team: Should we care about the efficiency of wmcs.openstack.cloudvirt.safe_reboot? - https://phabricator.wikimedia.org/T431337#12397561 (10Volans) For point 2 you can do it also without a full empty host on standby. Just start from the host with less... [19:55:37] 10Tool-itwiki: Improve OrphanizerBot to handle disambiguation pages differently - https://phabricator.wikimedia.org/T320693#12397728 (10valerio.bozzolan) a:05valerio.bozzolan→03None Up for grabs again. I've done T439982 T439978 T440130 T440145 T440111 and a private git lesson to Catania students, but not thi... [19:56:04] 10Tool-itwiki: If the section "Voci correlate" becomes empty, remove that section - https://phabricator.wikimedia.org/T338084#12397740 (10valerio.bozzolan) a:05valerio.bozzolan→03None Up for grabs again. I've done T439982 T439978 T440130 T440145 T440111 and a private git lesson to Catania students, but not t... [20:31:24] FIRING: [2x] SystemdUnitFailed: hdfs_rsync_mediawiki_content_history.service on clouddumps1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [20:47:15] 10Toolforge, 06tools-infrastructure-team, 06tools-platform-team: How long should archived toolforge tools last? - https://phabricator.wikimedia.org/T439486#12397997 (10Andrew) Using commands like ` find . -type f -newermt "2026-01-01" ! -newermt "2026-06-01" | xargs du -chs ` I get... ` # six month interv... [21:00:15] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for service: project,magnum [21:00:55] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for service: project,magnum [22:24:34] 10VPS-project-Wikistats: upgrade wikistats cloud VPS project to trixie - https://phabricator.wikimedia.org/T401859#12398234 (10Dzahn) 05In progress→03Resolved - detached volume "backup" from instance wikistats-bookworm - deleted volume backup - renamed volume backup-trixie to backup - deleted instance wi... [23:08:41] FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [23:11:09] 10Toolforge (Push-to-Deploy), 06tools-platform-team, 13Patch-For-Review: [components-api] pick component model based on component-type field, not data shape - https://phabricator.wikimedia.org/T439496#12398361 (10Raymond_Ndibe) >>! In T439496#12386077, @fnegri wrote: > One alternative that we discussed today... [23:14:24] 10Toolforge (Push-to-Deploy), 06tools-platform-team, 13Patch-For-Review: [components-api] pick component model based on component-type field, not data shape - https://phabricator.wikimedia.org/T439496#12398364 (10Raymond_Ndibe) [23:15:27] 10Toolforge (Push-to-Deploy), 06tools-platform-team, 13Patch-For-Review: [components-api] pick component model based on component-type field, not data shape - https://phabricator.wikimedia.org/T439496#12398365 (10Raymond_Ndibe) [23:18:41] RESOLVED: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks