[00:51:06] (03CR) 10MusikAnimal: [C:03+2] Repository: Make executeQueryBuilder apply the timeout and the breaker [labs/xtools] - 10https://gerrit.wikimedia.org/r/1344066 (https://phabricator.wikimedia.org/T437830) (owner: 10MusikAnimal) [00:51:26] (03PS4) 10MusikAnimal: Repository: Route links-table queries to their own replica section [labs/xtools] - 10https://gerrit.wikimedia.org/r/1344067 (https://phabricator.wikimedia.org/T437830) [00:51:26] (03PS4) 10MusikAnimal: PageInfo, Page: Read links tables from the links connection [labs/xtools] - 10https://gerrit.wikimedia.org/r/1344068 (https://phabricator.wikimedia.org/T437830) [00:51:26] (03PS4) 10MusikAnimal: CategoryEdits: Intersect categories and revisions in code [labs/xtools] - 10https://gerrit.wikimedia.org/r/1344069 (https://phabricator.wikimedia.org/T437830) [00:51:41] (03CR) 10MusikAnimal: Repository: Route links-table queries to their own replica section (031 comment) [labs/xtools] - 10https://gerrit.wikimedia.org/r/1344067 (https://phabricator.wikimedia.org/T437830) (owner: 10MusikAnimal) [00:51:49] (03Merged) 10jenkins-bot: Repository: Make executeQueryBuilder apply the timeout and the breaker [labs/xtools] - 10https://gerrit.wikimedia.org/r/1344066 (https://phabricator.wikimedia.org/T437830) (owner: 10MusikAnimal) [00:52:05] (03PS5) 10MusikAnimal: Repository: Route links-table queries to their own replica section [labs/xtools] - 10https://gerrit.wikimedia.org/r/1344067 (https://phabricator.wikimedia.org/T437830) [00:52:10] (03PS5) 10MusikAnimal: PageInfo, Page: Read links tables from the links connection [labs/xtools] - 10https://gerrit.wikimedia.org/r/1344068 (https://phabricator.wikimedia.org/T437830) [00:52:13] (03PS5) 10MusikAnimal: CategoryEdits: Intersect categories and revisions in code [labs/xtools] - 10https://gerrit.wikimedia.org/r/1344069 (https://phabricator.wikimedia.org/T437830) [01:20:24] FIRING: SystemdUnitFailed: backup_cinder_volumes.service on cloudbackup2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [01:35:24] RESOLVED: SystemdUnitFailed: backup_cinder_volumes.service on cloudbackup2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [01:56:33] 10Tool-wmf-openapi-linter, 06MediaWiki-API-Platform-Team, 10Wikidata, 13Patch-For-Review, 06Wikidata-Omega (Triaging): Unable to upgrade Wikibase to api-testing 1.8.0 - `fs.readFileSync is not a function` - https://phabricator.wikimedia.org/T438947#12357793 (10aaron) p:05Triage→03High [01:57:42] 10Tool-wmf-openapi-linter, 10Wikidata, 06MediaWiki-API-Platform-Team (MediaWiki-Interfaces-Sprints), 13Patch-For-Review, 06Wikidata-Omega (Triaging): Unable to upgrade Wikibase to api-testing 1.8.0 - `fs.readFileSync is not a function` - https://phabricator.wikimedia.org/T438947#12357797 (10aaron) [03:30:06] 10Toolforge (Push-to-Deploy), 06tools-platform-team, 07Documentation: [phase 1]Write tutorial for migrating build service jobs to push-to-deploy - https://phabricator.wikimedia.org/T437477#12357871 (10komla) Moved here: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Migrate_build_service_job_to_push_to_d... [05:37:32] 06tools-infrastructure-team, 07Epic, 06Test Platform (Epics): Explore if there are ways to tune the underlying hardware and configuration for the CI agents - https://phabricator.wikimedia.org/T437407#12357921 (10Peter) @fgiunchedi sounds great! Let me know how I can help out. > think it makes sense to test... [06:32:31] 10Cloud-Services: Unlink SUL account "TanvirSdq" from stale developer account on toolsadmin (Striker) - https://phabricator.wikimedia.org/T438120#12358230 (10TanvirSdq) Hi WMCS Team, To clarify the exact issue with my toolsadmin (Striker) profile settings: 1. My current Developer Account Username is: TanvirSdq... [06:36:13] 10Toolforge, 06tools-platform-team: Unlink SUL account "TanvirSdq" from stale developer account on toolsadmin (Striker) - https://phabricator.wikimedia.org/T438120#12358246 (10TanvirSdq) p:05Triage→03High [06:43:26] 06tools-infrastructure-team, 07Epic, 06Test Platform (Epics): Explore if there are ways to tune the underlying hardware and configuration for the CI agents - https://phabricator.wikimedia.org/T437407#12358265 (10fgiunchedi) >>! In T437407#12357921, @Peter wrote: > @fgiunchedi sounds great! Let me know how I... [06:49:57] 10Striker: Unlink SUL account "TanvirSdq" from stale developer account on toolsadmin (Striker) - https://phabricator.wikimedia.org/T438120#12358273 (10JJMC89) [07:14:22] 10Tool-eventfeedback: The same user can vote twice - https://phabricator.wikimedia.org/T439097 (10Strainu) 03NEW [07:15:28] 10Tool-eventfeedback: Admins do not see anonymous feedback - https://phabricator.wikimedia.org/T439098 (10Strainu) 03NEW [07:20:42] (03approved) 10tlepage: config: rename alerts.enabled to alerts.notify [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/192 (owner: 10dcaro) [07:21:10] 10Toolforge, 06tools-platform-team, 07OKR-Work: [components-api] change the alerts.enabled config value to alerts.notify - https://phabricator.wikimedia.org/T438995#12358346 (10TLepage-WMF) Associated MR: https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/192 [07:22:47] 10Toolforge, 06tools-platform-team, 07OKR-Work: [components-api] change the alerts.enabled config value to alerts.notify - https://phabricator.wikimedia.org/T438995#12358351 (10TLepage-WMF) 05Open→03In progress [07:27:59] (03approved) 10tlepage: models: updated the models to include webservice job [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/193 (owner: 10dcaro) [07:29:01] 10Toolforge, 06tools-platform-team: [components-api] fails to generate config if the tool has a webservice - https://phabricator.wikimedia.org/T438996#12358363 (10TLepage-WMF) Corresponding MR: https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/193 [07:29:17] 10Toolforge, 06tools-platform-team: [components-api] fails to generate config if the tool has a webservice - https://phabricator.wikimedia.org/T438996#12358365 (10TLepage-WMF) 05Open→03In progress [08:13:49] 10Toolforge (Push-to-Deploy), 06tools-platform-team: Toolforge push-to-deploy bootstrap - https://phabricator.wikimedia.org/T439030#12358492 (10dcaro) [08:14:53] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/16 [08:15:13] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [phase 1]Create Hello World deployment tutorial - https://phabricator.wikimedia.org/T437899#12358499 (10dcaro) [08:15:30] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/23 [08:17:51] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/80 [08:17:54] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/80 [08:19:37] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [phase 1]Create Hello World deployment tutorial - https://phabricator.wikimedia.org/T437899#12358514 (10dcaro) >>! In T437899#12356141, @komla wrote: > @dcaro, I just feel it is tidier when collapsed because the text on the page is getting long. > Up to y... [08:19:44] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/42 [08:19:46] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/42 [08:20:11] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/28 [08:20:18] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/28 [08:20:19] (03update) 10renovatebot: Update python Docker tag to v3.14.7 [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/18 [08:20:25] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/30 [08:20:31] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/30 [08:20:34] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewikisignbot] - 10https://gitlab.wikimedia.org/toolforge-repos/dewikisignbot/-/merge_requests/33 [08:20:39] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewikisignbot] - 10https://gitlab.wikimedia.org/toolforge-repos/dewikisignbot/-/merge_requests/33 [08:24:16] (03merge) 10dcaro: models: updated the models to include webservice job [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/193 [08:24:34] (03update) 10dcaro: config: rename alerts.enabled to alerts.notify [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/192 [08:27:53] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: components-api: bump to 0.0.218-20260924082431-41d95bab [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1431 (https://phabricator.wikimedia.org/T438996) [08:28:22] !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1078' (T435537) [08:28:29] T435537: Check and fix BIOS on new Supermicro cloudvirt* systems - https://phabricator.wikimedia.org/T435537 [08:31:03] 10Cloud-VPS (Project-requests): Request creation of wikivisage VPS project - https://phabricator.wikimedia.org/T438842#12358635 (10dcaro) > Envvars are injected tool-wide. The face service parses untrusted image bytes, so i'd rather it didn't receive the ToolsDB/OAuth secrets. Is a separate tool account the reco... [08:31:31] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api [08:32:34] 06cloud-services-team, 10Toolforge, 06tools-platform-team: [envvars] scope to jobs/components - https://phabricator.wikimedia.org/T405022#12358656 (10dcaro) [08:33:53] 10Toolforge, 06tools-platform-team: [components-api] create a ci scheduled pipeline to update the toolforge models - https://phabricator.wikimedia.org/T439108 (10dcaro) 03NEW [08:36:02] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api [08:36:10] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component components-api [08:41:25] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api [08:43:59] (03approved) 10dcaro: components-api: bump to 0.0.218-20260924082431-41d95bab [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1431 (https://phabricator.wikimedia.org/T438996) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [08:44:04] (03merge) 10dcaro: components-api: bump to 0.0.218-20260924082431-41d95bab [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1431 (https://phabricator.wikimedia.org/T438996) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [08:45:49] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [components-api] fails to generate config if the tool has a webservice - https://phabricator.wikimedia.org/T438996#12358773 (10dcaro) 05In progress→03Resolved [08:46:41] (03update) 10fnegri: config: rename alerts.enabled to alerts.notify [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/192 (owner: 10dcaro) [08:46:43] (03approved) 10fnegri: config: rename alerts.enabled to alerts.notify [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/192 (owner: 10dcaro) [08:47:44] (03merge) 10dcaro: config: rename alerts.enabled to alerts.notify [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/192 [08:51:12] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: components-api: bump to 0.0.219-20260924084756-8f855015 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1432 (https://phabricator.wikimedia.org/T438995) [08:51:20] !log filippo@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1078' (T435537) [08:51:29] T435537: Check and fix BIOS on new Supermicro cloudvirt* systems - https://phabricator.wikimedia.org/T435537 [08:55:26] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component alerts-api [08:55:31] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component alerts-api [08:56:09] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api [08:57:57] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api [08:59:41] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api [09:02:47] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api [09:06:09] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component components-api [09:11:39] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api [09:15:21] 10Toolforge, 06tools-platform-team: [builds-api,envvars-api] Provide mechanism for declaring envvars at build time, that will be used at runtime - https://phabricator.wikimedia.org/T435706#12358874 (10dcaro) [09:15:29] (03merge) 10dcaro: components-api: bump to 0.0.219-20260924084756-8f855015 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1432 (https://phabricator.wikimedia.org/T438995) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [09:17:58] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/28 [09:18:07] (03update) 10renovatebot: Update python Docker tag to v3.14.7 [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/18 [09:18:08] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/30 [09:18:13] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewikisignbot] - 10https://gitlab.wikimedia.org/toolforge-repos/dewikisignbot/-/merge_requests/33 [09:20:55] (03update) 10dcaro: global: add deploy events [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/42 (https://phabricator.wikimedia.org/T432571) [09:23:20] (03update) 10dcaro: global: add deploy events [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/42 (https://phabricator.wikimedia.org/T432571) [09:39:16] 10Toolforge, 06tools-platform-team, 07OKR-Work: [components-api] change the alerts.enabled config value to alerts.notify - https://phabricator.wikimedia.org/T438995#12359041 (10dcaro) 05In progress→03Resolved [09:54:22] !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (T435537) [09:54:26] !log filippo@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0) (T435537) [09:54:29] T435537: Check and fix BIOS on new Supermicro cloudvirt* systems - https://phabricator.wikimedia.org/T435537 [09:54:55] !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1079' (T435537) [10:05:32] 10Toolforge (Push-to-Deploy), 06tools-platform-team: Toolforge push-to-deploy bootstrap - https://phabricator.wikimedia.org/T439030#12359130 (10dcaro) I agree that allowing the initial bootstrap might be safe enough in this case. As long as it only allows it once (if the configuration does not exist), but once... [10:11:26] 10Cloud-VPS, 10Toolforge, 06tools-infrastructure-team: Build mechanism to sync non-public data sets to WMCS HAProxy hosts - https://phabricator.wikimedia.org/T438449#12359152 (10Volans) @taavi one option that came to mind is: 1) [optional] have on the haproxy hosts in cloudvps an rsync server, open only to... [10:14:34] 10Cloud Services Proposals, 10Toolforge (Push-to-Deploy), 06tools-platform-team, 05Cloud-Services-Origin-Team, and 3 others: [builds-api,components-api,webservice,jobs-api] Make Toolforge a proper platform as a service with push-to-deploy and build packs - https://phabricator.wikimedia.org/T194332#12359175 (... [10:16:00] 10Cloud Services Proposals, 10Toolforge (Push-to-Deploy), 06tools-platform-team, 05Cloud-Services-Origin-Team, and 3 others: [builds-api,components-api,webservice,jobs-api] Make Toolforge a proper platform as a service with push-to-deploy and build packs - https://phabricator.wikimedia.org/T194332#12359181 (... [10:21:50] !log filippo@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1079' (T435537) [10:21:58] T435537: Check and fix BIOS on new Supermicro cloudvirt* systems - https://phabricator.wikimedia.org/T435537 [10:29:58] 06cloud-services-team, 10Cloud-VPS (Quota-requests), 10Castor, 10Continuous-Integration-Infrastructure, 06Test Platform (San Miguel de Allende 31): Apply the 4x-iops Cinder volume type to the Castor cache volume (integration-castor06) - https://phabricator.wikimedia.org/T437127#12359266 (10AMarkossyan-WMF) [10:30:00] 10Cloud-VPS (Quota-requests), 10Castor, 10Continuous-Integration-Infrastructure, 06Test Platform (San Miguel de Allende 31): Provide more disk I/O headroom for the Castor cache volume (integration-castor06) - https://phabricator.wikimedia.org/T435571#12359268 (10AMarkossyan-WMF) [10:30:06] !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (T435537) [10:30:10] !log filippo@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0) (T435537) [10:30:14] T435537: Check and fix BIOS on new Supermicro cloudvirt* systems - https://phabricator.wikimedia.org/T435537 [10:30:33] !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1080' (T435537) [11:01:34] !log filippo@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1080' (T435537) [11:01:41] T435537: Check and fix BIOS on new Supermicro cloudvirt* systems - https://phabricator.wikimedia.org/T435537 [11:05:49] 10Tool-wsindex, 06MediaWiki-Media-Platform-Team, 10Wikipedia-iOS-App, 06WikipediaApp, and 2 others: Update apps to support for the move from upload.wikimedia.org to thumb.wikimedia.org - https://phabricator.wikimedia.org/T434821#12359449 (10Seddon) [11:21:33] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [components-cli] Create aliases for the subcommands - https://phabricator.wikimedia.org/T430988#12359480 (10LucasWerkmeister) +1 – as I [wrote in #wikimedia-cloud yesterday](https://wm-bot.wmcloud.org/browser/index.php?start=09%2F23%2F2026&end=09%2F23%2F2026... [11:48:01] 10Toolforge, 06tools-platform-team, 07OKR-Work: Create template for maintainer alerts emails - https://phabricator.wikimedia.org/T437081#12359632 (10fnegri) I did some small updates: {F104584244} [12:05:58] (03open) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/lexeme-forms] - 10https://gitlab.wikimedia.org/toolforge-repos/lexeme-forms/-/merge_requests/49 [12:45:09] !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (T435537) [12:45:12] !log filippo@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0) (T435537) [12:45:17] T435537: Check and fix BIOS on new Supermicro cloudvirt* systems - https://phabricator.wikimedia.org/T435537 [13:19:52] !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.set_maintenance (T437407) [13:19:56] !log filippo@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0) (T437407) [13:19:59] T437407: Explore if there are ways to tune the underlying hardware and configuration for the CI agents - https://phabricator.wikimedia.org/T437407 [13:29:47] 06tools-infrastructure-team, 07Epic, 06Test Platform (Epics): Explore if there are ways to tune the underlying hardware and configuration for the CI agents - https://phabricator.wikimedia.org/T437407#12360060 (10Peter) I added some new graphs today make it easier to see how some agent are faster than others... [13:38:08] 10PAWS, 10Toolforge, 06tools-infrastructure-team, 06Data-Platform-SRE (2026-09-18 - 2026-10-09), 07Essential-Work: Dumps NFS mount down - https://phabricator.wikimedia.org/T438062#12360099 (10Gehel) [13:38:09] 10PAWS, 10Toolforge, 06tools-infrastructure-team, 06Data-Platform-SRE (2026-09-18 - 2026-10-09), 07Essential-Work: Dumps NFS mount down - https://phabricator.wikimedia.org/T438062#12360100 (10Gehel) p:05Triage→03High [13:40:03] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 10Data-Engineering-Roadmap, and 3 others: Ability to create new views for tools and analytic purposes - https://phabricator.wikimedia.org/T424524#12360134 (10Gehel) p:05Triage→03Low [14:20:11] 10Tool-wmf-openapi-linter, 06MediaWiki-API-Platform-Team: OpenAPI Linter: api-testing fails in a browser context - https://phabricator.wikimedia.org/T439141 (10BPirkle) 03NEW [15:08:09] (03CR) 10Jsn.sherman: [C:03+2] "LGTM!" [labs/xtools] - 10https://gerrit.wikimedia.org/r/1344067 (https://phabricator.wikimedia.org/T437830) (owner: 10MusikAnimal) [15:08:51] (03Merged) 10jenkins-bot: Repository: Route links-table queries to their own replica section [labs/xtools] - 10https://gerrit.wikimedia.org/r/1344067 (https://phabricator.wikimedia.org/T437830) (owner: 10MusikAnimal) [15:15:42] (03CR) 10Jsn.sherman: [C:03+2] PageInfo, Page: Read links tables from the links connection [labs/xtools] - 10https://gerrit.wikimedia.org/r/1344068 (https://phabricator.wikimedia.org/T437830) (owner: 10MusikAnimal) [15:16:25] (03Merged) 10jenkins-bot: PageInfo, Page: Read links tables from the links connection [labs/xtools] - 10https://gerrit.wikimedia.org/r/1344068 (https://phabricator.wikimedia.org/T437830) (owner: 10MusikAnimal) [15:16:55] (03CR) 10Jsn.sherman: [C:03+2] "I reran this end-to-end locally and it does what it says on the tin; Thanks for your work!" [labs/xtools] - 10https://gerrit.wikimedia.org/r/1344069 (https://phabricator.wikimedia.org/T437830) (owner: 10MusikAnimal) [15:17:01] (03CR) 10CI reject: [V:04-1] CategoryEdits: Intersect categories and revisions in code [labs/xtools] - 10https://gerrit.wikimedia.org/r/1344069 (https://phabricator.wikimedia.org/T437830) (owner: 10MusikAnimal) [15:17:31] (03PS6) 10MusikAnimal: CategoryEdits: Intersect categories and revisions in code [labs/xtools] - 10https://gerrit.wikimedia.org/r/1344069 (https://phabricator.wikimedia.org/T437830) [15:18:18] (03CR) 10Jsn.sherman: "let's try that again" [labs/xtools] - 10https://gerrit.wikimedia.org/r/1344069 (https://phabricator.wikimedia.org/T437830) (owner: 10MusikAnimal) [15:18:19] (03open) 10andrew: Update k8stag for default magnum templates [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/363 [15:19:34] (03Merged) 10jenkins-bot: CategoryEdits: Intersect categories and revisions in code [labs/xtools] - 10https://gerrit.wikimedia.org/r/1344069 (https://phabricator.wikimedia.org/T437830) (owner: 10MusikAnimal) [15:22:25] (03merge) 10andrew: Update k8stag for default magnum templates [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/363 [15:22:39] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [15:23:44] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [15:30:15] (03update) 10r4356th: Implement visual regression testing [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/22 (https://phabricator.wikimedia.org/T415882) [15:41:09] 10Tool-spacemedia, 10Toolforge, 06tools-platform-team: Can't deploy anymore on Toolforge (pod stays pending) - https://phabricator.wikimedia.org/T438255#12360747 (10Don-vip) It happens again, are the resources allowed for tools being decreased? I wouldn't like being forced to decrease the resources allocated... [15:42:46] (03approved) 10kuldeepburjbhalaike: proper description for bn [toolforge-repos/bookitemizer] - 10https://gitlab.wikimedia.org/toolforge-repos/bookitemizer/-/merge_requests/1 (owner: 10bodhisattwa) [15:42:55] (03unapproved) 10kuldeepburjbhalaike: proper description for bn [toolforge-repos/bookitemizer] - 10https://gitlab.wikimedia.org/toolforge-repos/bookitemizer/-/merge_requests/1 (owner: 10bodhisattwa) [15:43:10] (03merge) 10kuldeepburjbhalaike: proper description for bn [toolforge-repos/bookitemizer] - 10https://gitlab.wikimedia.org/toolforge-repos/bookitemizer/-/merge_requests/1 (owner: 10bodhisattwa) [15:46:15] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [components-api] Don't mention namespaces when the config does not exist - https://phabricator.wikimedia.org/T439150 (10dcaro) 03NEW [15:46:32] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [components-api] Toolforge push-to-deploy bootstrap - https://phabricator.wikimedia.org/T439030#12360781 (10dcaro) [15:49:41] (03update) 10r4356th: Implement visual regression testing [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/22 (https://phabricator.wikimedia.org/T415882) [15:52:45] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [phase 1]Create Hello World deployment tutorial - https://phabricator.wikimedia.org/T437899#12360808 (10komla) Good idea! I will do that shortly. [15:54:14] (03update) 10r4356th: Implement visual regression testing for everything but templates that require arguments [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/22 (https://phabricator.wikimedia.org/T415882) [15:55:30] (03update) 10dcaro: global: add deploy events [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/42 (https://phabricator.wikimedia.org/T432571) [15:56:01] (03update) 10r4356th: Implement visual regression testing for everything but templates that require arguments [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/22 (https://phabricator.wikimedia.org/T415882) [15:56:35] (03update) 10dcaro: global: add deploy events [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/42 (https://phabricator.wikimedia.org/T432571) [15:57:32] (03update) 10dcaro: global: add deploy events [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/42 (https://phabricator.wikimedia.org/T432571) [16:03:34] (03update) 10r4356th: Implement visual regression testing for everything but templates that require arguments [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/22 (https://phabricator.wikimedia.org/T415882) [16:15:14] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [components-cli] Create aliases for the subcommands - https://phabricator.wikimedia.org/T430988#12360945 (10dcaro) [16:20:08] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [components-cli] Create aliases for the subcommands - https://phabricator.wikimedia.org/T430988#12360958 (10fnegri) +1 to the new versions that were just added by @dcaro to the task description. [16:22:39] (03open) 10bodhisattwa: Bengali --> Bangla [toolforge-repos/bookitemizer] - 10https://gitlab.wikimedia.org/toolforge-repos/bookitemizer/-/merge_requests/2 [16:25:44] 10Toolforge, 06tools-platform-team, 07OKR-Work: Create template for maintainer alerts emails - https://phabricator.wikimedia.org/T437081#12360991 (10fnegri) Iterated on it a bit more: {F104589691} One thing I'm unsure if we should add to the template something like //"We will email you again when the issue... [16:37:37] (03close) 10kuldeepburjbhalaike: Bengali --> Bangla [toolforge-repos/bookitemizer] - 10https://gitlab.wikimedia.org/toolforge-repos/bookitemizer/-/merge_requests/2 (owner: 10bodhisattwa) [16:37:37] (03update) 10kuldeepburjbhalaike: Bengali --> Bangla [toolforge-repos/bookitemizer] - 10https://gitlab.wikimedia.org/toolforge-repos/bookitemizer/-/merge_requests/2 (owner: 10bodhisattwa) [16:43:11] (03update) 10r4356th: Implement visual regression testing for everything but templates that require arguments [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/22 (https://phabricator.wikimedia.org/T415882) [16:46:36] 10Tool-wmf-openapi-linter, 10Wikidata, 06MediaWiki-API-Platform-Team (MediaWiki-Interfaces-Sprints), 06Wikidata-Omega (Triaging): Unable to upgrade Wikibase to api-testing 1.8.0 - `fs.readFileSync is not a function` - https://phabricator.wikimedia.org/T438947#12361075 (10AGhirelli-WMF) [16:47:48] (03update) 10r4356th: Implement visual regression testing for everything but templates that require arguments [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/22 (https://phabricator.wikimedia.org/T415882) [16:49:10] (03update) 10r4356th: Implement visual regression testing for everything but templates that require arguments [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/22 (https://phabricator.wikimedia.org/T415882) [16:49:53] (03open) 10bodhisattwa: P31 literary work [toolforge-repos/bookitemizer] - 10https://gitlab.wikimedia.org/toolforge-repos/bookitemizer/-/merge_requests/3 [16:49:54] 10Tool-wmf-openapi-linter, 06MediaWiki-API-Platform-Team: OpenAPI Linter: api-testing fails in a browser context - https://phabricator.wikimedia.org/T439141#12361099 (10BPirkle) [16:50:21] 10Tool-wmf-openapi-linter, 06MediaWiki-API-Platform-Team: OpenAPI Linter: api-testing fails in a browser context - https://phabricator.wikimedia.org/T439141#12361104 (10BPirkle) p:05Triage→03Medium [16:51:41] (03close) 10bodhisattwa: P31 literary work [toolforge-repos/bookitemizer] - 10https://gitlab.wikimedia.org/toolforge-repos/bookitemizer/-/merge_requests/3 [16:51:41] (03update) 10bodhisattwa: P31 literary work [toolforge-repos/bookitemizer] - 10https://gitlab.wikimedia.org/toolforge-repos/bookitemizer/-/merge_requests/3 [16:54:51] !log andrew@cloudcumin1001 tools START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster (T438255) [16:54:57] T438255: Can't deploy anymore on Toolforge (pod stays pending) - https://phabricator.wikimedia.org/T438255 [17:02:03] FIRING: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance tools-k8s-worker-nfs-84 in project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [17:03:00] !log andrew@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster [17:05:10] (03open) 10andrew: flavors: add new monster 24/32 flavor for toolforge [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/364 (https://phabricator.wikimedia.org/T390228) [17:06:08] (03update) 10andrew: flavors: add new monster 24core/32gig flavor for toolforge [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/364 (https://phabricator.wikimedia.org/T390228) [17:06:48] (03approved) 10dcaro: flavors: add new monster 24core/32gig flavor for toolforge [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/364 (https://phabricator.wikimedia.org/T390228) (owner: 10andrew) [17:07:32] 10Tool-spacemedia, 10Toolforge, 06tools-platform-team: Can't deploy anymore on Toolforge (pod stays pending) - https://phabricator.wikimedia.org/T438255#12361156 (10Don-vip) This time I managed to deploy by lowering only by 0.1 cpu :) ` toolforge webservice --cpu 2400m --mem 4500Mi --mount none buildservic... [17:07:52] (03update) 10r4356th: Implement visual regression testing for everything but templates that require arguments [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/22 (https://phabricator.wikimedia.org/T415882) [17:08:14] (03merge) 10andrew: flavors: add new monster 24core/32gig flavor for toolforge [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/364 (https://phabricator.wikimedia.org/T390228) [17:08:17] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [17:09:26] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [17:11:14] !log andrew@cloudcumin1001 tools START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster (T438255) [17:11:18] T438255: Can't deploy anymore on Toolforge (pod stays pending) - https://phabricator.wikimedia.org/T438255 [17:11:47] 10Toolforge, 06tools-infrastructure-team, 13Patch-For-Review: [toolforge] Rebuild Toolforge workers for bigger node sizes and in new VXLAN network - https://phabricator.wikimedia.org/T390228#12361167 (10Andrew) ftr: I'm creating new worker nodes today in response to T438255. At the moment we seem to be hitt... [17:12:03] RESOLVED: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance tools-k8s-worker-nfs-84 in project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [17:12:06] (03update) 10r4356th: Implement visual regression testing for everything but templates that require arguments [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/22 (https://phabricator.wikimedia.org/T415882) [17:15:24] 10Tool-spacemedia, 10Toolforge, 06tools-platform-team: Update Toolforge Deploy to Heroku 24_0.21.20+ to get openjdk 27 support - https://phabricator.wikimedia.org/T439154 (10Don-vip) 03NEW [17:17:59] (03open) 10don-vip: builds-api: update heroku builder and runner images for Java 27 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1433 (https://phabricator.wikimedia.org/T439154) [17:19:01] 10Tool-spacemedia, 10Toolforge, 06tools-platform-team, 13Patch-For-Review: Update Toolforge Deploy to Heroku 24_0.21.20+ to get openjdk 27 support - https://phabricator.wikimedia.org/T439154#12361208 (10Don-vip) [17:22:05] !log andrew@cloudcumin1001 tools Added a new k8s worker-nfs tools-k8s-worker-nfs-84.tools.eqiad1.wikimedia.cloud to the cluster [17:22:05] !log andrew@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster [17:24:46] 10Cloud-VPS (Quota-requests), 06tools-infrastructure-team, 06tools-platform-team: Quota increase request for toolforge - https://phabricator.wikimedia.org/T439157 (10Andrew) 03NEW [17:25:00] 10Cloud-VPS (Quota-requests), 06tools-infrastructure-team, 06tools-platform-team: Quota increase request for toolforge - https://phabricator.wikimedia.org/T439157#12361248 (10Andrew) I'm requesting this so I think that counts as a +1 [17:25:19] !log andrew@cloudcumin1001 tools START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster (T438255) [17:25:24] T438255: Can't deploy anymore on Toolforge (pod stays pending) - https://phabricator.wikimedia.org/T438255 [17:26:17] 10Cloud-VPS (Quota-requests), 06tools-infrastructure-team, 06tools-platform-team: Quota increase request for toolforge - https://phabricator.wikimedia.org/T439157#12361257 (10dcaro) +1 [17:38:12] !log andrew@cloudcumin1001 tools Added a new k8s worker-nfs tools-k8s-worker-nfs-85.tools.eqiad1.wikimedia.cloud to the cluster [17:38:12] !log andrew@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster [17:52:45] !log andrew@cloudcumin1001 tools START - Cookbook wmcs.openstack.quota_increase by 192 cores, 262144 ram (T439157) [17:52:50] T439157: Quota increase request for toolforge - https://phabricator.wikimedia.org/T439157 [17:52:52] !log andrew@cloudcumin1001 tools END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) by 192 cores, 262144 ram (T439157) [17:53:20] !log andrew@cloudcumin1001 tools START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster (T438255) [17:53:23] T438255: Can't deploy anymore on Toolforge (pod stays pending) - https://phabricator.wikimedia.org/T438255 [18:02:55] !log andrew@cloudcumin1001 tools Added a new k8s worker-nfs tools-k8s-worker-nfs-86.tools.eqiad1.wikimedia.cloud to the cluster [18:02:55] !log andrew@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster [18:28:14] !log andrew@cloudcumin1001 tools START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster (T438255) [18:28:20] T438255: Can't deploy anymore on Toolforge (pod stays pending) - https://phabricator.wikimedia.org/T438255 [18:37:05] FIRING: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance tools-k8s-worker-nfs-87 in project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [18:38:14] !log andrew@cloudcumin1001 tools Added a new k8s worker-nfs tools-k8s-worker-nfs-87.tools.eqiad1.wikimedia.cloud to the cluster [18:38:14] !log andrew@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster [18:42:05] RESOLVED: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance tools-k8s-worker-nfs-87 in project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [18:46:16] 06cloud-services-team, 10Toolforge (Push-to-Deploy), 06tools-platform-team: [components-api] Add source polling build trigger - https://phabricator.wikimedia.org/T362071#12361429 (10connorshea) If you limit this to only work with the WMF GitLab instance, I would think a GitLab API token with access to the to... [19:07:20] !log andrew@cloudcumin1001 tools START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster (T438255) [19:07:25] T438255: Can't deploy anymore on Toolforge (pod stays pending) - https://phabricator.wikimedia.org/T438255 [19:07:39] 06cloud-services-team, 10Toolforge, 06tools-platform-team, 10Elasticsearch, 07Epic: Deploy multi-tenant OpenSearch cluster as replacement for Elasticsearch - https://phabricator.wikimedia.org/T348943#12361537 (10fnegri) This was briefly discussed today during a Discord call in the Wikimedia community Dis... [19:20:51] !log andrew@cloudcumin1001 tools Added a new k8s worker-nfs tools-k8s-worker-nfs-88.tools.eqiad1.wikimedia.cloud to the cluster [19:20:51] !log andrew@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster [19:25:03] FIRING: PuppetAgentNoResources: No Puppet resources found on instance tools-k8s-worker-nfs-87 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [19:53:50] !log andrew@cloudcumin1001 tools START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster (T438255) [19:53:55] T438255: Can't deploy anymore on Toolforge (pod stays pending) - https://phabricator.wikimedia.org/T438255 [20:00:24] 10Toolforge, 06tools-platform-team, 07OKR-Work: Create template for maintainer alerts emails - https://phabricator.wikimedia.org/T437081#12361728 (10CCiufo-WMF) >>! In T437081#12360991, @fnegri wrote: > One thing I'm unsure is if we should add to the template something like //"We will email you again when th... [20:03:03] FIRING: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance tools-k8s-worker-nfs-89 in project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [20:04:20] 10Tool-spacemedia, 10Toolforge, 06tools-platform-team: Can't deploy anymore on Toolforge (pod stays pending) - https://phabricator.wikimedia.org/T438255#12361759 (10Andrew) I've just added a few more worker nodes; are things working better now? [20:05:56] !log andrew@cloudcumin1001 tools Added a new k8s worker-nfs tools-k8s-worker-nfs-89.tools.eqiad1.wikimedia.cloud to the cluster [20:05:56] !log andrew@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster [20:06:06] 10Tool-spacemedia, 10Toolforge, 06tools-platform-team: Can't deploy anymore on Toolforge (pod stays pending) - https://phabricator.wikimedia.org/T438255#12361773 (10Don-vip) Thanks! I try to restore the original values, let's see. [20:07:36] !log andrew@cloudcumin1001 tools START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster (T438255) [20:07:40] T438255: Can't deploy anymore on Toolforge (pod stays pending) - https://phabricator.wikimedia.org/T438255 [20:08:03] RESOLVED: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance tools-k8s-worker-nfs-89 in project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [20:13:42] (03approved) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/lexeme-forms] - 10https://gitlab.wikimedia.org/toolforge-repos/lexeme-forms/-/merge_requests/49 (owner: 10l10n-bot) [20:14:19] (03merge) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/lexeme-forms] - 10https://gitlab.wikimedia.org/toolforge-repos/lexeme-forms/-/merge_requests/49 (owner: 10l10n-bot) [20:15:03] RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance tools-k8s-worker-nfs-87 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [20:17:06] !log andrew@cloudcumin1001 tools Added a new k8s worker tools-k8s-worker-114.tools.eqiad1.wikimedia.cloud to the cluster [20:17:06] !log andrew@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster [20:17:31] !log andrew@cloudcumin1001 tools START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster (T438255) [20:17:36] T438255: Can't deploy anymore on Toolforge (pod stays pending) - https://phabricator.wikimedia.org/T438255 [20:26:08] !log andrew@cloudcumin1001 tools Added a new k8s worker tools-k8s-worker-115.tools.eqiad1.wikimedia.cloud to the cluster [20:26:08] !log andrew@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster [20:39:25] 10Tool-spacemedia, 10Toolforge, 06tools-platform-team: Can't deploy anymore on Toolforge (pod stays pending) - https://phabricator.wikimedia.org/T438255#12361859 (10Don-vip) [[ https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/jobs/974347 | It works ]]! Thanks a lot :) [20:53:03] FIRING: PuppetAgentNoResources: No Puppet resources found on instance tools-k8s-worker-nfs-89 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [20:58:30] 06tools-infrastructure-team: tools-k8s-worker-nfs going offline - https://phabricator.wikimedia.org/T439177 (10BLiviero-WMF) 03NEW [23:54:38] (03update) 10habs: T433052: implement RTL support via dynamic dir attribute [toolforge-repos/wikinewsie] - 10https://gitlab.wikimedia.org/toolforge-repos/wikinewsie/-/merge_requests/14 (owner: 10chimaifeanyi29)