[01:15:47] (03update) 10renovatebot: Update dependency brace-expansion to v5.0.9 [SECURITY] [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/31 [01:17:26] 10Cloud-VPS, 06tools-infrastructure-team, 10Cumin, 06Infrastructure-Foundations: Enable cumin hostfile backend on cloudcumin hosts - https://phabricator.wikimedia.org/T433916 (10Andrew) 03NEW [02:25:25] FIRING: SystemdUnitFailed: remove_dangling_cinder_snapshots.service on cloudbackup2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [02:31:29] 10tool-wscontest, 06RoadToWiki, 06Wiki-Mentor-Africa, 07good first task, 13Patch-For-Review: WSContest Tool admin cannot edit contest page if their name is multi-word separated by underscore (they can edit if it's a space, instead) - https://phabricator.wikimedia.org/T336157#12181563 (10Samwilson) 0... [02:48:41] RESOLVED: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [04:40:00] (03merge) 10countcount: Update dependency brace-expansion to v5.0.9 [SECURITY] [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/31 (owner: 10renovatebot) [05:09:15] (03merge) 10countcount: Update pnpm to v11.19.0 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/47 (owner: 10renovatebot) [05:09:28] (03merge) 10countcount: Update pnpm to v11.19.0 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/15 (owner: 10renovatebot) [05:09:41] (03merge) 10countcount: Update pnpm to v11.19.0 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/8 (owner: 10renovatebot) [05:10:16] (03merge) 10countcount: Update pnpm to v11.19.0 [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/21 (owner: 10renovatebot) [05:12:06] (03merge) 10countcount: Lock file maintenance [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/23 (owner: 10renovatebot) [05:12:34] (03merge) 10countcount: Lock file maintenance [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/74 (owner: 10renovatebot) [05:14:56] (03update) 10renovatebot: Update dependency @napi-rs/wasm-runtime to v1.2.2 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/5 [05:15:21] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/19 [05:15:38] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/14 [05:16:38] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/46 [05:17:40] (03update) 10renovatebot: Update pnpm to v11.19.0 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/30 [05:18:46] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/29 [06:14:12] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/6 [06:25:40] FIRING: SystemdUnitFailed: remove_dangling_cinder_snapshots.service on cloudbackup2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [07:38:13] 10Cloud-VPS (Debian Bullseye Deprecation), 10Internet In A Box: Iiab vps debian migration blocked on exposing rsync - https://phabricator.wikimedia.org/T433714#12181770 (10Tim-moody) Thanks. That's great. I don't think we need kiwix on this machine any longer and offline was for producing the mdwiki zim, whic... [07:46:27] 10Tool-itwiki, 10MediaWiki-Action-API, 06MediaWiki-API-Platform-Team: Missing categories from generator=categorymembers (just 1 instead of 2) in itwiki - https://phabricator.wikimedia.org/T433922 (10valerio.bozzolan) 03NEW [07:51:07] 10Cloud-VPS (Debian Bullseye Deprecation), 10Internet In A Box: Iiab vps debian migration blocked on exposing rsync - https://phabricator.wikimedia.org/T433714#12181786 (10Tim-moody) in nginx.conf ` upstream php { server unix:/run/php/php8.3-fpm.sock; } ` looks to point to the previous version... [07:51:45] 10Cloud-VPS (Debian Bullseye Deprecation), 10Internet In A Box: Iiab vps debian migration blocked on exposing rsync - https://phabricator.wikimedia.org/T433714#12181787 (10Tim-moody) And looks like medbox4 is already shut down. [08:13:07] (03update) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/30 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [08:13:10] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/30 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [08:13:15] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/30 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [08:13:23] (03update) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/31 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [08:15:48] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: logs-api: bump to 0.0.40-20260804081327-1c2cb621 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1356 [08:22:30] 10Tool-itwiki, 10MediaWiki-Action-API, 06MediaWiki-API-Platform-Team: Missing categories from generator=categorymembers (just 1 instead of 2) in itwiki - https://phabricator.wikimedia.org/T433922#12181872 (10valerio.bozzolan) [08:45:25] (03close) 10tlepage: Draft: Tentative update for glibc 2.36+ [repos/cloud/toolforge/buildpacks/locale-buildpack] (move_to_api_0.10) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/buildpacks/locale-buildpack/-/merge_requests/2 [08:46:48] (03approved) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/31 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [08:46:55] (03merge) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/31 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [08:47:20] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [08:47:22] !log dcaro@cloudcumin1001 toolsbeta END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api [08:47:31] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [08:47:36] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api [08:49:18] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: logs-api: bump to 0.0.41-20260804084710-047a28af [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1356 [08:49:22] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: logs-api: bump to 0.0.41-20260804084710-047a28af [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1356 [08:59:43] 10Toolforge, 06tools-platform-team, 07OKR-Work: [api-gateway] Allow specifying allowed urls for superusers - https://phabricator.wikimedia.org/T433930 (10dcaro) 03NEW [08:59:54] 10Toolforge, 06tools-platform-team, 07OKR-Work: [api-gateway] Allow specifying allowed urls for superusers - https://phabricator.wikimedia.org/T433930#12182032 (10dcaro) [08:59:58] 10Toolforge, 06tools-platform-team, 07OKR-Work: [api-gateway] Allow specifying allowed urls for superusers - https://phabricator.wikimedia.org/T433930#12182034 (10dcaro) 05Open→03In progress [09:00:02] 10Toolforge, 06tools-platform-team, 07OKR-Work: [api-gateway] Allow specifying allowed urls for superusers - https://phabricator.wikimedia.org/T433930#12182036 (10dcaro) a:03dcaro [09:02:38] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component logs-api [09:04:15] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logs-api [09:04:52] 10Toolforge, 06tools-platform-team, 07OKR-Work: [api-gateway] Allow specifying allowed urls for superusers - https://phabricator.wikimedia.org/T433930#12182049 (10dcaro) p:05Triage→03High [09:15:11] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component logs-api [09:15:12] !log dcaro@cloudcumin1001 tools END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component logs-api [09:15:35] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component logs-api [09:24:11] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api [09:34:05] (03open) 10dcaro: auth: Allow specifying allowed urls for superusers [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/103 [09:34:05] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component logs-api [09:34:27] (03update) 10countcount: Draft: Expose account block behavior flags [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/73 [09:34:54] (03update) 10countcount: Draft: Expose account block behavior flags [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/73 [09:35:54] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate WMCS-managed NFS servers off of Bullseye - https://phabricator.wikimedia.org/T401812#12182273 (10taavi) 05Resolved→03Open The VMs are still there, just in a shutoff status [09:40:44] (03merge) 10tlepage: [T432832] replace class mocks with magicmocks [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/361 (https://phabricator.wikimedia.org/T432832) (owner: 10mahveotm) [09:41:57] (03update) 10countcount: Draft: Expose account block behavior flags [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/73 [09:42:14] (03update) 10countcount: Draft: Expose account block behavior flags [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/73 [09:44:30] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.556-20260804094055-ad1142c2 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1357 (https://phabricator.wikimedia.org/T432832) [09:44:33] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.556-20260804094055-ad1142c2 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1357 (https://phabricator.wikimedia.org/T432832) [09:45:32] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api [09:48:41] (03update) 10countcount: Draft: Expose account block behavior flags [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/73 [09:49:02] (03update) 10countcount: Draft: Expose account block behavior flags [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/73 [09:50:36] !log tlepage@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [09:51:32] !log tlepage@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api [09:53:01] !log tlepage@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [09:54:28] (03update) 10dcaro: auth: Allow specifying allowed urls for superusers [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/103 [09:54:53] (03approved) 10dcaro: logs-api: bump to 0.0.41-20260804084710-047a28af [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1356 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [09:54:57] (03merge) 10dcaro: logs-api: bump to 0.0.41-20260804084710-047a28af [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1356 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [10:03:45] 10Cloud-VPS (Quota-requests), 10Wikidata: Quota increase request for project pm20database - https://phabricator.wikimedia.org/T433839#12182368 (10Ifrahkhanyaree_WMDE) [10:04:24] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/92 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [10:04:27] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/92 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [10:05:11] !log tlepage@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [10:05:48] !log tlepage@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [10:06:58] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: maintain-kubeusers: bump to 0.0.201-20260804100441-102b23dd [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1358 [10:07:05] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: maintain-kubeusers: bump to 0.0.201-20260804100441-102b23dd [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1358 [10:08:52] 10Cloud-VPS (Debian Bullseye Deprecation), 10Wikimedia-Developer-Portal: Update Dev Portal staging to Debian Bookworm - https://phabricator.wikimedia.org/T429490#12182390 (10KBach) a:03KBach [10:11:15] 10Toolforge, 06tools-infrastructure-team: See if we can borrow parts of the wikiprod WAF for Toolforge - https://phabricator.wikimedia.org/T409759#12182394 (10aputhin) [10:11:48] 10Toolforge, 06tools-platform-team, 07OKR-Work: [components-api] Create endpoint to get tools based on a set config - https://phabricator.wikimedia.org/T433690#12182406 (10aputhin) p:05Triage→03High [10:18:26] 10Toolforge, 06tools-platform-team, 07OKR-Work: [logs-api,loki] Implement a system-level logging endpoint (write) - https://phabricator.wikimedia.org/T432565#12182428 (10dcaro) [10:22:17] !log tlepage@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [10:23:18] 10Toolforge, 06tools-platform-team, 07OKR-Work: [logs-api,loki] Implement a system-level logging endpoint (write) - https://phabricator.wikimedia.org/T432565#12182449 (10dcaro) a:03dcaro [10:24:05] 10Cloud-VPS (Debian Bullseye Deprecation), 10Wikimedia-Developer-Portal: Update Dev Portal staging to Debian Bookworm - https://phabricator.wikimedia.org/T429490#12182451 (10KBach) I can look into this but need access to the devportal project in Horizon to copy instance settings and security groups. @bd808 @TB... [10:24:38] (03update) 10tlepage: jobs-api: bump to 0.0.556-20260804094055-ad1142c2 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1357 (https://phabricator.wikimedia.org/T432832) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [10:25:21] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/101 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [10:25:24] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/101 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [10:25:32] (03update) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/102 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [10:25:40] FIRING: SystemdUnitFailed: remove_dangling_cinder_snapshots.service on cloudbackup2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [10:25:48] (03approved) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/102 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [10:26:20] (03merge) 10tlepage: jobs-api: bump to 0.0.556-20260804094055-ad1142c2 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1357 (https://phabricator.wikimedia.org/T432832) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [10:27:30] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: api-gateway: bump to 0.0.102-20260804102536-df3faf4d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1359 [10:27:32] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: api-gateway: bump to 0.0.102-20260804102536-df3faf4d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1359 [10:27:36] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [jobs-api] replace class mocks with magicmocks - https://phabricator.wikimedia.org/T432832#12182455 (10TLepage-WMF) 05Open→03Resolved Thank you for your contribution @Mahveotm! [10:27:38] (03merge) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/102 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [10:27:44] (03update) 10dcaro: auth: Allow specifying allowed urls for superusers [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/103 [10:29:08] (03update) 10dcaro: auth: Allow specifying allowed urls for superusers [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/103 [10:29:48] (03update) 10dcaro: auth: Allow specifying allowed urls for superusers [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/103 [10:30:01] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: api-gateway: bump to 0.0.103-20260804102750-cee3756e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1359 [10:30:08] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: api-gateway: bump to 0.0.103-20260804102750-cee3756e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1359 [10:35:32] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component api-gateway [10:38:11] 06tools-platform-team: [jobs-cli,envvars-cli,toolforge-cli,builds-cli] enable usage stats collection - https://phabricator.wikimedia.org/T433954 (10dcaro) 03NEW [10:38:22] 10Toolforge, 06tools-platform-team: [jobs-cli,envvars-cli,toolforge-cli,builds-cli] enable usage stats collection - https://phabricator.wikimedia.org/T433954#12182543 (10dcaro) [10:39:05] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway [10:39:28] (03approved) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/70 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [10:39:31] (03merge) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/70 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [10:39:58] (03approved) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/144 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [10:40:02] (03merge) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/144 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [10:40:13] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/334 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [10:40:17] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/334 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [10:40:49] (03update) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/69 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [10:41:09] (03approved) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/115 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [10:41:12] (03merge) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/115 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [10:42:14] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/69 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [10:42:17] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/86 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [10:42:21] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/86 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [10:42:25] 10Tool-leximap: Add Malayalam language to Leximap - https://phabricator.wikimedia.org/T433956 (10SGill) 03NEW [10:42:28] (03update) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/169 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [10:43:05] (03update) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/143 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [10:43:14] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/117 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [10:43:18] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/117 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [10:43:33] (03update) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/87 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [10:43:44] (03approved) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/87 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [10:43:52] (03update) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/114 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [10:43:56] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/114 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [10:44:30] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component api-gateway [10:45:00] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/143 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [10:45:19] (03merge) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/87 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [10:45:48] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/114 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [10:46:02] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/misctools-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/misctools-cli/-/merge_requests/19 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [10:46:11] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/misctools-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/misctools-cli/-/merge_requests/19 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [10:49:02] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway [10:51:03] (03approved) 10dcaro: api-gateway: bump to 0.0.103-20260804102750-cee3756e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1359 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [10:51:07] (03merge) 10dcaro: api-gateway: bump to 0.0.103-20260804102750-cee3756e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1359 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [10:51:15] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers [10:55:56] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/169 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [10:55:57] (03update) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/169 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [10:56:06] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/169 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [11:02:36] 10Cloud-VPS, 06tools-infrastructure-team: Migrate cloudidp-dev to Redis ticket storage - https://phabricator.wikimedia.org/T433963 (10taavi) 03NEW [11:03:59] 10Cloud-VPS, 06tools-infrastructure-team: Migrate cloudidp-dev to Redis ticket storage - https://phabricator.wikimedia.org/T433963#12182753 (10taavi) a:03taavi [11:05:32] (03PS5) 10Alien4444: SimpleEditCounter: add latest edit and log [labs/xtools] - 10https://gerrit.wikimedia.org/r/1276371 (https://phabricator.wikimedia.org/T198872) [11:06:00] (03CR) 10Alien4444: "1. It's about the same. I'd forgotten I'd made that already (When A Patch Stays Alive Too Long TM). Have switched to using `getLatestEditA" [labs/xtools] - 10https://gerrit.wikimedia.org/r/1276371 (https://phabricator.wikimedia.org/T198872) (owner: 10Alien4444) [11:07:42] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers [11:22:52] 10Cloud-VPS (Debian Bullseye Deprecation), 10Wikimedia-Developer-Portal: Update Dev Portal staging to Debian Bookworm - https://phabricator.wikimedia.org/T429490#12182812 (10Aklapper) @KBach: I received and verified your NDA and hereby added you as a `devportal` member in Horizon [11:23:06] (03PS2) 10Slyngshede: Tofurkey: switch to a secrets file [labs/private] - 10https://gerrit.wikimedia.org/r/1320172 [11:29:40] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/68 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [11:29:43] (03update) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/68 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [11:29:44] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/68 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [11:30:05] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers [11:30:17] (03update) 10dcaro: maintain-kubeusers: bump to 0.0.201-20260804100441-102b23dd [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1358 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [11:32:28] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: maintain-harbor: bump to 0.0.73-20260804112956-fcba8c11 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1360 [11:32:31] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: maintain-harbor: bump to 0.0.73-20260804112956-fcba8c11 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1360 [11:38:25] (03PS3) 10Slyngshede: Tofurkey: switch to a secrets file [labs/private] - 10https://gerrit.wikimedia.org/r/1320172 [11:42:37] (03CR) 10Slyngshede: [V:03+2 C:03+2] Tofurkey: switch to a secrets file [labs/private] - 10https://gerrit.wikimedia.org/r/1320172 (owner: 10Slyngshede) [11:56:28] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers [12:03:03] (03approved) 10dcaro: maintain-kubeusers: bump to 0.0.201-20260804100441-102b23dd [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1358 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:03:07] (03merge) 10dcaro: maintain-kubeusers: bump to 0.0.201-20260804100441-102b23dd [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1358 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:04:05] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor [12:04:16] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/93 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:04:28] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/93 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:04:52] (03approved) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/57 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:04:56] (03merge) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/57 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:05:05] (03update) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/56 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:05:53] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: builds-builder: bump to 0.0.152-20260804120440-39fb286a [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1361 [12:06:00] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: builds-builder: bump to 0.0.152-20260804120440-39fb286a [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1361 [12:06:12] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor [12:06:18] 10Cloud-VPS (Quota-requests), 10Wikidata: Quota increase request for project pm20database - https://phabricator.wikimedia.org/T433839#12182874 (10Reedy) [12:06:20] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/77 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:06:27] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/77 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:06:58] (03approved) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/179 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:07:02] (03merge) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/179 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:07:11] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor [12:07:36] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-emailer: bump to 0.0.91-20260804120509-fad5dc4e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1362 [12:08:43] (03update) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/56 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:09:33] 10Toolforge, 06tools-platform-team: [all] first of the month dependency upgrades - https://phabricator.wikimedia.org/T433878#12182876 (10dcaro) p:05Triage→03Medium [12:09:43] 10Toolforge, 06tools-platform-team, 07OKR-Work: [logs-api,loki] Implement a system-level logging endpoint (write) - https://phabricator.wikimedia.org/T432565#12182878 (10dcaro) 05Open→03In progress [12:10:03] !log dcaro@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor [12:10:52] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: components-api: bump to 0.0.204-20260804120721-f816fd58 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1363 [12:12:56] (03CR) 10David Caro: webservice-runner: default to 8000 only if PORT and TOOL_WEB_PORT has no value (031 comment) [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/1310212 (https://phabricator.wikimedia.org/T432078) (owner: 10Raymond Ndibe) [12:13:46] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor [12:15:22] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: envvars-api: bump to 0.0.88-20260804120640-713d3bb6 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1364 [12:15:24] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: envvars-api: bump to 0.0.88-20260804120640-713d3bb6 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1364 [12:18:31] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor [12:22:37] (03open) 10dcaro: api-gateway: allow jobs-api to send logs to logs-api [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1365 [12:26:23] (03approved) 10dcaro: maintain-harbor: bump to 0.0.73-20260804112956-fcba8c11 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1360 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:26:28] (03update) 10dcaro: maintain-harbor: bump to 0.0.73-20260804112956-fcba8c11 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1360 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:26:54] (03merge) 10dcaro: maintain-harbor: bump to 0.0.73-20260804112956-fcba8c11 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1360 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:27:01] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/56 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:27:05] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/56 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:27:45] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/166 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:27:49] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/166 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:28:05] (03update) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/359 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:29:41] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-emailer: bump to 0.0.92-20260804122721-05c10f0c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1362 [12:29:45] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-emailer: bump to 0.0.92-20260804122721-05c10f0c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1362 [12:37:14] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: builds-api: bump to 0.0.220-20260804122801-4e357e0a [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1366 [12:37:16] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: builds-api: bump to 0.0.220-20260804122801-4e357e0a [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1366 [12:52:34] (03PS2) 10Majavah: vps: create_instance: Add flag to specify number of first server [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1308683 [12:52:34] (03PS1) 10Majavah: vps: create_instance: Add codfw1dev support [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1320917 [12:55:49] (03update) 10raymond-ndibe: tests: improve test coverage [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/362 [12:55:51] (03CR) 10CI reject: [V:04-1] vps: create_instance: Add codfw1dev support [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1320917 (owner: 10Majavah) [12:56:48] (03update) 10raymond-ndibe: tests: refactors, improvements and testing some untested branches [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/362 [12:58:10] (03PS2) 10Majavah: vps: create_instance: Add codfw1dev support [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1320917 [12:58:19] !log taavi@cloudcumin1001 cloudinfra-codfw1dev START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'ntp' [12:58:20] taavi@cloudcumin1001: Unknown project "cloudinfra-codfw1dev" [13:01:25] (03PS1) 10Alien4444: EditCounter: fix local-time breaking timecard [labs/xtools] - 10https://gerrit.wikimedia.org/r/1320920 (https://phabricator.wikimedia.org/T433394) [13:02:02] (03CR) 10CI reject: [V:04-1] EditCounter: fix local-time breaking timecard [labs/xtools] - 10https://gerrit.wikimedia.org/r/1320920 (https://phabricator.wikimedia.org/T433394) (owner: 10Alien4444) [13:02:35] 10Cloud-VPS (Debian Bullseye Deprecation), 10Wikimedia-Developer-Portal: Update Dev Portal staging to Debian Bookworm - https://phabricator.wikimedia.org/T429490#12182995 (10KBach) @Aklapper it seems to be working, thank you! [13:03:24] !log taavi@cloudcumin1001 cloudinfra-codfw1dev END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'ntp' [13:03:26] taavi@cloudcumin1001: Unknown project "cloudinfra-codfw1dev" [13:04:39] 10Cloud-VPS (Debian Bullseye Deprecation), 06tools-infrastructure-team: Replace all codfw1dev Bullseye VMs - https://phabricator.wikimedia.org/T401810#12183005 (10taavi) [13:07:25] !log taavi@cloudcumin1001 cloudinfra-codfw1dev START - Cookbook wmcs.vps.remove_instance for instance ntp-2 [13:07:25] taavi@cloudcumin1001: Unknown project "cloudinfra-codfw1dev" [13:07:50] !log taavi@cloudcumin1001 cloudinfra-codfw1dev END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance ntp-2 [13:07:50] taavi@cloudcumin1001: Unknown project "cloudinfra-codfw1dev" [13:08:23] !log taavi@cloudcumin1001 cloudinfra-codfw1dev START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'enc' [13:08:24] taavi@cloudcumin1001: Unknown project "cloudinfra-codfw1dev" [13:08:55] 10Cloud-VPS (Quota-requests), 10Wikidata: Quota increase request for project pm20database - https://phabricator.wikimedia.org/T433839#12183027 (10Andrew) Hello @bking -- can we get a bit of a history lesson here? It looks like this project was created as a database container for toolforge in T359785 -- typical... [13:11:34] (03PS2) 10Alien4444: EditCounter: fix local-time breaking timecard [labs/xtools] - 10https://gerrit.wikimedia.org/r/1320920 (https://phabricator.wikimedia.org/T433394) [13:14:35] !log taavi@cloudcumin1001 cloudinfra-codfw1dev END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'enc' [13:14:36] taavi@cloudcumin1001: Unknown project "cloudinfra-codfw1dev" [13:16:59] 10Cloud-VPS (Quota-requests), 10Wikidata: Quota increase request for project pm20database - https://phabricator.wikimedia.org/T433839#12183092 (10bking) @Andrew Apologies, I'm showing my ignorance of WMCS/Toolforge. If Toolforge is the best place for this, we are happy to run it there (that's based on the comm... [13:18:21] 10Cloud-VPS (Debian Bullseye Deprecation), 06tools-infrastructure-team, 13Patch-For-Review: Replace all codfw1dev Bullseye VMs - https://phabricator.wikimedia.org/T401810#12183096 (10taavi) a:03taavi [13:20:56] 10Cloud-VPS (Quota-requests): Quota increase request for project wlm-it-visual - https://phabricator.wikimedia.org/T427731#12183116 (10Andrew) Hi Folks! Just fyi, the oom-killer is firing today on the wlm-staging-app host. This showed up while I was checking another issue and I thought you would like to know! [13:26:10] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer [13:29:45] 10Tool-clip2commons: Silent encode produces audio-only output for macOS Photos-edited videos - https://phabricator.wikimedia.org/T433972 (10Dev) 03NEW [13:34:00] 10Tool-clip2commons: Silent encode produces audio-only output for macOS Photos-edited videos - https://phabricator.wikimedia.org/T433972#12183239 (10Dev) [13:42:25] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer [13:51:32] (03update) 10raymond-ndibe: tests: refactors, improvements and testing some untested branches [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/362 [13:51:40] !log taavi@cloudcumin1001 cloudinfra-codfw1dev START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'acme-chief' [13:51:42] taavi@cloudcumin1001: Unknown project "cloudinfra-codfw1dev" [13:52:37] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer [13:52:40] (03update) 10raymond-ndibe: tests: refactors, improvements and testing some untested branches [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/362 [13:57:54] !log taavi@cloudcumin1001 cloudinfra-codfw1dev END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'acme-chief' [13:57:55] taavi@cloudcumin1001: Unknown project "cloudinfra-codfw1dev" [13:58:18] !log taavi@cloudcumin1001 cloudinfra-codfw1dev START - Cookbook wmcs.vps.remove_instance for instance cloudinfra-dbbackup-1 [13:58:19] taavi@cloudcumin1001: Unknown project "cloudinfra-codfw1dev" [13:58:25] !log taavi@cloudcumin1001 cloudinfra-codfw1dev END (FAIL) - Cookbook wmcs.vps.remove_instance (exit_code=99) for instance cloudinfra-dbbackup-1 [13:58:26] taavi@cloudcumin1001: Unknown project "cloudinfra-codfw1dev" [14:01:42] (03PS1) 10Alien4444: Repository: use tables not schemata in getDbList [labs/xtools] - 10https://gerrit.wikimedia.org/r/1320944 (https://phabricator.wikimedia.org/T344090) [14:05:04] !log taavi@cloudcumin1001 cloudinfra-codfw1dev START - Cookbook wmcs.vps.remove_instance for instance cloudinfra-dbbackup-1 [14:05:12] taavi@cloudcumin1001: Unknown project "cloudinfra-codfw1dev" [14:05:53] !log taavi@cloudcumin1001 cloudinfra-codfw1dev END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance cloudinfra-dbbackup-1 [14:05:54] taavi@cloudcumin1001: Unknown project "cloudinfra-codfw1dev" [14:06:02] !log taavi@cloudcumin1001 cloudinfra-codfw1dev START - Cookbook wmcs.vps.remove_instance for instance acme-chief-2 [14:06:03] taavi@cloudcumin1001: Unknown project "cloudinfra-codfw1dev" [14:06:24] !log taavi@cloudcumin1001 cloudinfra-codfw1dev END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance acme-chief-2 [14:06:24] taavi@cloudcumin1001: Unknown project "cloudinfra-codfw1dev" [14:10:58] !log taavi@cloudcumin1001 tools-codfw1dev START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-codfw1dev-k8s-worker' [14:11:00] taavi@cloudcumin1001: Unknown project "tools-codfw1dev" [14:12:45] (03open) 10raymond-ndibe: models: move cmd field out of CommonJob into job-type models [repos/cloud/toolforge/jobs-api] (increase_test_coverage) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/364 [14:14:40] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer [14:16:54] !log taavi@cloudcumin1001 tools-codfw1dev END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-codfw1dev-k8s-worker' [14:16:55] taavi@cloudcumin1001: Unknown project "tools-codfw1dev" [14:17:52] !log taavi@cloudcumin1001 tools-codfw1dev START - Cookbook wmcs.vps.remove_instance for instance tools-codfw1dev-k8s-worker-2 [14:17:53] taavi@cloudcumin1001: Unknown project "tools-codfw1dev" [14:18:08] !log taavi@cloudcumin1001 tools-codfw1dev END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-codfw1dev-k8s-worker-2 [14:18:08] taavi@cloudcumin1001: Unknown project "tools-codfw1dev" [14:18:17] !log taavi@cloudcumin1001 cloudinfra-codfw1dev START - Cookbook wmcs.vps.remove_instance for instance ntp-1 [14:18:17] taavi@cloudcumin1001: Unknown project "cloudinfra-codfw1dev" [14:19:06] !log taavi@cloudcumin1001 cloudinfra-codfw1dev END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance ntp-1 [14:19:06] taavi@cloudcumin1001: Unknown project "cloudinfra-codfw1dev" [14:20:19] 10Cloud-VPS (Debian Bullseye Deprecation), 06tools-infrastructure-team, 13Patch-For-Review: Replace all codfw1dev Bullseye VMs - https://phabricator.wikimedia.org/T401810#12183558 (10taavi) a:05taavi→03None Updated list: ` taavi@cloudcontrol2010-dev ~ $ os server list --all-projects | grep bullseye | 858... [14:20:47] (03update) 10andrew: cert-manager: update deploy to 1.20.3 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1355 (https://phabricator.wikimedia.org/T408786) [14:21:23] (03update) 10andrew: cert-manager: update deploy to 1.20.3 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1355 (https://phabricator.wikimedia.org/T408786) [14:25:40] FIRING: SystemdUnitFailed: remove_dangling_cinder_snapshots.service on cloudbackup2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [14:26:08] (03approved) 10dcaro: jobs-emailer: bump to 0.0.92-20260804122721-05c10f0c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1362 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:26:12] (03merge) 10dcaro: jobs-emailer: bump to 0.0.92-20260804122721-05c10f0c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1362 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:26:17] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-api [14:30:57] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api [14:32:27] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component builds-api [14:35:55] (03update) 10raymond-ndibe: models: move cmd field out of CommonJob into job-type models [repos/cloud/toolforge/jobs-api] (increase_test_coverage) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/364 [14:35:56] (03open) 10dcaro: global: add log write endpoint [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/32 [14:37:55] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api [14:41:48] (03update) 10raymond-ndibe: refactor: move observability fields out of CommonJob into ObservabilityJob [repos/cloud/toolforge/jobs-api] (increase_test_coverage) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/363 [14:42:08] (03update) 10raymond-ndibe: refactor: move observability fields out of CommonJob into ObservabilityJob [repos/cloud/toolforge/jobs-api] (move_cmd_out_of_common_job) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/363 [14:42:21] (03update) 10raymond-ndibe: refactor: move observability fields out of CommonJob into ObservabilityJob [repos/cloud/toolforge/jobs-api] (move_cmd_out_of_common_job) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/363 [14:42:26] (03update) 10raymond-ndibe: refactor: move observability fields out of CommonJob into ObservabilityJob [repos/cloud/toolforge/jobs-api] (move_cmd_out_of_common_job) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/363 [14:42:34] (03open) 10andrew: remove "wait: true" from certmanager ansible script [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/335 (https://phabricator.wikimedia.org/T408786) [14:51:18] (03approved) 10dcaro: builds-api: bump to 0.0.220-20260804122801-4e357e0a [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1366 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:51:21] (03update) 10dcaro: builds-api: bump to 0.0.220-20260804122801-4e357e0a [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1366 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:51:31] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component envvars-api [14:51:50] (03merge) 10dcaro: builds-api: bump to 0.0.220-20260804122801-4e357e0a [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1366 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:52:32] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api [14:54:14] (03update) 10dcaro: global: upgrade pre-commits [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/177 [14:54:40] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component envvars-api [14:55:34] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api [14:55:48] (03approved) 10dcaro: envvars-api: bump to 0.0.88-20260804120640-713d3bb6 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1364 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:55:54] (03update) 10dcaro: envvars-api: bump to 0.0.88-20260804120640-713d3bb6 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1364 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:56:17] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-builder [14:56:29] (03merge) 10dcaro: envvars-api: bump to 0.0.88-20260804120640-713d3bb6 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1364 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:56:40] (03update) 10dcaro: builds-builder: bump to 0.0.152-20260804120440-39fb286a [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1361 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [15:00:24] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder [15:01:26] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component builds-builder [15:06:30] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder [15:07:11] (03update) 10andrew: cert-manager: update deploy to 1.20.3 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1355 (https://phabricator.wikimedia.org/T408786) [15:12:04] 10Toolforge, 06tools-platform-team, 07Documentation: List of best practices for Toolforge tools - https://phabricator.wikimedia.org/T178836#12183859 (10TBurmeister) 05Open→03Resolved a:03TBurmeister I believe https://wikitech.wikimedia.org/wiki/Help:Toolforge/Developing_successful_tools and the lar... [15:13:24] (03update) 10raymond-ndibe: models: support webservice job-type [repos/cloud/toolforge/jobs-api] (models_refactor) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/264 (https://phabricator.wikimedia.org/T428898) [15:13:25] (03update) 10raymond-ndibe: models: support webservice job-type [repos/cloud/toolforge/jobs-api] (models_refactor) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/264 (https://phabricator.wikimedia.org/T428898) [15:14:41] (03update) 10renovatebot: Update pnpm to v11.20.0 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/9 [15:14:44] (03open) 10renovatebot: Update pnpm to v11.20.0 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/9 [15:15:00] (03update) 10renovatebot: Update pnpm to v11.20.0 [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/22 [15:15:03] (03open) 10renovatebot: Update pnpm to v11.20.0 [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/22 [15:15:19] (03update) 10renovatebot: Update pnpm to v11.20.0 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/16 [15:15:26] (03open) 10renovatebot: Update pnpm to v11.20.0 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/16 [15:16:24] (03update) 10renovatebot: Update pnpm to v11.20.0 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/48 [15:16:28] (03open) 10renovatebot: Update pnpm to v11.20.0 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/48 [15:17:29] (03update) 10renovatebot: Update pnpm to v11.19.0 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/30 [15:17:32] (03update) 10renovatebot: Update pnpm to v11.20.0 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/30 [15:18:39] (03approved) 10dcaro: builds-builder: bump to 0.0.152-20260804120440-39fb286a [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1361 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [15:18:42] (03merge) 10dcaro: builds-builder: bump to 0.0.152-20260804120440-39fb286a [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1361 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [15:26:35] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api [15:26:36] (03update) 10dcaro: components-api: bump to 0.0.204-20260804120721-f816fd58 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1363 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [15:30:55] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api [15:31:14] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component components-api [15:32:17] (03approved) 10raymond-ndibe: global: upgrade pre-commits [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/177 (owner: 10dcaro) [15:34:46] 10Toolforge, 06tools-platform-team, 07OKR-Work: [loki,alloy,logs-api] reconfigure the labels and add metadata - https://phabricator.wikimedia.org/T432969#12183954 (10TLepage-WMF) Proposed checklist: - [ ] With our new approach of having a single label (`source = run-logs`) for all logs (still split by tool t... [15:36:01] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api [15:36:24] 06cloud-services-team, 10Toolforge, 06tools-platform-team, 07Documentation: Update and Improve Toolforge and Cloud VPS Technical Documentation - https://phabricator.wikimedia.org/T203131#12183971 (10TBurmeister) [15:39:34] (03approved) 10dcaro: components-api: bump to 0.0.204-20260804120721-f816fd58 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1363 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [15:39:37] (03merge) 10dcaro: components-api: bump to 0.0.204-20260804120721-f816fd58 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1363 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [15:57:55] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: Regression: Images built with the build service images do no longer contain/configure locales specified in .locales - https://phabricator.wikimedia.org/T428230#12184085 (10Raymond_Ndibe) 05In progress→03Resolved [16:05:41] (03PS1) 10Andrew Bogott: Add copy_certmanager_images_to_registry cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1320996 (https://phabricator.wikimedia.org/T408786) [16:07:49] !log andrew@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.copy_certmanager_images_to_registry for certmanager 1.20.3 (T408786) [16:07:54] T408786: Upgrade cert-manager past 1.15 - https://phabricator.wikimedia.org/T408786 [16:08:43] (03CR) 10CI reject: [V:04-1] Add copy_certmanager_images_to_registry cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1320996 (https://phabricator.wikimedia.org/T408786) (owner: 10Andrew Bogott) [16:08:45] !log andrew@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.copy_certmanager_images_to_registry (exit_code=0) for certmanager 1.20.3 (T408786) [16:12:31] 10Tool-Gerrit-Patch-Uploader: Gerrit patch uploader should use standard image thumbnail sizes - https://phabricator.wikimedia.org/T433997 (10taavi) 03NEW [16:14:56] 06cloud-services-team, 10PAWS, 06tools-platform-team: PAWS disk space is running out - https://phabricator.wikimedia.org/T429578#12184181 (10aputhin) 05In progress→03Resolved Closing for now as we have cleaned up and also automated the clean up script on T433572 [16:16:06] (03update) 10renovatebot: Update pnpm to v11.20.0 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/30 [16:19:01] 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team (Doing 😎): Planning for a plan to migrate Beta Cluster again - https://phabricator.wikimedia.org/T433998 (10bd808) 03NEW [16:22:51] (03PS2) 10Andrew Bogott: Add copy_certmanager_images_to_registry cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1320996 (https://phabricator.wikimedia.org/T408786) [16:26:54] (03PS3) 10Andrew Bogott: Add copy_certmanager_images_to_registry cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1320996 (https://phabricator.wikimedia.org/T408786) [16:38:08] (03update) 10dcaro: global: add log write endpoint [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/32 [16:38:20] (03PS4) 10Andrew Bogott: Add copy_certmanager_images cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1320996 (https://phabricator.wikimedia.org/T408786) [16:38:41] !log andrew@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.copy_certmanager_images for certmanager 1.20.3 (T408786) [16:38:46] T408786: Upgrade cert-manager past 1.15 - https://phabricator.wikimedia.org/T408786 [16:38:59] !log andrew@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.copy_certmanager_images (exit_code=0) for certmanager 1.20.3 (T408786) [16:43:06] 10Cloud-VPS, 06tools-infrastructure-team, 10Research-collaborations, 13Patch-For-Review: Change mount propagation for /mnt/nfs in wikinav-bookworm.research-collaborations-api - https://phabricator.wikimedia.org/T433668#12184320 (10fkaelin) The service has been redeployed with this [[ https://gitlab.wikimed... [16:49:56] 10Tool-itwiki, 10MediaWiki-Action-API, 06MediaWiki-API-Platform-Team: Missing categories from generator=categorymembers (just 1 instead of 2) in itwiki - https://phabricator.wikimedia.org/T433922#12184370 (10Umherirrender) The category is shown with a higher limit for `prop=categories` (`cllimit=500`, that w... [17:16:58] 10Tool-wikinewsie: WikiPortraits mode for Wikinewsie - https://phabricator.wikimedia.org/T432573#12184483 (10Pharos) Yeah, more outlets is great! I guess I was thinking less of the Wikimedia Commons category per se, and more of whatever platform could host a narrative story connected to a set of photographs, whi... [17:19:10] (03update) 10andrew: cert-manager: update deploy to 1.20.3 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1355 (https://phabricator.wikimedia.org/T408786) [17:20:00] (03update) 10andrew: cert-manager: update deploy to 1.20.3 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1355 (https://phabricator.wikimedia.org/T408786) [17:21:55] 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team (Doing 😎): Planning for a plan to migrate Beta Cluster again - https://phabricator.wikimedia.org/T433998#12184511 (10bd808) I would like us to try adopting the {T394316} idea as a core part of the migration.... [17:26:25] 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team (Doing 😎): Planning for a plan to migrate Beta Cluster again - https://phabricator.wikimedia.org/T433998#12184537 (10bd808) 05Open→03In progress @Southparkfan reached out to me yesterday to ask if they mi... [18:01:50] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate WMCS-managed NFS servers off of Bullseye - https://phabricator.wikimedia.org/T401812#12184680 (10Andrew) >>! In T401812#12182273, @taavi wrote: > The VMs are still there, just in a shutoff status no longer! [18:10:03] FIRING: PuppetStaleCertificates: Found non-revoked Puppet certificates for 1 deleted instances on paws-puppetserver-1 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [18:25:40] FIRING: SystemdUnitFailed: remove_dangling_cinder_snapshots.service on cloudbackup2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [18:27:04] 10Cloud-VPS, 06tools-infrastructure-team: Improve VM backup performance - https://phabricator.wikimedia.org/T434014 (10Andrew) 03NEW [18:29:08] (03CR) 10Majavah: [C:03+2] Update dev environment and its documentation (031 comment) [labs/striker] - 10https://gerrit.wikimedia.org/r/1319553 (https://phabricator.wikimedia.org/T355525) (owner: 10Perryprog) [18:31:45] 10Cloud-VPS, 06tools-infrastructure-team: Improve VM backup performance - https://phabricator.wikimedia.org/T434014#12184839 (10Andrew) The first thing to experiment with is the ceph throttle. Before I do that I'd like to get solid baseline backup times after a recent rebalancing change, and also get a clearer... [18:32:26] (03Merged) 10jenkins-bot: Update dev environment and its documentation [labs/striker] - 10https://gerrit.wikimedia.org/r/1319553 (https://phabricator.wikimedia.org/T355525) (owner: 10Perryprog) [18:37:48] 10Striker, 06tools-platform-team, 13Patch-For-Review: Striker dev env gitlab root credentials do not work - https://phabricator.wikimedia.org/T355525#12184866 (10Perryprog) 05Open→03Resolved [18:38:41] FIRING: CloudVPSDesignateLeaks: Detected 8 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [18:40:03] RESOLVED: PuppetStaleCertificates: Found non-revoked Puppet certificates for 1 deleted instances on paws-puppetserver-1 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [18:50:41] 06cloud-services-team, 10Cloud-VPS (Debian Bullseye Deprecation), 06Data-Engineering (Q1 FS26/27 July 1st - September 30th), 10Event-Platform: Upgrade (replace) beta|deployment-prep event platform VPS instances - https://phabricator.wikimedia.org/T429497#12184900 (10Ottomata) FWIW: https://wikitech.wikime... [19:05:25] RESOLVED: SystemdUnitFailed: remove_dangling_cinder_snapshots.service on cloudbackup2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [19:05:57] 10VPS-project-Extdist, 10ExtensionDistributor: extdist servers need to be upgraded - https://phabricator.wikimedia.org/T429659#12184959 (10Bawolff) 05Open→03Resolved a:03Bawolff Done [19:12:07] 10Cloud-VPS (Quota-requests), 06Moderator-Tools-Team, 10WikiWho: Remove increased quota for project wikiwho - https://phabricator.wikimedia.org/T434023 (10MusikAnimal) 03NEW [19:17:42] 10Tool-wiki-gender-stats: Merge data and snapshot dates fetching methods - https://phabricator.wikimedia.org/T434025 (10Danya) 03NEW [19:17:45] 10Tool-wiki-gender-stats: Merge data and snapshot dates fetching methods - https://phabricator.wikimedia.org/T434025#12185016 (10Danya) p:05Triage→03High [19:17:55] FIRING: [2x] SystemdUnitFailed: remove_dangling_cinder_snapshots.service on cloudbackup2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [19:17:58] 10Tool-wiki-gender-stats: Merge data and snapshot dates fetching methods - https://phabricator.wikimedia.org/T434025#12185019 (10Danya) [19:18:09] FIRING: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [19:18:42] 10Tool-wiki-gender-stats: Merge data and snapshot dates fetching methods - https://phabricator.wikimedia.org/T434025#12185021 (10Danya) [19:18:43] 10Tool-wiki-gender-stats: [VERSION] 0.4.0 - https://phabricator.wikimedia.org/T432521#12185022 (10Danya) [19:40:39] RESOLVED: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [20:01:24] 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: cloudceph HEALTH_WARN, multiple OSD(s) experiencing slow operations in BlueStore - https://phabricator.wikimedia.org/T429387#12185113 (10Andrew) I've applied the attached patches and am now rebooting everything to pick up config changes. [20:01:31] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.roll_reboot_osds (T429387) [20:01:38] T429387: cloudceph HEALTH_WARN, multiple OSD(s) experiencing slow operations in BlueStore - https://phabricator.wikimedia.org/T429387 [20:03:15] !log andrew@cloudcumin1001 admin END (ERROR) - Cookbook wmcs.ceph.roll_reboot_osds (exit_code=97) (T429387) [20:04:54] (03CR) 10Andrew Bogott: [C:03+2] roll_reboot_osds: set maintenance per-osd rather than for the whole run [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1293792 (https://phabricator.wikimedia.org/T427295) (owner: 10Andrew Bogott) [20:05:11] (03CR) 10Andrew Bogott: [C:03+2] is_cluster_in_maintenance: ignore muted checks [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1318216 (owner: 10Andrew Bogott) [20:06:59] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.roll_reboot_osds (T429387) [20:07:05] T429387: cloudceph HEALTH_WARN, multiple OSD(s) experiencing slow operations in BlueStore - https://phabricator.wikimedia.org/T429387 [20:13:38] 10Tool-wmf-openapi-linter, 06Tech-Docs-Team, 06MediaWiki-API-Platform-Team (MediaWiki-Interfaces-Sprints), 05MW-1.47-notes (1.47.0-wmf.13; 2026-07-28), 07OKR-Work: Fix description and example issues for page, revision, search, and file paths in the Media... - https://phabricator.wikimedia.org/T428153#12185129 [20:15:45] 10Tool-wmf-openapi-linter, 06Tech-Docs-Team, 06MediaWiki-API-Platform-Team (MediaWiki-Interfaces-Sprints), 05MW-1.47-notes (1.47.0-wmf.13; 2026-07-28), 07OKR-Work: Fix description and example issues for page, revision, search, and file paths in the Media... - https://phabricator.wikimedia.org/T428153#12185139 [20:18:09] FIRING: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [20:18:41] RESOLVED: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [20:19:10] RESOLVED: SystemdUnitFailed: ceph-osd@61.servic.service on cloudcephosd1016:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [20:20:40] 10Tool-wmf-openapi-linter, 06MediaWiki-API-Platform-Team (MediaWiki-Interfaces-Sprints), 07OKR-Work, 13Patch-For-Review: Fix top-level issues in the MediaWiki REST API OAD - https://phabricator.wikimedia.org/T428147#12185149 (10AGhirelli-WMF) 05In progress→03Resolved [20:41:43] 10Tool-wiki-gender-stats: Merge data and snapshot dates fetching methods - https://phabricator.wikimedia.org/T434025#12185217 (10Danya) [20:42:55] 10Tool-wiki-gender-stats: Notify maintainers when snapshot capture fails - https://phabricator.wikimedia.org/T424279#12185220 (10Danya) [20:43:09] RESOLVED: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [20:48:09] FIRING: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [20:58:50] 10Tool-wmf-openapi-linter, 06Tech-Docs-Team, 06MediaWiki-API-Platform-Team (MediaWiki-Interfaces-Sprints), 05MW-1.47-notes (1.47.0-wmf.13; 2026-07-28), 07OKR-Work: Fix description and example issues for page, revision, search, and file paths in the Media... - https://phabricator.wikimedia.org/T428153#12185251 [21:03:09] RESOLVED: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [21:05:09] FIRING: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [21:10:25] 10Cloud-VPS (Quota-requests), 10Wikidata: Quota increase request for project pm20database - https://phabricator.wikimedia.org/T433839#12185276 (10Raymond_Ndibe) @bking you can request for toolforge access here https://toolsadmin.wikimedia.org/register/ to begin experimenting with toolforge [21:10:56] 10Tool-wmf-openapi-linter, 06Content-Platform-Team (Work In Progress), 06MediaWiki-API-Platform-Team (MediaWiki-Interfaces-Sprints), 07OKR-Work, 13Patch-For-Review: Add parameter examples to TransformHandler.php - https://phabricator.wikimedia.org/T432228#12185277 (10AGhirelli-WMF) Uploaded PS 2, removin... [21:25:09] RESOLVED: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [21:31:09] FIRING: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [21:36:09] RESOLVED: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [21:41:47] FIRING: NodeDown: Node cloudcephosd1036 is down. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NodeDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcephosd1036 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [21:42:09] FIRING: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [21:46:47] RESOLVED: NodeDown: Node cloudcephosd1036 is down. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NodeDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcephosd1036 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [21:47:09] RESOLVED: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [21:49:09] FIRING: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [22:05:31] (03update) 10danyya: Draft: Resolve T433199 "[TF] Compute aggregated stats for each wiki project version" [toolforge-repos/wiki-gender-stats] (develop) - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-gender-stats/-/merge_requests/96 [23:02:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [23:07:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [23:30:35] (03CR) 10Ladsgroup: [C:03+2] "I try to do the upgrade soon 😄" [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1267343 (owner: 10Krinkle) [23:38:04] (03Merged) 10jenkins-bot: hound: Upgrade Hound from v0.4.0 to v0.7.1 [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1267343 (owner: 10Krinkle) [23:47:11] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.roll_reboot_osds (exit_code=99) (T429387) [23:47:21] T429387: cloudceph HEALTH_WARN, multiple OSD(s) experiencing slow operations in BlueStore - https://phabricator.wikimedia.org/T429387