[00:05:48] FIRING: [2x] PuppetFailure: Puppet has failed on clouddb1024:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [00:10:25] FIRING: [4x] SystemdUnitFailed: wmf-pt-kill@s4.service on clouddb1024:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [00:13:47] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/16 [00:13:54] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/28 [00:14:00] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/23 [00:14:54] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/55 [00:15:33] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/40 [01:07:09] 10Tool-wikinewsie: Excluded categories not working in Wikinewsie custom portals - https://phabricator.wikimedia.org/T435710#12310655 (10ZhaoFJx) a:03ZhaoFJx [01:07:18] 10Tool-wikinewsie: Excluded categories not working in Wikinewsie custom portals - https://phabricator.wikimedia.org/T435710#12310657 (10ZhaoFJx) 05Open→03In progress [01:19:08] 10Tool-wikinewsie: Excluded categories not working in Wikinewsie custom portals - https://phabricator.wikimedia.org/T435710#12310666 (10ZhaoFJx) 05In progress→03Resolved [02:05:25] FIRING: [5x] SystemdUnitFailed: wmf-pt-kill@s4.service on clouddb1024:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [03:23:54] 10Tool-projo: Projo: Retain actionable pageviews failure diagnostics after collector jobs finish - https://phabricator.wikimedia.org/T437671 (10Audiodude) 03NEW [04:06:03] FIRING: [2x] PuppetFailure: Puppet has failed on clouddb1024:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [04:14:11] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/16 [04:14:13] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/16 [04:14:27] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/28 [04:14:29] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/28 [04:14:41] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/23 [04:14:43] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/23 [04:15:24] 10Tool-projo: Address performance issues of NPP review page - https://phabricator.wikimedia.org/T437673 (10Audiodude) 03NEW [04:15:52] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/55 [04:15:53] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/55 [04:17:24] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/40 [04:17:26] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/40 [05:04:13] 10Tool-projo: Projo: Move NPP snapshot state into MariaDB and remove NFS publication locking - https://phabricator.wikimedia.org/T437544#12310804 (10Audiodude) 05Open→03Resolved a:03Audiodude [05:13:54] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/16 [05:14:03] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/28 [05:14:10] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/23 [05:15:10] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/55 [05:15:49] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/40 [05:37:04] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06DBA, 13Patch-For-Review: Productionize new clouddb* hosts (clouddb1022-1033) - https://phabricator.wikimedia.org/T409557#12310831 (10Marostegui) clouddb1025 is now running x4 - it is not yet ready to hold views, I need to make sure ever... [06:05:40] FIRING: [5x] SystemdUnitFailed: wmf-pt-kill@s4.service on clouddb1024:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [06:14:03] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/16 [06:14:16] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/28 [06:14:28] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/23 [06:15:28] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/55 [06:16:50] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/40 [06:35:48] FIRING: [2x] PuppetFailure: Puppet has failed on clouddb1024:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [06:40:25] FIRING: [5x] SystemdUnitFailed: wmf-pt-kill@s4.service on clouddb1024:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [06:50:48] RESOLVED: [2x] PuppetFailure: Puppet has failed on clouddb1024:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [07:16:37] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/80 [07:16:39] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/80 [07:18:04] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/42 [07:18:05] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/42 [07:18:30] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/28 [07:18:31] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/28 [07:18:42] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/30 [07:18:43] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/30 [07:18:50] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewikisignbot] - 10https://gitlab.wikimedia.org/toolforge-repos/dewikisignbot/-/merge_requests/33 [07:18:53] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewikisignbot] - 10https://gitlab.wikimedia.org/toolforge-repos/dewikisignbot/-/merge_requests/33 [07:25:25] FIRING: [4x] SystemdUnitFailed: wmf-pt-kill@s4.service on clouddb1024:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:27:47] 10Tool-wikinewsie: i18n: add call for translatewiki.net translation - https://phabricator.wikimedia.org/T437683 (10Yug) 03NEW [08:29:36] 10Tool-wikinewsie: i18n: add call for translatewiki.net translation - https://phabricator.wikimedia.org/T437683#12311116 (10Yug) [08:32:37] 10Tool-wikinewsie: i18n: add call for translatewiki.net translation - https://phabricator.wikimedia.org/T437683#12311129 (10Yug) [08:41:38] (03merge) 10tlepage: refactor: Generalize job event models [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/38 (https://phabricator.wikimedia.org/T432566) [08:41:40] (03update) 10tlepage: refactor: Generalize Loki push format [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/41 (https://phabricator.wikimedia.org/T432566) [08:45:49] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: logs-api: bump to 0.0.48-20260911084153-e8e7be4a [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1417 [08:45:50] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: logs-api: bump to 0.0.48-20260911084153-e8e7be4a [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1417 [08:46:51] !log tlepage@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component logs-api [08:57:24] !log tlepage@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api [08:58:01] !log tlepage@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component logs-api [09:08:14] !log tlepage@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api [09:09:11] (03merge) 10tlepage: logs-api: bump to 0.0.48-20260911084153-e8e7be4a [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1417 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [09:09:13] (03update) 10tlepage: logs-api: bump to 0.0.48-20260911084153-e8e7be4a [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1417 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [09:13:44] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/16 [09:13:50] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/28 [09:13:56] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/23 [09:14:48] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/55 [09:15:07] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/80 [09:15:23] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/40 [09:15:52] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/42 [09:16:08] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/28 [09:16:12] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/30 [09:16:16] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewikisignbot] - 10https://gitlab.wikimedia.org/toolforge-repos/dewikisignbot/-/merge_requests/33 [09:18:03] 06tools-platform-team: Decision request - tools-platform internal review workflow - https://phabricator.wikimedia.org/T437573#12311251 (10aputhin) 05Open→03In progress a:05dcaro→03aputhin Yep, tho I don't think this needs to be a "decision request"? It's more formalizing our internal discussions. [09:33:17] 10Cloud-VPS, 06tools-infrastructure-team: Advice for Hybrid Architecture for AI Tool - https://phabricator.wikimedia.org/T416590#12311304 (10Odinaldo) This is very helpful @Andrew and we will follow that advice. [09:43:51] (03update) 10tlepage: events: Add the GET /events endpoint [repos/cloud/toolforge/logs-api] (loki-refactor-for-get-events) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/40 (https://phabricator.wikimedia.org/T432566) [09:43:58] (03update) 10tlepage: refactor: Parse Loki result streams into events [repos/cloud/toolforge/logs-api] (generalize-loki-push-format) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/39 (https://phabricator.wikimedia.org/T432566) [09:44:17] 06tools-infrastructure-team, 10Ceph, 06Data-Platform-SRE (2026-08-28 - 2026-09-18), 13Patch-For-Review: ceph: Fix the CephX key guard so that Puppet can correctly update the key material - https://phabricator.wikimedia.org/T437233#12311316 (10BTullis) I have written and deployed the check script: `/usr/loc... [09:45:25] (03update) 10tlepage: refactor: Parse Loki result streams into logs [repos/cloud/toolforge/logs-api] (generalize-loki-push-format) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/39 (https://phabricator.wikimedia.org/T432566) [09:46:50] 06tools-infrastructure-team, 10Ceph, 06Data-Platform-SRE (2026-08-28 - 2026-09-18), 13Patch-For-Review: ceph: Fix the CephX key guard so that Puppet can correctly update the key material - https://phabricator.wikimedia.org/T437233#12311325 (10BTullis) Here is the output when running the script on the two W... [09:48:24] (03update) 10tlepage: events: Add the GET /{tool}/logs endpoint [repos/cloud/toolforge/logs-api] (loki-refactor-for-get-events) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/40 (https://phabricator.wikimedia.org/T432566) [09:51:01] 06tools-infrastructure-team, 10Ceph, 06Data-Platform-SRE (2026-08-28 - 2026-09-18), 13Patch-For-Review: ceph: Fix the CephX key guard so that Puppet can correctly update the key material - https://phabricator.wikimedia.org/T437233#12311328 (10BTullis) p:05Triage→03High [10:03:35] 06tools-infrastructure-team, 10Ceph, 06Data-Platform-SRE (2026-08-28 - 2026-09-18), 13Patch-For-Review: ceph: Fix the CephX key guard so that Puppet can correctly update the key material - https://phabricator.wikimedia.org/T437233#12311364 (10BTullis) [10:27:59] 10Striker, 06tools-platform-team, 10GitLab (Pipeline Services Migration🐤): Migrate striker to GitLab - https://phabricator.wikimedia.org/T344765#12311481 (10Nikerabbit) [10:29:47] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 10MediaWiki-extensions-OAuth, and 3 others: Create a view of oauth_registered_consumer table in cloud - https://phabricator.wikimedia.org/T247800#12311494 (10Bugreporter) [11:12:22] 10Data-Services, 06tools-platform-team, 13Patch-For-Review, 07User-notice: Update Wiki Replicas documentation for x4 rollout - https://phabricator.wikimedia.org/T437208#12311689 (10taavi) Hi, thanks! >>! In T437208#12304453, @STei-WMF wrote: > This is how I have interpreted your instruction as an entry: >... [11:25:40] FIRING: [4x] SystemdUnitFailed: wmf-pt-kill@s4.service on clouddb1024:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:27:54] 06tools-infrastructure-team, 07Epic, 06Test Platform (Epics): Explore if there are ways to tune the underlying hardware and configuration for the CI agents - https://phabricator.wikimedia.org/T437407#12311919 (10Peter) > Is there also any kind of view that shows how long CI tests are "waiting" for the queue... [12:43:13] 10Toolforge, 06tools-infrastructure-team, 06tools-platform-team, 07Incident Severity 3, 07Wikimedia-Incident: 2026-07-11 Incident - Toolforge haproxies flapping - https://phabricator.wikimedia.org/T431946#12311956 (10MLechvien-WMF) Looks good to me, thanks David [13:00:05] 10Cloud-VPS, 10Data-Services, 06tools-infrastructure-team, 06Data-Platform-SRE (2026-08-28 - 2026-09-18): Export additional nfsd stats from /proc/fs/nfsd - https://phabricator.wikimedia.org/T436252#12311974 (10Gehel) [13:00:12] 10Cloud-VPS, 10Data-Services, 06tools-infrastructure-team, 06tools-platform-team, and 4 others: Plan to make clouddumps more resilient and easier to operate - https://phabricator.wikimedia.org/T411248#12311975 (10Gehel) [13:10:41] 06cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE, 13Patch-For-Review: Q3:rack/setup/install cloudcephosd105[3456] - https://phabricator.wikimedia.org/T419892#12312045 (10elukey) @Andrew my understanding is that cloudcephosd[1048-52] shouldn't be able to reimage with the current settings. If... [14:14:06] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/16 [14:14:11] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/16 [14:14:20] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/28 [14:14:22] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/28 [14:14:32] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/23 [14:14:34] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/23 [14:16:09] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/55 [14:16:15] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/55 [14:16:31] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/80 [14:16:32] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/80 [14:17:08] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/40 [14:17:14] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/40 [14:18:03] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/42 [14:18:27] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/28 [14:18:30] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/28 [14:18:39] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/30 [14:18:40] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/30 [14:18:47] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewikisignbot] - 10https://gitlab.wikimedia.org/toolforge-repos/dewikisignbot/-/merge_requests/33 [14:18:48] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewikisignbot] - 10https://gitlab.wikimedia.org/toolforge-repos/dewikisignbot/-/merge_requests/33 [15:14:27] (03update) 10renovatebot: Update pnpm to v12.4.0 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/20 [15:14:31] (03open) 10renovatebot: Update pnpm to v12.4.0 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/20 [15:14:50] (03update) 10renovatebot: Update pnpm to v12.4.0 [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/32 [15:14:57] (03open) 10renovatebot: Update pnpm to v12.4.0 [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/32 [15:15:13] (03update) 10renovatebot: Update pnpm to v12.4.0 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/27 [15:15:21] (03open) 10renovatebot: Update pnpm to v12.4.0 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/27 [15:16:53] (03update) 10renovatebot: Update pnpm to v12.4.0 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/59 [15:16:57] (03open) 10renovatebot: Update pnpm to v12.4.0 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/59 [15:18:07] (03update) 10renovatebot: Update pnpm to v12.4.0 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/44 [15:18:20] (03open) 10renovatebot: Update pnpm to v12.4.0 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/44 [15:23:56] 10Quarry, 06tools-platform-team: quarry deployment fails with S3 403 - https://phabricator.wikimedia.org/T395238#12312525 (10Andrew) 05Open→03Resolved a:03Andrew This is done but the same updates will be needed in PAWS. [15:32:06] 06tools-infrastructure-team, 10Ceph, 06Data-Platform-SRE (2026-08-28 - 2026-09-18), 13Patch-For-Review: ceph: Fix the CephX key guard so that Puppet can correctly update the key material - https://phabricator.wikimedia.org/T437233#12312572 (10BTullis) [16:04:47] 10Data-Services, 06tools-platform-team, 13Patch-For-Review, 07User-notice: Update Wiki Replicas documentation for x4 rollout - https://phabricator.wikimedia.org/T437208#12312663 (10STei-WMF) @taavi, done: https://meta.wikimedia.org/wiki/Tech/News/2026/38 [16:14:15] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/16 [16:14:21] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/28 [16:14:27] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/23 [16:15:19] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/55 [16:15:39] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/80 [16:15:57] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/40 [16:16:26] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/42 [16:16:41] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/28 [16:16:46] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/30 [16:16:50] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewikisignbot] - 10https://gitlab.wikimedia.org/toolforge-repos/dewikisignbot/-/merge_requests/33 [16:21:50] (03update) 10raymond-ndibe: models: support webservice job-type [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/264 (https://phabricator.wikimedia.org/T428898) [17:14:03] (03update) 10renovatebot: Update pnpm to v12.4.0 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/20 [17:14:35] (03update) 10renovatebot: Update pnpm to v12.4.1 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/20 [17:14:50] (03update) 10renovatebot: Update pnpm to v12.4.0 [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/32 [17:14:52] (03update) 10renovatebot: Update pnpm to v12.4.1 [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/32 [17:15:12] (03update) 10renovatebot: Update pnpm to v12.4.1 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/27 [17:15:13] (03update) 10renovatebot: Update pnpm to v12.4.1 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/27 [17:16:15] (03update) 10renovatebot: Update pnpm to v12.4.0 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/59 [17:16:17] (03update) 10renovatebot: Update pnpm to v12.4.1 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/59 [17:17:18] (03update) 10renovatebot: Update pnpm to v12.4.0 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/44 [17:17:21] (03update) 10renovatebot: Update pnpm to v12.4.1 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/44 [18:03:34] 10Cloud-VPS (Debian Bullseye Deprecation), 03Readers Essential Work (Sunsetting and cleanup): Migrate reading-web-staging away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435780#12313085 (10Jdlrobson-WMF) I'm just waiting for @Jdrewniak to confirm that the simple-summary-server... [18:08:43] 10Openstack-Magnum, 10Quarry, 06tools-infrastructure-team: Rebuild quarry with the new magnum-cluster-api driver - https://phabricator.wikimedia.org/T437501#12313091 (10Andrew) This gets a cluster up and running ` diff --git a/tofu/127b.tf b/tofu/127b.tf index 6d370b5..d0eeb24 100644 --- a/tofu/127b.tf +++... [18:14:18] (03update) 10renovatebot: Update pnpm to v12.4.1 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/20 [18:14:27] (03update) 10renovatebot: Update pnpm to v12.4.1 [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/32 [18:14:35] (03update) 10renovatebot: Update pnpm to v12.4.1 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/27 [18:15:31] (03update) 10renovatebot: Update pnpm to v12.4.1 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/59 [18:16:11] (03update) 10renovatebot: Update pnpm to v12.4.1 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/44 [20:11:35] 10Cloud-VPS (Debian Bullseye Deprecation), 03Readers Essential Work (Sunsetting and cleanup): Migrate reading-web-staging away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435780#12313413 (10Jdlrobson-WMF) 05Open→03Resolved This should be done now. [22:14:25] (03update) 10renovatebot: Update pnpm to v12.4.1 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/20 [22:14:36] (03update) 10renovatebot: Update pnpm to v12.4.1 [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/32 [22:14:48] (03update) 10renovatebot: Update pnpm to v12.4.1 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/27 [22:15:46] (03update) 10renovatebot: Update pnpm to v12.4.1 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/59 [22:16:34] (03update) 10renovatebot: Update pnpm to v12.4.1 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/44