[07:19:07] 10Gerrit, 06collaboration-services: Validate Gerrit backup recoverability after the fileset exclusions - https://phabricator.wikimedia.org/T430775#12177667 (10ABran-WMF) @hashar Thanks, but I think the hot-write concern is [[ https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/d11353967b82663650... [07:49:45] 06Release-Engineering-Team, 10Wikidata, 06Wikidata-Omega (Completed Tasks): quibble-vendor-mysql-php83-selenium selenium-test failing with exit status 133 without obvious reason - https://phabricator.wikimedia.org/T432826#12177753 (10karapayneWMDE) [08:54:23] 10Phabricator: Consider stop using Phame - https://phabricator.wikimedia.org/T428225#12177915 (10Aklapper) p:05Triage→03Low [08:58:25] 10Gerrit, 06collaboration-services: Validate Gerrit backup recoverability after the fileset exclusions - https://phabricator.wikimedia.org/T430775#12177927 (10jcrespo) I would agree with the snapshotting option- this can be implemented in pre-backup and post-backup bacula automation to create and free the snap... [09:06:39] 06Release-Engineering-Team (Doing 😎), 10Catalyst (Luka Ijo Pimeja Jan), 07Essential-Work: SPIKE: Investigate KEDA autoscaling - https://phabricator.wikimedia.org/T433249#12177943 (10jnuche) 05Open→03Resolved I got the autoscaler working for the code server on my local cluster. My overall assessment... [09:31:26] 06Release-Engineering-Team (Doing 😎), 10Catalyst (Luka Ijo Pimeja Jan), 07Essential-Work: KEDA autoscaler for Patchdemo workloads - https://phabricator.wikimedia.org/T433823 (10jnuche) 03NEW [09:31:39] 06Release-Engineering-Team (Doing 😎), 10Catalyst (Luka Ijo Pimeja Jan), 07Essential-Work: KEDA autoscaler for Patchdemo workloads - https://phabricator.wikimedia.org/T433823#12178045 (10jnuche) [09:35:17] 06Release-Engineering-Team (Doing 😎), 10Catalyst (Luka Ijo Pimeja Jan), 07Essential-Work: KEDA autoscaler for Patchdemo workloads - https://phabricator.wikimedia.org/T433823#12178060 (10jnuche) [09:35:22] (03CR) 10Hashar: Zuul: Add a unit test for validating email regexes (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/1319875 (https://phabricator.wikimedia.org/T421143) (owner: 10SomeRandomDeveloper) [09:35:34] (03CR) 10Hashar: [C:03+2] "Awesome thank you very much for the test!" [integration/config] - 10https://gerrit.wikimedia.org/r/1319875 (https://phabricator.wikimedia.org/T421143) (owner: 10SomeRandomDeveloper) [09:37:37] (03Merged) 10jenkins-bot: Zuul: Add a unit test for validating email regexes [integration/config] - 10https://gerrit.wikimedia.org/r/1319875 (https://phabricator.wikimedia.org/T421143) (owner: 10SomeRandomDeveloper) [11:34:36] 10Phabricator: Allow Security Team members to see MFA status of a Phab user account - https://phabricator.wikimedia.org/T401106#12178391 (10Aklapper) 05Open→03Stalled > I'd first like to see both T410305 and https://we.phorge.it/T15947 tackled before looking further into this, thus unassigning myself for now... [11:40:42] 10Phabricator (2026-07-28), 06Release-Engineering-Team (Doing 😎): Add a sentence about [[mw:Phabricator/Bots]] to the user account creation form - https://phabricator.wikimedia.org/T432888#12178402 (10Aklapper) 05Open→03Resolved Deployed last week. [11:40:51] 10Phabricator (2026-07-28), 06Release-Engineering-Team (Doing 😎): Display better guidance for new technical contributors below Maniphest comment field - https://phabricator.wikimedia.org/T433137#12178404 (10Aklapper) 05Open→03Resolved Deployed last week. [12:47:25] 06Release-Engineering-Team (Doing 😎), 10Catalyst (Luka Ijo Pimeja Jan), 07Essential-Work: KEDA autoscaler for Patchdemo workloads - https://phabricator.wikimedia.org/T433823#12178600 (10jnuche) [14:06:07] 10Gerrit, 06collaboration-services: Validate Gerrit backup recoverability after the fileset exclusions - https://phabricator.wikimedia.org/T430775#12178897 (10ABran-WMF) LVM is indeed setup, I'll file a task to track this effort so this one stays focused on the restore validation itself Thanks for the answer! [14:08:41] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.47.0-wmf.14 deployment blockers - https://phabricator.wikimedia.org/T430833#12178903 (10jnuche) >>! In T430833#12176845, @Bugreporter wrote: > https://gerrit.wikimedia.org/r/c/mediawiki/extensions/cldr/+/13... [14:27:56] 10Continuous-Integration-Config, 10CirrusSearch, 10Wikidata, 10Wikidata Lexicographical data, and 4 others: LexemeFullTextQueryBuilderTest::testSearchElastic CI test errors - https://phabricator.wikimedia.org/T428975#12178988 (10pfischer) 05Open→03Resolved [14:28:14] 10Beta-Cluster-Infrastructure, 06Growth-Team, 10GrowthExperiments-NewcomerTasks, 06Discovery-Search (2026.06.01 - 2026.07.03): [beta-cluster] Fetching task suggestions failed: cirrussearch-backend-error - https://phabricator.wikimedia.org/T427196#12178998 (10pfischer) 05Open→03Resolved [14:34:12] 10Phabricator, 06Release-Engineering-Team (Priority Backlog 📥): Update to Phorge/Arcanist upstream 2026.xx - https://phabricator.wikimedia.org/T428547#12179024 (10Aklapper) [14:35:01] 10Phabricator, 06Release-Engineering-Team (Doing 😎): Add isReadOnlyAPI() to our custom downstream ConduitAPIMethods - https://phabricator.wikimedia.org/T433842 (10Aklapper) 03NEW p:05Triage→03Low [14:35:09] 10Phabricator, 06Release-Engineering-Team (Doing 😎): Add isReadOnlyAPI() to our custom downstream ConduitAPIMethods - https://phabricator.wikimedia.org/T433842#12179046 (10Aklapper) [14:35:10] 10Phabricator, 06Release-Engineering-Team (Priority Backlog 📥): Update to Phorge/Arcanist upstream 2026.xx - https://phabricator.wikimedia.org/T428547#12179047 (10Aklapper) [14:37:19] (03open) 10aklapper: Add isReadOnlyAPI() to ConduitAPIMethods [repos/phabricator/extensions] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/extensions/-/merge_requests/69 (https://phabricator.wikimedia.org/T433842) [14:38:02] (03merge) 10aklapper: Add isReadOnlyAPI() to ConduitAPIMethods [repos/phabricator/extensions] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/extensions/-/merge_requests/69 (https://phabricator.wikimedia.org/T433842) [14:45:23] 10Phabricator (phabricator-next), 06Release-Engineering-Team (Doing 😎): Add isReadOnlyAPI() to our custom downstream ConduitAPIMethods - https://phabricator.wikimedia.org/T433842#12179122 (10Aklapper) 05Open→03Resolved [15:02:52] 10Continuous-Integration-Config, 10MediaWiki-extensions-ReadingLists, 05FY252627 Reading Lists (Graduating from Beta), 13Patch-For-Review, 06Reader Experience Team (REx Sprint 3 [Q1 FY26-27 July 29-Aug 11]): ReadingLists tests not run in jenkins gate and... - https://phabricator.wikimedia.org/T403560#12179223 [15:10:22] 06Release-Engineering-Team (Seen), 06collaboration-services, 06SRE: URL shortener subdomains for useful Wikimedia infrastructure - https://phabricator.wikimedia.org/T223319#12179266 (10LSobanski) 05Open→03Declined I don't see Collab working on this any time soon. [15:56:13] 10GitLab (Infrastructure), 06collaboration-services: Make sure GitLab does not exceed apus object storage quotas - https://phabricator.wikimedia.org/T413871#12179573 (10LSobanski) [15:56:14] 10GitLab (Infrastructure), 06collaboration-services: Make sure GitLab scales with more usage - https://phabricator.wikimedia.org/T374448#12179571 (10LSobanski) →14Duplicate dup:03T413871 [15:57:51] 10Phabricator, 06Release-Engineering-Team (Priority Backlog 📥), 06collaboration-services: Phab Aphlict Notification Servers config shows Connection error - https://phabricator.wikimedia.org/T402396#12179581 (10LSobanski) p:05Medium→03Low [16:06:35] 06Release-Engineering-Team (Doing 😎), 10Catalyst (Luka Ijo Pimeja Jan), 07Essential-Work: KEDA autoscaler for Patchdemo workloads - https://phabricator.wikimedia.org/T433823#12179634 (10brennen) 05Open→03In progress [16:19:59] (03CR) 10Hashar: [C:03+1] "Vaughn and I are pairing on this one." [integration/config] - 10https://gerrit.wikimedia.org/r/1312592 (https://phabricator.wikimedia.org/T427006) (owner: 10Vaughn Walters) [16:33:03] (03PS2) 10Jforrester: Zuul: Drop REL1_44 testing, EOL [integration/config] - 10https://gerrit.wikimedia.org/r/1318676 (https://phabricator.wikimedia.org/T428911) [16:33:06] (03PS3) 10Jforrester: Zuul: Drop REL1_44 testing, EOL [integration/config] - 10https://gerrit.wikimedia.org/r/1318676 (https://phabricator.wikimedia.org/T428911) [16:33:10] (03CR) 10Jforrester: [C:03+2] Zuul: Drop REL1_44 testing, EOL [integration/config] - 10https://gerrit.wikimedia.org/r/1318676 (https://phabricator.wikimedia.org/T428911) (owner: 10Jforrester) [16:33:11] 06Release-Engineering-Team, 06SRE, 10SRE-Access-Requests: Requesting access to Jenkins server for PWangai-WMF - https://phabricator.wikimedia.org/T433563#12179849 (10brennen) Noting here that I've added this to the RelEng team discussion agenda for Weds this week; we'll get back to you. [16:36:36] (03Merged) 10jenkins-bot: Zuul: Drop REL1_44 testing, EOL [integration/config] - 10https://gerrit.wikimedia.org/r/1318676 (https://phabricator.wikimedia.org/T428911) (owner: 10Jforrester) [16:37:02] !log Zuul: Drop REL1_44 testing, EOL, for T428911 [16:37:04] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:37:04] T428911: Formally EOL MW 1.44 - https://phabricator.wikimedia.org/T428911 [16:37:35] 06Release-Engineering-Team, 06SRE, 10SRE-Access-Requests: Requesting access to Jenkins server for PWangai-WMF - https://phabricator.wikimedia.org/T433563#12179888 (10AMarkossyan-WMF) Thank you, Brennen! On my end, I know that this requires my approval, so here you go: 👍 In general, as this is blocking us... [16:40:36] !log gerrit: added Vaughn Walters to integration group until he get added to the ciadmin LDAP group | T433615 [16:40:41] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:40:42] T433615: Grant Access to ciadmin for Vaughn Walters - https://phabricator.wikimedia.org/T433615 [16:42:27] (03CR) 10Vaughn Walters: [C:03+2] jjb: [catalyst-daily-CentralNotice] Add CentralNotice job [integration/config] - 10https://gerrit.wikimedia.org/r/1312592 (https://phabricator.wikimedia.org/T427006) (owner: 10Vaughn Walters) [16:42:49] (03CR) 10Vaughn Walters: [C:03+2] "I have deployed this job, and test is green." [integration/config] - 10https://gerrit.wikimedia.org/r/1312592 (https://phabricator.wikimedia.org/T427006) (owner: 10Vaughn Walters) [16:51:23] (03Merged) 10jenkins-bot: jjb: [catalyst-daily-CentralNotice] Add CentralNotice job [integration/config] - 10https://gerrit.wikimedia.org/r/1312592 (https://phabricator.wikimedia.org/T427006) (owner: 10Vaughn Walters) [16:57:08] 06Release-Engineering-Team, 06SRE, 10SRE-Access-Requests: Requesting access to Jenkins server for PWangai-WMF - https://phabricator.wikimedia.org/T433563#12180047 (10Dzahn) I strongly recommend to define the proper solution right now and avoid a diverting "path of least resistence". (Also there is no "just a... [17:12:47] 06Release-Engineering-Team: Mobileapps fails on image publishing CI step - https://phabricator.wikimedia.org/T433875 (10Jgiannelos) 03NEW [17:25:58] 06Release-Engineering-Team: Mobileapps fails on image publishing CI step - https://phabricator.wikimedia.org/T433875#12180211 (10Jgiannelos) p:05Triage→03High [17:50:57] 06Release-Engineering-Team: Mobileapps fails on image publishing CI step - https://phabricator.wikimedia.org/T433875#12180274 (10dancy) Please provide a link to a Jenkins job run. [17:52:56] 10Continuous-Integration-Config, 07Security: Add a test to the integration/config repo that validates whether entries on the CI allowlist are correctly formatted/escaped/etc. - https://phabricator.wikimedia.org/T421143#12180277 (10SomeRandomDeveloper) 05Open→03Resolved [18:01:19] 06Release-Engineering-Team: Mobileapps fails on image publishing CI step - https://phabricator.wikimedia.org/T433875#12180293 (10Dbrant) https://integration.wikimedia.org/ci/job/mobileapps-pipeline-publish/414/console [18:12:49] 06Release-Engineering-Team: Mobileapps fails on image publishing CI step - https://phabricator.wikimedia.org/T433875#12180309 (10dancy) `update_version/update_version.py` comes from https://gerrit.wikimedia.org/g/operations/deployment-charts/+/c6ecb431ad5cf1d5f27ae243739710ed6232e8f8/update_version/update_versio... [18:25:25] 06Release-Engineering-Team: Mobileapps fails on image publishing CI step - https://phabricator.wikimedia.org/T433875#12180316 (10dancy) 05Open→03In progress a:03dancy [18:25:41] 06Release-Engineering-Team, 07Essential-Work: Mobileapps fails on image publishing CI step - https://phabricator.wikimedia.org/T433875#12180318 (10dancy) [18:41:56] 10GitLab (CI & Job Runners), 06Release-Engineering-Team, 07Essential-Work: Buildkit v0.32.1 released - https://phabricator.wikimedia.org/T433879 (10dancy) 03NEW [18:54:37] 06Release-Engineering-Team, 07Essential-Work, 13Patch-For-Review: Mobileapps fails on image publishing CI step - https://phabricator.wikimedia.org/T433875#12180393 (10dancy) 05In progress→03Resolved Fixed: https://integration.wikimedia.org/ci/job/mobileapps-pipeline-publish/415/console [19:12:19] 10Scap, 10MediaWiki-Configuration: Consider using `--extensions-dir` instead of `--list-file` with mergeMessageFileList.php - https://phabricator.wikimedia.org/T417954#12180436 (10dancy) [19:12:36] 10Scap, 10MediaWiki-Configuration: Consider using `--extensions-dir` instead of `--list-file` with mergeMessageFileList.php - https://phabricator.wikimedia.org/T417954#12180437 (10dancy) [19:19:10] Project mediawiki-core-doxygen build #22030: 04FAILURE in 1 min 8 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/22030/ [19:20:54] 10Scap, 10MediaWiki-Configuration: Consider using `--extensions-dir` instead of `--list-file` with mergeMessageFileList.php - https://phabricator.wikimedia.org/T417954#12180442 (10dancy) > We assume that attempting to build the localized message cache for all extensions and skins would be too slow at build tim... [20:16:22] 10GitLab (CI & Job Runners), 06Release-Engineering-Team, 07Essential-Work: Buildkit v0.32.1 released - https://phabricator.wikimedia.org/T433879#12180615 (10dancy) docker-registry.wikimedia.org/repos/releng/buildkit:wmf-v0.32.1@sha256:905218d79b70fa73c6216c5db850c70e1034d2afd1e1c08a52c37fbbb133173b [20:33:53] 10GitLab (Project Migration), 06collaboration-services: Migrate SRE repositories to GitLab - operations/software - https://phabricator.wikimedia.org/T341504#12180751 (10Dzahn) I think this is also effectively declined like similar tasks about moving repos. [20:34:30] Yippee, build fixed! [20:34:30] Project mediawiki-core-doxygen build #22031: 09FIXED in 16 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/22031/ [20:36:59] 10GitLab (Project Migration), 06collaboration-services: Migrate SRE repositories to GitLab - https://phabricator.wikimedia.org/T341468#12180766 (10Dzahn) I think this is also effectively declined - similar to other tickets about moving repos? [20:43:26] (03open) 10dancy: Bump buildkitd to v0.32.1 [auto-deploy] [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/624 (https://phabricator.wikimedia.org/T433879) [20:44:29] (03merge) 10dancy: Bump buildkitd to v0.32.1 [auto-deploy] [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/624 (https://phabricator.wikimedia.org/T433879) [20:45:33] !log Updating buildkitd to v0.32.1 on gitlab-cloud-runners (staging and production) (T433879) [20:45:35] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:45:36] T433879: Buildkit v0.32.1 released - https://phabricator.wikimedia.org/T433879 [20:52:11] 10GitLab (CI & Job Runners), 06Release-Engineering-Team, 07Essential-Work, 13Patch-For-Review: Buildkit v0.32.1 released - https://phabricator.wikimedia.org/T433879#12180822 (10dancy) [21:19:56] Project mediawiki-core-doxygen build #22032: 04FAILURE in 1 min 54 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/22032/ [21:25:51] 10GitLab (CI & Job Runners), 06Release-Engineering-Team, 07Essential-Work, 13Patch-For-Review: Buildkit v0.32.1 released - https://phabricator.wikimedia.org/T433879#12180951 (10dancy) 05Open→03Resolved [21:36:59] (03open) 10taavi: Use Python logger module instead of raw print() [repos/ci-tools/libup] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup/-/merge_requests/122 [21:38:44] (03update) 10taavi: Use Python logger module instead of raw print() [repos/ci-tools/libup] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup/-/merge_requests/122 [21:38:46] (03update) 10taavi: db: Commit changes more frequently [repos/ci-tools/libup] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup/-/merge_requests/123 [21:38:50] (03open) 10taavi: db: Commit changes more frequently [repos/ci-tools/libup] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup/-/merge_requests/123 [21:41:27] (03approved) 10jforrester: Use Python logger module instead of raw print() [repos/ci-tools/libup] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup/-/merge_requests/122 (owner: 10taavi) [21:41:51] (03approved) 10jforrester: db: Commit changes more frequently [repos/ci-tools/libup] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup/-/merge_requests/123 (owner: 10taavi) [21:43:04] (03merge) 10taavi: Use Python logger module instead of raw print() [repos/ci-tools/libup] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup/-/merge_requests/122 [21:43:08] (03update) 10taavi: db: Commit changes more frequently [repos/ci-tools/libup] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup/-/merge_requests/123 [21:43:53] 06Release-Engineering-Team, 10Scap, 07Developer Productivity, 07Essential-Work, and 2 others: l10n rebuild errors are hidden - https://phabricator.wikimedia.org/T433104#12181025 (10dancy) 05Open→03Resolved Should be fixed. [21:46:15] ... of course T433794 doesn't happen anymore the moment I start looking into its causes [21:46:16] T433794: libup scheduler fails with "MySQL server has gone away" - https://phabricator.wikimedia.org/T433794 [22:19:10] Project mediawiki-core-doxygen build #22033: 04STILL FAILING in 1 min 7 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/22033/ [23:33:47] Yippee, build fixed! [23:33:47] Project mediawiki-core-doxygen build #22034: 09FIXED in 15 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/22034/