[00:00:17] (03Merged) 10jenkins-bot: castor-save-workspacecache: check remote dir exists [integration/config] - 10https://gerrit.wikimedia.org/r/1074573 (https://phabricator.wikimedia.org/T282893) (owner: 10Thcipriani) [00:45:17] 10Continuous-Integration-Infrastructure, 07Jenkins, 06Release-Engineering-Team (Priority Backlog 📥), 07ci-test-error, and 2 others: Various CI jobs failing after "mkdir: cannot create directory ‘log’: Permission denied" - https://phabricator.wikimedia.org/T282893#11741636 (10thcipriani) 05Open→03Res... [01:15:07] 10Continuous-Integration-Infrastructure, 07Jenkins, 10Quibble: All Cargo patches failing Quibble validation due to excess test duration? - https://phabricator.wikimedia.org/T421027 (10Yaron_Koren) 03NEW [01:36:49] 10Continuous-Integration-Infrastructure, 07Jenkins, 10Quibble: All Cargo patches failing Quibble validation due to excess test duration? - https://phabricator.wikimedia.org/T421027#11741717 (10Pppery) That's just some debugging info, not the actual failure. The actual failure, many lines above, appears to b... [02:49:08] 10Continuous-Integration-Infrastructure, 07Jenkins, 10Quibble: All Cargo patches failing Quibble validation due to excess test duration? - https://phabricator.wikimedia.org/T421027#11741819 (10Yaron_Koren) Oh, I see - renaming this bug now. [02:50:19] 10Continuous-Integration-Infrastructure, 07Jenkins, 10Quibble: All Cargo patches failing Quibble validation due to excess test duration? - https://phabricator.wikimedia.org/T421027#11741820 (10Yaron_Koren) Actually, it looks like this was already reported in a different bug... [02:50:29] 10Continuous-Integration-Infrastructure, 07Jenkins, 10Quibble: All Cargo patches failing Quibble validation due to excess test duration? - https://phabricator.wikimedia.org/T421027#11741822 (10Yaron_Koren) →14Duplicate dup:03T420849 [06:49:22] 10Continuous-Integration-Config, 07Epic, 07PHP 8.5 support: Make PHP 8.5 voting on development (master) branch of MW ecosystem (core, vendor, extensions, skins, libraries) in CI - https://phabricator.wikimedia.org/T411814#11741996 (10Krinkle) [07:00:07] 10Continuous-Integration-Config, 07Epic, 07PHP 8.5 support: Make PHP 8.5 voting on development (master) branch of MW ecosystem (core, vendor, extensions, skins, libraries) in CI - https://phabricator.wikimedia.org/T411814#11742017 (10Novem_Linguae) [07:14:19] 10Phabricator-Bot-Requests: Create mcp-phabricator bot account - https://phabricator.wikimedia.org/T420446#11742027 (10kostajh) >>! In T420446#11735337, @Aklapper wrote: > FYI I filed "Add a Read-Only permission scope for API Tokens" in https://we.phorge.it/T16568 with a potential, more simple implementation pro... [07:22:36] (03PS1) 10Krinkle: zuul: Enable php84 and php85 builds for cvn-api repo [integration/config] - 10https://gerrit.wikimedia.org/r/1259755 (https://phabricator.wikimedia.org/T419075) [07:23:35] (03CR) 10Krinkle: [C:03+2] zuul: Enable php84 and php85 builds for cvn-api repo [integration/config] - 10https://gerrit.wikimedia.org/r/1259755 (https://phabricator.wikimedia.org/T419075) (owner: 10Krinkle) [07:25:11] (03Merged) 10jenkins-bot: zuul: Enable php84 and php85 builds for cvn-api repo [integration/config] - 10https://gerrit.wikimedia.org/r/1259755 (https://phabricator.wikimedia.org/T419075) (owner: 10Krinkle) [07:42:08] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1259755 [07:42:09] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [07:47:52] 10Phabricator, 10Bitu, 06cloud-services-team, 06Infrastructure-Foundations, and 2 others: Inconsistent OAuth endpoint for Wikimedia Global Account (SUL) across services - https://phabricator.wikimedia.org/T420599#11742071 (10Aklapper) That custom Phabricator code would be in https://gitlab.wikimedia.org/re... [08:21:21] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team (Seen): Phase out python3.4 from CI infrastructure - https://phabricator.wikimedia.org/T270268#11742112 (10hashar) 05Open→03Resolved a:03hashar Support for python3.4 has been removed as we upgraded the CI images from Debian Stretch t... [08:22:26] 10Continuous-Integration-Config, 06Release-Engineering-Team (Doing 😎): Jenkins: In the integration project in WMCS, test out larger workers for CI - https://phabricator.wikimedia.org/T383254#11742121 (10hashar) [08:22:39] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team: Run Jenkins jobs from tmpfs rather than disk - https://phabricator.wikimedia.org/T340073#11742124 (10hashar) [08:25:04] 10Continuous-Integration-Infrastructure: Re-build CI images with a PHP version with a fix for CVE-2021-21703 once that's in Wikimedia APT - https://phabricator.wikimedia.org/T294304#11742133 (10hashar) 05Open→03Resolved After five years, I am assuming this has been fixed. [08:26:26] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team (Seen): Fix partitions on CI slaves , some are missing /var/lib/docker - https://phabricator.wikimedia.org/T221040#11742138 (10hashar) 05Open→03Resolved a:03hashar [08:27:37] 10Gerrit, 06Release-Engineering-Team, 14Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 06collaboration-services, and 2 others: Jenkins job failing intermittently due to Gerrit HTTP 502 errors when interacting with repos - https://phabricator.wikimedia.org/T246763#11742144 (10ABran-WMF) [08:29:02] Project mediawiki-core-doxygen build #18892: 04FAILURE in 10 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/18892/ [08:53:49] 10Phabricator: Have Phabricator link Gerrit change-id to Gerrit search query - https://phabricator.wikimedia.org/T420363#11742244 (10Aklapper) That doesn't sound too complicated: ` commit bb4b37045a6da6a7163ebc6eab94c5c72d8a2030 (HEAD -> master) Author: Andre Klapper Date: Tue Mar 24... [09:14:51] 06Project-Admins, 06tech-decision-forum: Archive tech-decision-forum - https://phabricator.wikimedia.org/T368599#11742290 (10A_smart_kitten) Is it worth reconsidering this task's declination? I do not work for the WMF, but I cannot remember seeing anything during the time that I have been active in Wikimed... [09:20:32] (03open) 10dondersmooi: PhabricatorMediaWikiAuthProvider: Use meta.wikimedia.org as OAuth server [repos/phabricator/extensions] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/extensions/-/merge_requests/62 (https://phabricator.wikimedia.org/T420599) [09:23:33] 10Phabricator, 10Bitu, 06cloud-services-team, 06Infrastructure-Foundations, and 3 others: Inconsistent OAuth endpoint for Wikimedia Global Account (SUL) across services - https://phabricator.wikimedia.org/T420599#11742308 (10Dondersmooi) The actual OAuth server URL is stored in the database (PROPERTY_MEDIA... [09:31:58] Yippee, build fixed! [09:31:58] Project mediawiki-core-doxygen build #18893: 09FIXED in 13 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/18893/ [09:36:16] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 06Traffic: gerrit: Add envoy in Gerrit's stack - https://phabricator.wikimedia.org/T420909#11742352 (10ABran-WMF) Merging the change on gerrit-spare shows that it works: ` arnaudb@gerrit2002:~ $ curl https://localhost:8443 -k ; echo (03open) 10gmodena: projects: add wikidata-platform/wdqs-proxy [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/160 (https://phabricator.wikimedia.org/T418837 https://phabricator.wikimedia.org/T420727) [09:36:45] (03update) 10gmodena: projects: add wikidata-platform/wdqs-proxy [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/160 (https://phabricator.wikimedia.org/T418837 https://phabricator.wikimedia.org/T420727) [10:23:28] 10Phabricator (Upstream), 07Upstream: EXCEPTION: (Exception) Policy identifier is an object PHID (''), but no object handle was provided. A handle must be provided for object policies. - https://phabricator.wikimedia.org/T361459#11742562 (10Aklapper) https://we.phorge.it/rPec7a82e58a5512b572753bfb24781b4c70a98... [10:23:34] 10Phabricator (Upstream), 07Upstream: EXCEPTION: (Exception) Policy identifier is an object PHID (''), but no object handle was provided. A handle must be provided for object policies. - https://phabricator.wikimedia.org/T361459#11742563 (10Aklapper) [10:23:36] 10Phabricator, 06Release-Engineering-Team (Priority Backlog 📥): Update to Phorge/Arcanist upstream (2026.xx) - https://phabricator.wikimedia.org/T410849#11742564 (10Aklapper) [10:23:44] (03update) 10gmodena: projects: add wikidata-platform/wdqs-proxy [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/160 (https://phabricator.wikimedia.org/T418837 https://phabricator.wikimedia.org/T420727) [10:28:03] (03update) 10trueg: projects: add wikidata-platform/wdqs-proxy [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/160 (https://phabricator.wikimedia.org/T418837 https://phabricator.wikimedia.org/T420727) (owner: 10gmodena) [10:36:39] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 06Traffic, 13Patch-For-Review: gerrit: Add envoy in Gerrit's stack - https://phabricator.wikimedia.org/T420909#11742610 (10ABran-WMF) [10:39:25] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 06Traffic, 13Patch-For-Review: gerrit: Add envoy in Gerrit's stack - https://phabricator.wikimedia.org/T420909#11742621 (10ABran-WMF) the first attempt at mapping gerrit-spare to its Envoy endpoint was blocked because the block: ` par... [10:44:29] 10Phabricator, 10Bitu, 06cloud-services-team, 06Infrastructure-Foundations, and 3 others: Inconsistent OAuth endpoint for Wikimedia Global Account (SUL) across services - https://phabricator.wikimedia.org/T420599#11742630 (10A_smart_kitten) FWIW, changing the OAuth domain used by Phabricator from mediawiki... [10:44:37] 10Gerrit (Gerrit 3.12): Upgrade to Gerrit 3.12 - https://phabricator.wikimedia.org/T392448#11742633 (10hashar) [10:45:30] 10Gerrit (Gerrit 3.10): Upgrade Gerrit from 3.10.6 to 3.10.9 - https://phabricator.wikimedia.org/T400688#11742664 (10hashar) [10:46:18] 10Gerrit (Gerrit 3.10): Upgrade Gerrit from 3.10.6 to 3.10.9 - https://phabricator.wikimedia.org/T400688#11742667 (10hashar) I have removed T420822 as a subtask since the fix was not backport to EOL 3.10 (it was not backported to 3.11 either). [10:46:57] 10Gerrit (Gerrit 3.11), 06Release-Engineering-Team (Priority Backlog 📥), 06collaboration-services, 13Patch-For-Review: Upgrade to Gerrit 3.11 - https://phabricator.wikimedia.org/T379714#11742669 (10hashar) [10:47:36] 10Gerrit (Gerrit 3.11), 06Release-Engineering-Team (Priority Backlog 📥), 06collaboration-services, 13Patch-For-Review: Upgrade to Gerrit 3.11 - https://phabricator.wikimedia.org/T379714#11742673 (10hashar) I have removed T420822 as a subtask since upstream has not backported it to 3.11 either [10:50:46] 10Gerrit (Gerrit 3.12): Upgrade to Gerrit 3.12 - https://phabricator.wikimedia.org/T392448#11742700 (10hashar) @ABran-WMF I have removed the subtasks since they are not blockers for the 3.12 upgrade. T420822 will be resolved once we upgrade rather than blocking it, it is already marked with #gerrit3.12 ;-) [10:53:00] 10Beta-Cluster-Infrastructure, 10Cassandra, 06Data-Persistence: Cassandra killed by oom-killer and prometheus scrapes failing intermittently on deployment-sessionstore06 - https://phabricator.wikimedia.org/T415021#11742709 (10hashar) @bd808 thank you so much for having resolved that issue! :-] [10:54:08] 10Phabricator, 10Bitu, 06cloud-services-team, 06Infrastructure-Foundations, and 3 others: Inconsistent OAuth endpoint for Wikimedia Global Account (SUL) across services - https://phabricator.wikimedia.org/T420599#11742712 (10A_smart_kitten) (IIUC, it would also mean that the logo & site-name in the "Log In... [10:55:19] 10Phabricator, 10Bitu, 06cloud-services-team, 06Infrastructure-Foundations, and 3 others: Inconsistent OAuth endpoint for Wikimedia Global Account (SUL) across services - https://phabricator.wikimedia.org/T420599#11742715 (10taavi) Meh. Doing Phabricator authentication via mw.o does have some minor advanta... [10:58:02] 10Diffusion, 10Phabricator, 06collaboration-services, 10Wikidata, and 3 others: WikibaseLexeme submodule is not available - https://phabricator.wikimedia.org/T409519#11742730 (10hashar) Can requestctl loosen the rules for requests made over the git protocol? That would be URl matching at least `GET .*info... [11:06:38] 10Phabricator, 10Bitu, 06cloud-services-team, 06Infrastructure-Foundations, and 3 others: Inconsistent OAuth endpoint for Wikimedia Global Account (SUL) across services - https://phabricator.wikimedia.org/T420599#11742745 (10Dondersmooi) >In T420599#11742712, @A_smart_kitten wrote: > True; however, I guess... [12:04:19] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 06Traffic, 13Patch-For-Review: gerrit: Add envoy in Gerrit's stack - https://phabricator.wikimedia.org/T420909#11742909 (10ABran-WMF) [12:04:54] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 06Traffic, 13Patch-For-Review: gerrit: Add envoy in Gerrit's stack - https://phabricator.wikimedia.org/T420909#11742914 (10ABran-WMF) `gerrit-spare` now uses Envoy to expose its service to the CDN. [12:10:38] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 06Traffic, 13Patch-For-Review: gerrit: Add envoy in Gerrit's stack - https://phabricator.wikimedia.org/T420909#11742964 (10ABran-WMF) [12:21:35] 06Release-Engineering-Team, 06collaboration-services: gerrit: Simplify dropping abusers with requestctl - https://phabricator.wikimedia.org/T421056 (10ABran-WMF) 03NEW [12:53:40] !log integration: deleted old Puppet 5 compiler agents from Jenkins ( pcc-worker1014.puppet-diffs.eqiad1.wikimedia.cloud , pcc-worker1015.puppet-diffs.eqiad1.wikimedia.cloud , pcc-worker1016.puppet-diffs.eqiad1.wikimedia.cloud ) # T367399 [12:53:42] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [12:53:43] T367399: Default to the Puppet 7 PCC CI test, make it voting and eventually remove the Puppet 5 one - https://phabricator.wikimedia.org/T367399 [13:17:39] 10Continuous-Integration-Config: New phan dependencies significantly slowed down CI tests - https://phabricator.wikimedia.org/T225112#11743357 (10hashar) [13:20:00] 10Continuous-Integration-Config, 13Patch-For-Review: Create a CI job to enforce tests to pass with solely required extensions - https://phabricator.wikimedia.org/T407797#11743383 (10hashar) [13:20:01] 10Continuous-Integration-Infrastructure: run extension jobs with and without optional dependencies - https://phabricator.wikimedia.org/T114691#11743386 (10hashar) →14Duplicate dup:03T407797 [13:20:10] 10Continuous-Integration-Infrastructure: run extension jobs with and without optional dependencies - https://phabricator.wikimedia.org/T114691#11743390 (10hashar) More than a decade later, this will eventually be addressed by {T407797} [13:25:08] (03CR) 10Phedenskog: "Ah cool I didn't know thank you." [integration/quibble] - 10https://gerrit.wikimedia.org/r/1258947 (https://phabricator.wikimedia.org/T419583) (owner: 10Phedenskog) [13:31:37] 10Continuous-Integration-Infrastructure: Re-build CI images with a PHP version with a fix for CVE-2021-21703 once that's in Wikimedia APT - https://phabricator.wikimedia.org/T294304#11743464 (10Jdforrester-WMF) Oops, yes. https://gerrit.wikimedia.org/r/plugins/gitiles/integration/config/+/1b6a0eda4ad5477cd0... [13:49:58] 10Phabricator, 10Bitu, 06cloud-services-team, 06Infrastructure-Foundations, and 3 others: Inconsistent OAuth endpoint for Wikimedia Global Account (SUL) across services - https://phabricator.wikimedia.org/T420599#11743666 (10A_smart_kitten) >>! In T420599#11742745, @Dondersmooi wrote: > My understanding is... [14:02:15] 10Phabricator, 10Bitu, 06cloud-services-team, 06Infrastructure-Foundations, and 3 others: Inconsistent OAuth endpoint for Wikimedia Global Account (SUL) across services - https://phabricator.wikimedia.org/T420599#11743968 (10bd808) >>! In T420599#11730366, @bd808 wrote: > The clients could be reconfigured,... [14:06:05] 10Continuous-Integration-Config, 06Release-Engineering-Team, 06Security-Team, 07SecTeam-Processed, and 2 others: Some entries on the CI allowlist for individual email addresses will also match other, different, email addresses - https://phabricator.wikimedia.org/T420838#11744072 (10sbassett) p:05Triag... [14:07:31] 10Continuous-Integration-Infrastructure: Rebuild all Jenkins agents VM to support Java 21 - https://phabricator.wikimedia.org/T421114 (10hashar) 03NEW [14:09:26] 10Continuous-Integration-Infrastructure, 10Castor, 10Pywikibot: CASTOR sync for pywikibot-core-tox can take over 5 minutes in some cases - https://phabricator.wikimedia.org/T403266#11744149 (10Xqt) [14:12:56] 10Continuous-Integration-Infrastructure: Rebuild all Jenkins agents VM to support Java 21 - https://phabricator.wikimedia.org/T421114#11744164 (10hashar) [14:14:53] 10Continuous-Integration-Infrastructure, 07Jenkins, 06collaboration-services: Update Jenkins hosts from Java 17 to Java 21 - https://phabricator.wikimedia.org/T418109#11744171 (10hashar) [14:14:53] 10Continuous-Integration-Infrastructure: Rebuild all Jenkins agents VM to support Java 21 - https://phabricator.wikimedia.org/T421114#11744172 (10hashar) [14:28:33] 10Continuous-Integration-Infrastructure: Rebuild all Jenkins agents VM to support Java 21 - https://phabricator.wikimedia.org/T421114#11744258 (10hashar) [14:33:55] (03update) 10dondersmooi: PhabricatorMediaWikiAuthProvider: Use meta.wikimedia.org as OAuth server [repos/phabricator/extensions] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/extensions/-/merge_requests/62 (https://phabricator.wikimedia.org/T420599) [14:33:56] (03close) 10dondersmooi: PhabricatorMediaWikiAuthProvider: Use meta.wikimedia.org as OAuth server [repos/phabricator/extensions] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/extensions/-/merge_requests/62 (https://phabricator.wikimedia.org/T420599) [14:38:06] 10Phabricator, 10Bitu, 06cloud-services-team, 06Infrastructure-Foundations, and 3 others: Inconsistent OAuth endpoint for Wikimedia Global Account (SUL) across services - https://phabricator.wikimedia.org/T420599#11744332 (10Dondersmooi) >> In T420599#11743968, @bd808 wrote: > Developers, including the fol... [14:59:19] 10GitLab, 06Release-Engineering-Team, 06Research: Disable Issues in GitLab - https://phabricator.wikimedia.org/T412882#11744460 (10fkaelin) All open gitlab issues have been migrated to phabricator and closed. The issues can now be disabled on gitlab. [15:26:35] !log Unblock 47.194.0.0/16 (T421127) [15:26:36] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:31:06] 10GitLab, 06Release-Engineering-Team, 06collaboration-services, 06Research: Disable Issues in GitLab - https://phabricator.wikimedia.org/T412882#11744677 (10Aklapper) Thank you a lot! Note to myself: I assume resolving this ticket means emptying the `issue_wiki_allowlist` array in https://gitlab.wikimedia... [15:32:22] 10Continuous-Integration-Infrastructure, 07Jenkins, 06Release-Engineering-Team (Priority Backlog 📥), 07ci-test-error, and 2 others: Various CI jobs failing after "mkdir: cannot create directory ‘log’: Permission denied" - https://phabricator.wikimedia.org/T282893#11744684 (10dancy) Nice fixin' @thcipri... [15:34:20] !log gitlab1004: manual test run of `configure-projects` with cleared issue allowlist (T412882) [15:34:24] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:34:25] T412882: Disable Issues in GitLab - https://phabricator.wikimedia.org/T412882 [15:34:37] (03open) 10brennen: configure-proejcts: clear out issue allowlist [repos/releng/gitlab-settings] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-settings/-/merge_requests/81 (https://phabricator.wikimedia.org/T412882) [15:35:32] 10GitLab, 06Release-Engineering-Team, 06collaboration-services, 06Research, 13Patch-For-Review: Disable Issues in GitLab - https://phabricator.wikimedia.org/T412882#11744698 (10brennen) a:03brennen [15:35:45] 10GitLab, 06Release-Engineering-Team, 06collaboration-services, 06Research, 13Patch-For-Review: Disable Issues in GitLab - https://phabricator.wikimedia.org/T412882#11744699 (10brennen) [15:39:38] 10GitLab, 06Release-Engineering-Team, 06collaboration-services, 06Research, 13Patch-For-Review: Disable Issues in GitLab - https://phabricator.wikimedia.org/T412882#11744711 (10brennen) > I assume resolving this ticket means emptying the `issue_wiki_allowlist` array in https://gitlab.wikimedia.org/repos/... [15:41:49] 10GitLab (CI & Job Runners), 06Release-Engineering-Team (Priority Backlog 📥): Hanging NotReady status following OOM on node - https://phabricator.wikimedia.org/T405287#11744712 (10dancy) 05Open→03Declined I'm closing this report with Declined status. In the meantime we have set cpu and memory limits o... [15:53:15] (03approved) 10arnaudb: configure-proejcts: clear out issue allowlist [repos/releng/gitlab-settings] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-settings/-/merge_requests/81 (https://phabricator.wikimedia.org/T412882) (owner: 10brennen) [15:56:18] (03update) 10arnaudb: configure-proejcts: clear out issue allowlist [repos/releng/gitlab-settings] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-settings/-/merge_requests/81 (https://phabricator.wikimedia.org/T412882) (owner: 10brennen) [15:57:06] (03merge) 10brennen: configure-proejcts: clear out issue allowlist [repos/releng/gitlab-settings] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-settings/-/merge_requests/81 (https://phabricator.wikimedia.org/T412882) [16:02:31] 10Phabricator-Bot-Requests, 06Release-Engineering-Team (Doing 😎), 07Essential-Work: Request bot account @Slack-bot for Slack integration - https://phabricator.wikimedia.org/T393773#11744844 (10Aklapper) [16:03:34] 10GitLab, 06Release-Engineering-Team, 06collaboration-services, 06Research, 13Patch-For-Review: Disable Issues in GitLab - https://phabricator.wikimedia.org/T412882#11744848 (10brennen) 05Open→03Resolved [16:13:03] 10Continuous-Integration-Infrastructure, 07Jenkins, 10Cloud-VPS (Quota-requests): Large temporary quota increase for integration project to ease Jenkins migration - https://phabricator.wikimedia.org/T421139 (10hashar) 03NEW [16:14:54] 10Continuous-Integration-Infrastructure, 07Jenkins, 06collaboration-services: Update Jenkins hosts from Java 17 to Java 21 - https://phabricator.wikimedia.org/T418109#11744945 (10hashar) [16:15:00] 10Continuous-Integration-Infrastructure, 07Jenkins, 10Cloud-VPS (Quota-requests): Large temporary quota increase for integration project to ease Jenkins migration - https://phabricator.wikimedia.org/T421139#11744946 (10hashar) [16:22:43] 10Phabricator, 06Release-Engineering-Team (Doing 😎): Delete some now invalid PhabricatorFeedQuery saved queries in the Phab DB - https://phabricator.wikimedia.org/T415453#11745036 (10Aklapper) 05Open→03Resolved a:03Aklapper Deleted 10 rows from the DB, expecting no more `Call to undefined method Phab... [16:33:30] 10Beta-Cluster-Infrastructure: Beta cluster is slow as sludge / serves 503 and 504 - https://phabricator.wikimedia.org/T420833#11745193 (10Scott_French) [16:34:04] 10Phabricator, 06collaboration-services, 06DC-Ops, 10ops-codfw: phab2002: SEL System Event:, System Board Front LED Panel, Critical, management controller unavailable - https://phabricator.wikimedia.org/T420228#11745195 (10Jhancock.wm) I would definitely start with that and see if it clears the issue. the... [16:38:12] 10Phabricator, 10Bitu, 06cloud-services-team, 06Infrastructure-Foundations, and 2 others: Inconsistent OAuth endpoint for Wikimedia Global Account (SUL) across services - https://phabricator.wikimedia.org/T420599#11745208 (10A_smart_kitten) >>! In T420599#11742745, @Dondersmooi wrote: > I thought the purpo... [16:52:42] 10Continuous-Integration-Config, 07Security: Add a test to the integration/config repo that validates whether entries on the CI allowlist are correctly formatted/escaped/etc. - https://phabricator.wikimedia.org/T421143 (10A_smart_kitten) 03NEW [16:54:22] 10Continuous-Integration-Config, 06Release-Engineering-Team, 06Security-Team, 07SecTeam-Processed, and 2 others: Some entries on the CI allowlist for individual email addresses will also match other, different, email addresses - https://phabricator.wikimedia.org/T420838#11745320 (10A_smart_kitten) >>!... [17:02:52] (03update) 10dduvall: policy: Use Rego policies instead of Go templates [repos/releng/jwt-authorizer] - 10https://gitlab.wikimedia.org/repos/releng/jwt-authorizer/-/merge_requests/22 (https://phabricator.wikimedia.org/T346331) [17:12:57] (03approved) 10dduvall: Replace nginx ingress with traefik [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/562 (https://phabricator.wikimedia.org/T420743) (owner: 10dancy) [17:50:41] 10Continuous-Integration-Infrastructure: Rebuild all Jenkins agents VM to support Java 21 - https://phabricator.wikimedia.org/T421114#11745749 (10hashar) [17:52:41] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team (Doing 😎): Jenkins: In the integration project in WMCS, test out larger workers for CI - https://phabricator.wikimedia.org/T383254#11745750 (10hashar) [18:03:52] 10Continuous-Integration-Infrastructure, 07Jenkins, 06collaboration-services, 10Cloud-VPS (Quota-requests): Large temporary quota increase for integration project to ease Jenkins migration - https://phabricator.wikimedia.org/T421139#11745822 (10Dzahn) [18:04:44] 10GitLab, 06Release-Engineering-Team, 06collaboration-services, 06Research: Disable Issues in GitLab - https://phabricator.wikimedia.org/T412882#11745825 (10Dzahn) Thank you for this! That's a relief. [18:08:16] 10Phabricator, 06collaboration-services, 06DC-Ops, 10ops-codfw: phab2002: SEL System Event:, System Board Front LED Panel, Critical, management controller unavailable - https://phabricator.wikimedia.org/T420228#11745847 (10Dzahn) @Jhancock.wm Thank you! I just downtimed the server in monitoring for the nex... [19:09:01] (03update) 10gmodena: projects: add wikidata-platform/wdqs-proxy [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/160 (https://phabricator.wikimedia.org/T418837 https://phabricator.wikimedia.org/T420727) [19:12:17] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services, 07Essential-Work: Set up zuul web on zuul1001/zuul2001 - https://phabricator.wikimedia.org/T405119#11746081 (10dduvall) @Dzahn circling back to this. Thanks for pushing forward with it! Still seeing this currently: ` dduvall@... [19:13:49] (03merge) 10dancy: projects: add wikidata-platform/wdqs-proxy [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/160 (https://phabricator.wikimedia.org/T418837 https://phabricator.wikimedia.org/T420727) (owner: 10gmodena) [19:22:44] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services, 07Essential-Work: Set up zuul web on zuul1001/zuul2001 - https://phabricator.wikimedia.org/T405119#11746119 (10Dzahn) Thanks, but it already works. The cert issues were solved. But what we need is a wipe of the database. [19:28:15] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services, 07Essential-Work: Set up zuul web on zuul1001/zuul2001 - https://phabricator.wikimedia.org/T405119#11746167 (10Dzahn) Multiple issues before were related to certs and chains and multiple fixes were deployed including building a... [19:29:40] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services, 07Essential-Work: Set up zuul web on zuul1001/zuul2001 - https://phabricator.wikimedia.org/T405119#11746173 (10dduvall) Yeah, I was looking at those patches. Why do I still see connection failures in the zuul-web logs? [19:29:45] (03update) 10dancy: Replace nginx ingress with traefik [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/562 (https://phabricator.wikimedia.org/T420743) [19:35:21] (03update) 10dancy: Replace nginx ingress with traefik [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/562 (https://phabricator.wikimedia.org/T420743) [19:38:45] (03PS2) 10Sbisson: Zuul: [mediawiki/extensions/ArticleGuidance]: Add TestKitchen as phan dep [integration/config] - 10https://gerrit.wikimedia.org/r/1255746 (https://phabricator.wikimedia.org/T420385) [19:38:48] (03CR) 10Reedy: [C:03+2] Zuul: [mediawiki/extensions/ArticleGuidance]: Add TestKitchen as phan dep [integration/config] - 10https://gerrit.wikimedia.org/r/1255746 (https://phabricator.wikimedia.org/T420385) (owner: 10Sbisson) [19:40:23] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/ArticleGuidance]: Add TestKitchen as phan dep [integration/config] - 10https://gerrit.wikimedia.org/r/1255746 (https://phabricator.wikimedia.org/T420385) (owner: 10Sbisson) [19:40:54] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1255746 [19:40:56] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:26:06] (03update) 10dduvall: Replace nginx ingress with traefik [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/562 (https://phabricator.wikimedia.org/T420743) (owner: 10dancy) [20:34:30] 10Beta-Cluster-Infrastructure, 10MediaWiki-HTMLForm, 07Beta-Cluster-reproducible: Special:Import form loses configuration data when $wgImportSources contains duplicate subprojects - https://phabricator.wikimedia.org/T410109#11746621 (10matmarex) 05In progress→03Resolved [20:41:56] 10Beta-Cluster-Infrastructure, 10MediaWiki-Core-Snapshots, 10MediaWiki-HTMLForm, 07Beta-Cluster-reproducible: Special:Import form loses configuration data when $wgImportSources contains duplicate subprojects - https://phabricator.wikimedia.org/T410109#11746648 (10A_smart_kitten) [21:33:59] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for [MaryyannL] - https://phabricator.wikimedia.org/T421180 (10MCrichton-WMF) 03NEW [21:36:52] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for [MaryyannL] - https://phabricator.wikimedia.org/T421180#11746852 (10derenrich) i vouch for her [21:58:14] 10Beta-Cluster-Infrastructure, 10MediaWiki-Core-Snapshots, 10MediaWiki-HTMLForm, 07Beta-Cluster-reproducible, 05MW-1.46-notes (1.46.0-wmf.22; 2026-03-31): Special:Import form loses configuration data when $wgImportSources contains duplicate subprojects... - https://phabricator.wikimedia.org/T410109#11746902 [21:59:17] 10Continuous-Integration-Infrastructure, 10Gerrit, 06Release-Engineering-Team, 07ci-test-error (WMF-deployed Build Failure): Tests failing with "23:12:20 stderr: 'fatal: unable to access 'https://gerrit.wikimedia.org/r/REPO: GnuTLS recv error (-54): Erro... - https://phabricator.wikimedia.org/T420865#11746904 [22:02:55] 06Release-Engineering-Team, 10Catalyst (PatchDemo): patchdemo staging: Creation of new demos with catalyst backend fails on connecting to db - https://phabricator.wikimedia.org/T421181#11746919 (10brennen) [22:03:47] 06Release-Engineering-Team, 10Catalyst (PatchDemo): patchdemo staging: Creation of new demos with catalyst backend fails on connecting to db - https://phabricator.wikimedia.org/T421181#11746925 (10jeena) [22:12:20] 06Release-Engineering-Team, 10Catalyst (PatchDemo): patchdemo staging: Creation of new demos with catalyst backend fails on connecting to db - https://phabricator.wikimedia.org/T421181#11746941 (10jeena) [22:17:37] 06Release-Engineering-Team (Radar), 06SRE Observability (FY2025/2026-Q3): Upgrade Observability mwlog hosts to trixie - https://phabricator.wikimedia.org/T417002#11746967 (10Krinkle) Updated instance refs (mwlog1002 -> mw1003): * https://wikitech.wikimedia.org/wiki/Mwlog Swapped instance ref to generic ro... [22:24:21] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for [MaryyannL] - https://phabricator.wikimedia.org/T421180#11746975 (10Peachey88) @MCrichton-WMF: If you link your developer account in your phabricator settings (Settings -> External Accounts -> LDAP), I've added yo... [23:18:47] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for [MaryyannL] - https://phabricator.wikimedia.org/T421180#11747156 (10MCrichton-WMF) [23:38:35] 10Beta-Cluster-Infrastructure: Beta cluster is slow as sludge / serves 503 and 504 - https://phabricator.wikimedia.org/T420833#11747208 (10bd808) Still more Microsoft ranges found in the last 24 hours of logs: https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/b63e1c02a6dd9522ab8ef78e8fad47c...