[06:47:30] <wikibugs>	 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 06Traffic, 13Patch-For-Review: Gerrit: Debug connection re-use on Gerrit's httpd causing Gerrit interface to be very slow - https://phabricator.wikimedia.org/T420189#11736789 (10ABran-WMF) Outcome of the change:  - `AsyncRequestWorkerFactor...
[06:58:00] <wikibugs>	 10Gerrit (Gerrit 3.10): Upgrade Gerrit from 3.10.6 to 3.10.9 - https://phabricator.wikimedia.org/T400688#11736860 (10ABran-WMF)
[06:58:01] <wikibugs>	 10Gerrit (Gerrit 3.12): Upgrade to Gerrit 3.12 - https://phabricator.wikimedia.org/T392448#11736861 (10ABran-WMF)
[06:58:06] <wikibugs>	 10Gerrit (Gerrit 3.11), 06Release-Engineering-Team (Priority Backlog 📥), 06collaboration-services, 13Patch-For-Review: Upgrade to Gerrit 3.11 - https://phabricator.wikimedia.org/T379714#11736862 (10ABran-WMF)
[08:13:45] <wikibugs>	 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for [@benoistjc] - https://phabricator.wikimedia.org/T420846#11736938 (10Jean-Christophe_BENOIST) Hello !  Should be @benoistjc .  I am a little confused because I began https://meteofrancetoclimat.toolforge.org/input...
[08:54:20] <wikibugs>	 10Gerrit: [wm-checks-api] ZuulProcessor should handle skipped jobs - https://phabricator.wikimedia.org/T420885#11737028 (10hashar) p:05Triage→03Low
[09:08:56] <wikibugs>	 (03PS1) 10Phedenskog: docker: Add developer docker file and README [integration/quibble] - 10https://gerrit.wikimedia.org/r/1258947 (https://phabricator.wikimedia.org/T419583)
[09:17:34] <wikibugs>	 (03PS4) 10Hashar: Add Python 3.12 and 3.13 testing [integration/quibble] - 10https://gerrit.wikimedia.org/r/1250540 (https://phabricator.wikimedia.org/T419675)
[09:24:14] <wikibugs>	 (03CR) 10Hashar: "@jforrester@wikimedia.org I have fixed the faulty test. There is probably a better wait to mock `time.time()` and/or assert the reported t" [integration/quibble] - 10https://gerrit.wikimedia.org/r/1250540 (https://phabricator.wikimedia.org/T419675) (owner: 10Hashar)
[09:27:59] <wikibugs>	 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for [@benoistjc] - https://phabricator.wikimedia.org/T420846#11737119 (10Aklapper) 05Stalled→03Invalid Ah, thanks! The GitLab account `benoistjc` is already approved and you signed in several times; what makes...
[09:34:04] <wikibugs>	 (03PS2) 10Phedenskog: docker: Add developer docker file and README [integration/quibble] - 10https://gerrit.wikimedia.org/r/1258947 (https://phabricator.wikimedia.org/T419583)
[09:34:04] <wikibugs>	 (03CR) 10Phedenskog: "This helped me to test changes in Quibble locally." [integration/quibble] - 10https://gerrit.wikimedia.org/r/1258947 (https://phabricator.wikimedia.org/T419583) (owner: 10Phedenskog)
[09:35:18] <wikibugs>	 (03PS3) 10Hashar: Zuul: Add Akoopal as trusted user [integration/config] - 10https://gerrit.wikimedia.org/r/1258309 (owner: 10Lokal Profil)
[09:35:55] <wikibugs>	 (03CR) 10Hashar: [C:03+2] "I have slightly adjusted the commit message.  Once deployed this will allow CI to run tests for @andre@molens.org ;-]" [integration/config] - 10https://gerrit.wikimedia.org/r/1258309 (owner: 10Lokal Profil)
[09:37:06] <wikibugs>	 (03CR) 10Arthur taylor: "@phedenskog@wikimedia.org not sure if you're aware of this tool, but this also has options for reproducing CI runs with a modified Quibble" [integration/quibble] - 10https://gerrit.wikimedia.org/r/1258947 (https://phabricator.wikimedia.org/T419583) (owner: 10Phedenskog)
[09:37:35] <wikibugs>	 (03Merged) 10jenkins-bot: Zuul: Add Akoopal as trusted user [integration/config] - 10https://gerrit.wikimedia.org/r/1258309 (owner: 10Lokal Profil)
[09:49:37] <wikibugs>	 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for [@benoistjc] - https://phabricator.wikimedia.org/T420846#11737269 (10Jean-Christophe_BENOIST) I thought it is not approved because of a warning banner at top when I log in, leading to this approval process.  [...
[10:17:44] <wikibugs>	 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 06Traffic, 13Patch-For-Review: Gerrit: Debug connection re-use on Gerrit's httpd causing Gerrit interface to be very slow - https://phabricator.wikimedia.org/T420189#11737488 (10ABran-WMF) @hashar suggested aligning Gerrit's stack to what's...
[10:21:46] <wikibugs>	 06Gerrit-Privilege-Requests, 10GoogleLogin: Maintainership/Review request for GoogleLogin: patch 1074565 has been pending for over a year - https://phabricator.wikimedia.org/T414991#11737502 (10Pastakhov) 05Open→03Resolved a:03Pastakhov Hey @Florian, I'm glad that you are available. I updated the pat...
[10:41:58] <wikibugs>	 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for [@benoistjc] - https://phabricator.wikimedia.org/T420846#11737604 (10Aklapper) //[Please strip unneeded full quotes from emails to keep it readable, or use a proper delimiter (`-- `, two dashes and a space) ab...
[10:54:06] <wikibugs>	 (03PS1) 10Bhsd: Zuul: [SecurePoll] Add phan dependency on CodeMirror [integration/config] - 10https://gerrit.wikimedia.org/r/1259014
[11:15:46] <wikibugs>	 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 06Traffic, 13Patch-For-Review: ATS: align ATS and Gerrit Apache timeouts to reenable connection re-use - https://phabricator.wikimedia.org/T417998#11737711 (10hashar) @ABran-WMF and I had a long debugging session last Thursday which I have s...
[11:25:57] <wikibugs>	 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 06Traffic: gerrit: Add envoy in Gerrit's stack - https://phabricator.wikimedia.org/T420909 (10ABran-WMF) 03NEW
[11:27:03] <wikibugs>	 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 06Traffic, 13Patch-For-Review: gerrit: Add envoy in Gerrit's stack - https://phabricator.wikimedia.org/T420909#11737751 (10ABran-WMF) 05Open→03In progress p:05Triage→03Medium
[11:27:59] <wikibugs>	 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 06Traffic, 13Patch-For-Review: Gerrit: Debug connection re-use on Gerrit's httpd causing Gerrit interface to be very slow - https://phabricator.wikimedia.org/T420189#11737757 (10ABran-WMF) follow up task: {T420909}
[11:32:27] <wmf-insecte>	 Project mediawiki-core-doxygen build #18862: 04FAILURE in 14 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/18862/
[12:13:08] <wikibugs>	 (03PS3) 10Prashant_32194: fresh-install: fix fresh-node24 checksum and install validation [fresh] - 10https://gerrit.wikimedia.org/r/1227819
[12:31:12] <wmf-insecte>	 Yippee, build fixed!
[12:31:12] <wmf-insecte>	 Project mediawiki-core-doxygen build #18863: 09FIXED in 13 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/18863/
[12:46:55] <wikibugs>	 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: setup 2 contint machines for jenkins - https://phabricator.wikimedia.org/T418521#11737973 (10ABran-WMF)
[12:53:19] <wikibugs>	 (03CR) 10Jforrester: [C:03+2] Add Python 3.12 and 3.13 testing [integration/quibble] - 10https://gerrit.wikimedia.org/r/1250540 (https://phabricator.wikimedia.org/T419675) (owner: 10Hashar)
[12:54:02] <wikibugs>	 (03PS2) 10Hashar: build: upgrade flake8 3.7.x..7.3.0 [integration/quibble] - 10https://gerrit.wikimedia.org/r/1250614
[13:12:14] <wikibugs>	 (03Merged) 10jenkins-bot: Add Python 3.12 and 3.13 testing [integration/quibble] - 10https://gerrit.wikimedia.org/r/1250540 (https://phabricator.wikimedia.org/T419675) (owner: 10Hashar)
[13:15:03] <wikibugs>	 (03CR) 10Jforrester: [C:03+2] build: upgrade flake8 3.7.x..7.3.0 [integration/quibble] - 10https://gerrit.wikimedia.org/r/1250614 (owner: 10Hashar)
[13:17:11] <wikibugs>	 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 06Traffic, 13Patch-For-Review: ATS: align ATS and Gerrit Apache timeouts to reenable connection re-use - https://phabricator.wikimedia.org/T417998#11738124 (10ABran-WMF) 05Stalled→03Resolved
[13:18:32] <wikibugs>	 (03CR) 10CI reject: [V:04-1] build: upgrade flake8 3.7.x..7.3.0 [integration/quibble] - 10https://gerrit.wikimedia.org/r/1250614 (owner: 10Hashar)
[13:24:21] <wikibugs>	 10Gerrit: Switch Gerrit from Java 17 to Java 21 - https://phabricator.wikimedia.org/T392465#11738171 (10ABran-WMF)
[13:24:22] <wikibugs>	 10Gerrit (Gerrit 3.12): Upgrade to Gerrit 3.12 - https://phabricator.wikimedia.org/T392448#11738172 (10ABran-WMF)
[13:24:25] <wikibugs>	 10Gerrit (Gerrit 3.11), 06Release-Engineering-Team (Priority Backlog 📥), 06collaboration-services, 13Patch-For-Review: Upgrade to Gerrit 3.11 - https://phabricator.wikimedia.org/T379714#11738173 (10ABran-WMF)
[13:56:15] <wikibugs>	 10Gerrit, 06Release-Engineering-Team, 07Upstream: Enable visual image differences in Gerrit (was: Install gerrit image-diff plugin) - https://phabricator.wikimedia.org/T341291#11738360 (10Nemoralis) Does this plugin support SVG diffs? GitHub's compare view for SVGs is great, I especially like the "Swipe" mod...
[13:57:22] <wikibugs>	 (03PS1) 10Hashar: build: make pytest verbose by default [integration/quibble] - 10https://gerrit.wikimedia.org/r/1259110
[13:57:40] <wikibugs>	 (03CR) 10CI reject: [V:04-1] build: make pytest verbose by default [integration/quibble] - 10https://gerrit.wikimedia.org/r/1259110 (owner: 10Hashar)
[14:02:49] <wikibugs>	 (03CR) 10Hashar: "The `ReportVersionsTest.test_formatting` fails because the commands are run in parallel so there is a race condition between the command n" [integration/quibble] - 10https://gerrit.wikimedia.org/r/1250614 (owner: 10Hashar)
[14:30:21] <wikibugs>	 (03PS1) 10Jforrester: Docker: [quibble-bullseye] Drop Python 2 from images [integration/config] - 10https://gerrit.wikimedia.org/r/1259122
[14:32:47] <wikibugs>	 (03CR) 10Jforrester: "It's possibly we also don't need gcc/etc., but that's much harder to check." [integration/config] - 10https://gerrit.wikimedia.org/r/1259122 (owner: 10Jforrester)
[14:37:50] <wikibugs>	 (03PS2) 10Hashar: build: make pytest verbose by default [integration/quibble] - 10https://gerrit.wikimedia.org/r/1259110
[14:37:50] <wikibugs>	 (03PS1) 10Hashar: build: require pytest marks to be defined [integration/quibble] - 10https://gerrit.wikimedia.org/r/1259127
[14:38:08] <wikibugs>	 (03CR) 10Hashar: [C:03+2] build: upgrade flake8 3.7.x..7.3.0 [integration/quibble] - 10https://gerrit.wikimedia.org/r/1250614 (owner: 10Hashar)
[14:46:48] <wikibugs>	 (03Merged) 10jenkins-bot: build: upgrade flake8 3.7.x..7.3.0 [integration/quibble] - 10https://gerrit.wikimedia.org/r/1250614 (owner: 10Hashar)
[15:28:17] <Lucas_WMDE>	 !log ssh integration-castor05.integration.eqiad1.wikimedia.cloud sudo -u jenkins-deploy rm -rf /srv/castor/castor-mw-ext-and-skins/master/mediawiki-node20 # fix failure seen in mediawiki-node20 90272
[15:28:17] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[15:29:10] <wikibugs>	 06Release-Engineering-Team (Priority Backlog 📥), 07OKR-Work: wmf/next branch cut job on releases-jenkins and systemd timer on deployment server times overlap - https://phabricator.wikimedia.org/T404399#11738849 (10dancy) >>! In T404399#11733119, @bd808 wrote: >>>! In T404399#11174624, @bd808 wrote: >> Setting...
[15:31:05] <wikibugs>	 06Release-Engineering-Team (Priority Backlog 📥), 07OKR-Work: wmf/next branch cut job on releases-jenkins and systemd timer on deployment server times overlap - https://phabricator.wikimedia.org/T404399#11738855 (10dancy) >>! In T404399#11738849, @dancy wrote: >>>! In T404399#11733119, @bd808 wrote: >>>>! In T4...
[15:37:33] <wikibugs>	 (03open) 10bd808: jobs: Use `TZ=UTC` in branchMWSingleVersion.groovy trigger [repos/releng/jenkins-deploy] - 10https://gitlab.wikimedia.org/repos/releng/jenkins-deploy/-/merge_requests/111 (https://phabricator.wikimedia.org/T404399)
[15:37:37] <wikibugs>	 (03update) 10bd808: jobs: Use `TZ=UTC` in branchMWSingleVersion.groovy trigger [repos/releng/jenkins-deploy] - 10https://gitlab.wikimedia.org/repos/releng/jenkins-deploy/-/merge_requests/111 (https://phabricator.wikimedia.org/T404399)
[15:41:22] <wikibugs>	 (03approved) 10dancy: jobs: Use `TZ=UTC` in branchMWSingleVersion.groovy trigger [repos/releng/jenkins-deploy] - 10https://gitlab.wikimedia.org/repos/releng/jenkins-deploy/-/merge_requests/111 (https://phabricator.wikimedia.org/T404399) (owner: 10bd808)
[15:44:08] <wikibugs>	 10Continuous-Integration-Infrastructure, 07Jenkins, 06Release-Engineering-Team (Priority Backlog 📥), 07ci-test-error, and 2 others: Various CI jobs failing after "mkdir: cannot create directory ‘log’: Permission denied" - https://phabricator.wikimedia.org/T282893#11738942 (10Lucas_Werkmeister_WMDE) Still h...
[15:52:14] <wikibugs>	 (03PS1) 10GautamSudhanshu: Zuul: Add GautamSudhanshu to CI allow list [integration/config] - 10https://gerrit.wikimedia.org/r/1259146
[15:52:31] <jinxer-wm>	 FIRING: ProbeDown: Service gerrit2003:443 has failed probes (http_gerrit_tls_ip6) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit2003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown
[15:52:37] <wikibugs>	 06Release-Engineering-Team, 06collaboration-services: ProbeDown - https://phabricator.wikimedia.org/T420949 (10phaultfinder) 03NEW
[15:55:47] <wikibugs>	 10Beta-Cluster-Infrastructure, 10MediaWiki-HTMLForm, 07Beta-Cluster-reproducible: Special:Import form loses configuration data when $wgImportSources contains duplicate subprojects - https://phabricator.wikimedia.org/T410109#11739038 (10bd808)
[15:56:16] <wikibugs>	 10Beta-Cluster-Infrastructure, 10MediaWiki-HTMLForm, 07Beta-Cluster-reproducible: Special:Import form loses configuration data when $wgImportSources contains duplicate subprojects - https://phabricator.wikimedia.org/T410109#11739045 (10bd808)
[15:57:31] <jinxer-wm>	 RESOLVED: ProbeDown: Service gerrit2003:443 has failed probes (http_gerrit_tls_ip6) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit2003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown
[15:58:52] <wikibugs>	 (03PS2) 10GautamSudhanshu: Zuul: Add GautamSudhanshu to CI allow list [integration/config] - 10https://gerrit.wikimedia.org/r/1259146
[15:59:11] <wikibugs>	 10Phabricator, 06collaboration-services, 06DC-Ops, 10ops-codfw: phab2002: SEL System Event:, System Board Front LED Panel, Critical, management controller unavailable - https://phabricator.wikimedia.org/T420228#11739070 (10ABran-WMF) 05Open→03In progress p:05Triage→03Medium
[15:59:20] <wikibugs>	 (03merge) 10bd808: jobs: Use `TZ=UTC` in branchMWSingleVersion.groovy trigger [repos/releng/jenkins-deploy] - 10https://gitlab.wikimedia.org/repos/releng/jenkins-deploy/-/merge_requests/111 (https://phabricator.wikimedia.org/T404399)
[15:59:51] <wikibugs>	 (03open) 10dancy: staging.tfvars: Bump buildkit to wmf-v0.28.0 [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/563 (https://phabricator.wikimedia.org/T415284)
[15:59:52] <wikibugs>	 (03update) 10dancy: staging.tfvars: Bump buildkit to wmf-v0.28.0 [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/563 (https://phabricator.wikimedia.org/T415284)
[16:01:43] <wikibugs>	 (03merge) 10dancy: staging.tfvars: Bump buildkit to wmf-v0.28.0 [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/563 (https://phabricator.wikimedia.org/T415284)
[16:03:06] <wikibugs>	 10Phabricator, 06collaboration-services, 06DC-Ops, 10ops-codfw: phab2002: SEL System Event:, System Board Front LED Panel, Critical, management controller unavailable - https://phabricator.wikimedia.org/T420228#11739121 (10Dzahn) @Jhancock.wm This server is currently not the active production Phabricator....
[16:03:39] <wikibugs>	 06Release-Engineering-Team, 06collaboration-services: ProbeDown - gerrit2003 (IPv6) - https://phabricator.wikimedia.org/T420949#11739128 (10Dzahn)
[16:06:21] <wikibugs>	 06Release-Engineering-Team, 06collaboration-services: ProbeDown - gerrit2003 (IPv6) - https://phabricator.wikimedia.org/T420949#11739152 (10ABran-WMF) 05Open→03Resolved a:03ABran-WMF this was a transient issue, already handled in {T420595}
[16:06:30] <wikibugs>	 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 06Traffic, 13Patch-For-Review: gerrit: Add envoy in Gerrit's stack - https://phabricator.wikimedia.org/T420909#11739157 (10Dzahn) Agreed, envoy is the standard around here to terminate TLS and we do need TLS termination between ATS and the s...
[16:16:44] <wikibugs>	 10Gerrit, 06Release-Engineering-Team, 14Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 07ci-test-error (WMF-deployed Build Failure), 13Patch-For-Review: Jenkins job failing intermittently due to Gerrit HTTP 502 errors when interacting with repo... - https://phabricator.wikimedia.org/T246763#11739261
[16:18:02] <wikibugs>	 10GitLab (CI & Job Runners), 06Release-Engineering-Team, 07Essential-Work, 13Patch-For-Review: Buildkit v0.28.0 released - https://phabricator.wikimedia.org/T415284#11739280 (10dancy) @dduvall I deployed buildkit 0.28.0 to gitlab-cloud-runners staging.  It seems to be working fine but before deploying to p...
[16:22:05] <wikibugs>	 (03CR) 10Hashar: [C:03+2] "I am pairing with Jesse" [integration/config] - 10https://gerrit.wikimedia.org/r/1135116 (owner: 10JHathaway)
[16:23:54] <wikibugs>	 (03Merged) 10jenkins-bot: puppet: use bind mounts for local runs [integration/config] - 10https://gerrit.wikimedia.org/r/1135116 (owner: 10JHathaway)
[16:45:47] <wikibugs>	 (03PS1) 10JHathaway: puppet: Add libffi-dev dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1259156
[16:46:14] <wikibugs>	 (03CR) 10Hashar: [C:03+2] puppet: Add libffi-dev dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1259156 (owner: 10JHathaway)
[16:47:53] <wikibugs>	 (03Merged) 10jenkins-bot: puppet: Add libffi-dev dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1259156 (owner: 10JHathaway)
[17:05:31] <wikibugs>	 (03PS1) 10Hashar: jjb: update image for operations-puppet-tests-bullseye [integration/config] - 10https://gerrit.wikimedia.org/r/1259163
[17:08:13] <wikibugs>	 10Beta-Cluster-Infrastructure, 06SRE: Beta cluster is slow as sludge / serves 503 and 504 - https://phabricator.wikimedia.org/T420833#11739553 (10bd808) Load has been spiky over the last 7 days with increased spike frequency on 2026-03-22 for sure. {F73533258,size=full} We likely have either a new range that b...
[17:19:35] <wikibugs>	 06Release-Engineering-Team (Priority Backlog 📥), 07OKR-Work: wmf/next branch cut job on releases-jenkins and systemd timer on deployment server times overlap - https://phabricator.wikimedia.org/T404399#11739626 (10bd808) 05Open→03Resolved a:03bd808 Both jobs now use UTC for scheduling and have an hou...
[17:34:08] <wikibugs>	 06Release-Engineering-Team, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: CiviCRM tests timing out in CI - https://phabricator.wikimedia.org/T356361#11739776 (10Ejegg) Happening again. Newer ticket complaining of the same is {T406735}
[17:36:10] <wikibugs>	 10Beta-Cluster-Infrastructure, 10MediaWiki-HTMLForm, 07Beta-Cluster-reproducible: Special:Import form loses configuration data when $wgImportSources contains duplicate subprojects - https://phabricator.wikimedia.org/T410109#11739815 (10Esanders) You'd need to look at HTMLSelectField.php and how that builds t...
[18:01:40] <wikibugs>	 (03update) 10dduvall: policy: Use Rego policies instead of Go templates [repos/releng/jwt-authorizer] - 10https://gitlab.wikimedia.org/repos/releng/jwt-authorizer/-/merge_requests/22 (https://phabricator.wikimedia.org/T346331)
[18:05:18] <wikibugs>	 (03update) 10dduvall: policy: Use Rego policies instead of Go templates [repos/releng/jwt-authorizer] - 10https://gitlab.wikimedia.org/repos/releng/jwt-authorizer/-/merge_requests/22 (https://phabricator.wikimedia.org/T346331)
[18:45:47] <wikibugs>	 06Project-Admins: Create project tag for Wishlist-FY25-26 - https://phabricator.wikimedia.org/T415913#11740213 (10Aklapper) 05Open→03Stalled Any news on this?
[19:26:07] <wikibugs>	 (03update) 10dancy: Replace nginx ingress with traefik [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/562 (https://phabricator.wikimedia.org/T420743)
[19:37:43] <wikibugs>	 (03CR) 10Hashar: [C:04-1] "I don't think it is that much of a great idea, that makes the output way longer when I am just looking at having the diffs longer.  Maybe " [integration/quibble] - 10https://gerrit.wikimedia.org/r/1259110 (owner: 10Hashar)
[19:46:45] <wikibugs>	 (03CR) 10Hashar: [C:03+2] "Done by Iaa59fc925a7ed0473f35d49e56f00179ab6839fd" [integration/quibble] - 10https://gerrit.wikimedia.org/r/1250614 (owner: 10Hashar)
[19:47:59] <wikibugs>	 (03PS1) 10Hashar: Fix ReportVersion test due to parallelism [integration/quibble] - 10https://gerrit.wikimedia.org/r/1259204 (https://phabricator.wikimedia.org/T417409)
[20:04:13] <wikibugs>	 (03update) 10dduvall: policy: Use Rego policies instead of Go templates [repos/releng/jwt-authorizer] - 10https://gitlab.wikimedia.org/repos/releng/jwt-authorizer/-/merge_requests/22 (https://phabricator.wikimedia.org/T346331)
[20:05:26] <wikibugs>	 (03CR) 10Jforrester: Fix ReportVersion test due to parallelism (031 comment) [integration/quibble] - 10https://gerrit.wikimedia.org/r/1259204 (https://phabricator.wikimedia.org/T417409) (owner: 10Hashar)
[20:06:31] <wikibugs>	 (03update) 10dduvall: policy: Use Rego policies instead of Go templates [repos/releng/jwt-authorizer] - 10https://gitlab.wikimedia.org/repos/releng/jwt-authorizer/-/merge_requests/22 (https://phabricator.wikimedia.org/T346331)
[20:14:19] <wikibugs>	 10Continuous-Integration-Infrastructure: Make code coverage output tell users what commit hash coverage was generated against - https://phabricator.wikimedia.org/T420999 (10Reedy) 03NEW
[21:23:27] <wikibugs>	 10Continuous-Integration-Infrastructure, 07Jenkins, 06collaboration-services: Update Jenkins hosts from Java 17 to Java 21 - https://phabricator.wikimedia.org/T418109#11741069 (10hashar) One of the issue I flagged was whether a controller using Java 21 could communicate with agents running Java 17. They can'...
[21:46:48] <wikibugs>	 10Beta-Cluster-Infrastructure, 10MediaWiki-HTMLForm, 07Beta-Cluster-reproducible: Special:Import form loses configuration data when $wgImportSources contains duplicate subprojects - https://phabricator.wikimedia.org/T410109#11741131 (10matmarex) @bd808 Looks correct to me.  The issue here is essentially that...
[21:51:34] <wikibugs>	 (03update) 10dduvall: policy: Use Rego policies instead of Go templates [repos/releng/jwt-authorizer] - 10https://gitlab.wikimedia.org/repos/releng/jwt-authorizer/-/merge_requests/22 (https://phabricator.wikimedia.org/T346331)
[21:54:20] <wmf-insecte>	 Project mediawiki-core-doxygen build #18877: 04FAILURE in 12 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/18877/
[22:06:27] <wmf-insecte>	 Yippee, build fixed!
[22:06:28] <wmf-insecte>	 Project mediawiki-core-doxygen build #18878: 09FIXED in 12 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/18878/
[22:21:38] <wikibugs>	 (03update) 10dduvall: policy: Use Rego policies instead of Go templates [repos/releng/jwt-authorizer] - 10https://gitlab.wikimedia.org/repos/releng/jwt-authorizer/-/merge_requests/22 (https://phabricator.wikimedia.org/T346331)
[22:23:27] <wikibugs>	 (03update) 10dduvall: policy: Use Rego policies instead of Go templates [repos/releng/jwt-authorizer] - 10https://gitlab.wikimedia.org/repos/releng/jwt-authorizer/-/merge_requests/22 (https://phabricator.wikimedia.org/T346331)
[22:31:44] <wikibugs>	 (03update) 10dduvall: policy: Use Rego policies instead of Go templates [repos/releng/jwt-authorizer] - 10https://gitlab.wikimedia.org/repos/releng/jwt-authorizer/-/merge_requests/22 (https://phabricator.wikimedia.org/T346331)
[23:18:14] <wikibugs>	 10Beta-Cluster-Infrastructure, 06SRE: Beta cluster is slow as sludge / serves 503 and 504 - https://phabricator.wikimedia.org/T420833#11741525 (10bd808) https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/5d4298ce7a31d1650f6741e2b9051b82e9661c8a%5E%21/#F0 ` diff --git a/deployment-prep/_.yam...
[23:35:59] <wikibugs>	 10Beta-Cluster-Infrastructure, 06SRE: Beta cluster is slow as sludge / serves 503 and 504 - https://phabricator.wikimedia.org/T420833#11741537 (10bd808) https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/b304067573816dedc4548607ca96202083700afd%5E%21/#F0 ` diff --git a/deployment-prep/_.yam...
[23:39:20] <wikibugs>	 10Beta-Cluster-Infrastructure, 06SRE: Beta cluster is slow as sludge / serves 503 and 504 - https://phabricator.wikimedia.org/T420833#11741539 (10bd808) https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/af98ae0206c2602a25a4d88414d77291788c7f0f%5E%21/#F0 ` diff --git a/deployment-prep/_.yam...
[23:46:16] <wikibugs>	 10Beta-Cluster-Infrastructure, 06SRE: Beta cluster is slow as sludge / serves 503 and 504 - https://phabricator.wikimedia.org/T420833#11741541 (10bd808) https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/bda711470077c72c5c1d40f9b34a1f036bbd3981%5E%21/#F0 ` diff --git a/deployment-prep/_.yam...
[23:47:27] <wikibugs>	 10Beta-Cluster-Infrastructure, 06SRE: Beta cluster is slow as sludge / serves 503 and 504 - https://phabricator.wikimedia.org/T420833#11741542 (10bd808)
[23:47:30] <wikibugs>	 10Beta-Cluster-Infrastructure, 07Epic: 2025 tracking task for Beta Cluster (deployment-prep) traffic overload protection (blocking unwanted crawlers) - https://phabricator.wikimedia.org/T393487#11741543 (10bd808)
[23:47:53] <wikibugs>	 10Beta-Cluster-Infrastructure, 07Epic: 2025-2026 tracking task for Beta Cluster (deployment-prep) traffic overload protection (blocking unwanted crawlers) - https://phabricator.wikimedia.org/T393487#11741545 (10bd808)
[23:48:41] <wikibugs>	 (03PS2) 10Thcipriani: castor-save-workspacecache: check remote dir exists [integration/config] - 10https://gerrit.wikimedia.org/r/1074573 (https://phabricator.wikimedia.org/T282893)
[23:53:37] <wikibugs>	 (03CR) 10Thcipriani: castor-save-workspacecache: check remote dir exists (034 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/1074573 (https://phabricator.wikimedia.org/T282893) (owner: 10Thcipriani)
[23:56:53] <wikibugs>	 10Continuous-Integration-Infrastructure, 07Jenkins, 06Release-Engineering-Team (Priority Backlog 📥), 07ci-test-error, and 2 others: Various CI jobs failing after "mkdir: cannot create directory ‘log’: Permission denied" - https://phabricator.wikimedia.org/T282893#11741566 (10thcipriani) >>! In T282893#1173...
[23:58:37] <wikibugs>	 (03CR) 10Thcipriani: [C:03+2] castor-save-workspacecache: check remote dir exists [integration/config] - 10https://gerrit.wikimedia.org/r/1074573 (https://phabricator.wikimedia.org/T282893) (owner: 10Thcipriani)
[23:59:50] <wikibugs>	 10Beta-Cluster-Infrastructure, 06SRE: Beta cluster is slow as sludge / serves 503 and 504 - https://phabricator.wikimedia.org/T420833#11741568 (10bd808) That's most of the really active networks Beta has seen in the last 24 hours blocked. Let's see what the 15 minute load graph looks like over the next couple...