[00:09:17] !log Disable duplicate publishing noise from extension-MediaUploader, ref T143162, T389450 [00:09:19] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [00:09:20] T143162: Reduce task notification noise/frequency of changes to associated open patchsets - https://phabricator.wikimedia.org/T143162 [00:09:20] T389450: Upgrade from Sinon 1.x to Sinon 18 - https://phabricator.wikimedia.org/T389450 [00:09:59] 10Diffusion, 14Phabricator (2024-10-22), 10Release-Engineering-Team (Yak Shaving πŸƒπŸͺ’), 07Developer Productivity: Reduce task notification noise/frequency of changes to associated open patchsets - https://phabricator.wikimedia.org/T143162#10711225 (10Krinkle) @Aklapper Are the above two cases expected? [00:35:34] !log integration-agent-docker-1041 marked offline due to /srv disk space [00:35:35] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [00:50:31] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Priority Backlog πŸ“₯): Jenkins jobs leaving workdirs, filling disk on integration-agent-docker-1041 - https://phabricator.wikimedia.org/T391060 (10thcipriani) 03NEW [00:52:23] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 05Release, 05Train Deployments: 1.44.0-wmf.23 deployment blockers - https://phabricator.wikimedia.org/T386218#10711274 (10thcipriani) 05Openβ†’03Resolved [00:53:57] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Priority Backlog πŸ“₯): Jenkins jobs leaving workdirs, filling disk on integration-agent-docker-1041 - https://phabricator.wikimedia.org/T391060#10711279 (10thcipriani) We could wipe this and put it back, but left offline for some investigation:... [05:17:45] (03PS1) 10Hashar: dockerfiles: raise default python from 3.7 to 3.9 [integration/config] - 10https://gerrit.wikimedia.org/r/1134052 [05:17:49] (03CR) 10Hashar: [C:03+2] dockerfiles: raise default python from 3.7 to 3.9 [integration/config] - 10https://gerrit.wikimedia.org/r/1134052 (owner: 10Hashar) [05:19:04] (03Merged) 10jenkins-bot: dockerfiles: raise default python from 3.7 to 3.9 [integration/config] - 10https://gerrit.wikimedia.org/r/1134052 (owner: 10Hashar) [05:21:34] 10Phabricator, 06Release-Engineering-Team, 06collaboration-services, 06DBA: Prepare a database test for m3 - https://phabricator.wikimedia.org/T390034#10711534 (10Marostegui) Thank you both for the information, I will get a host ready hopefully next week. It is most likely going to be db1176. A quick check... [05:43:22] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Priority Backlog πŸ“₯): Jenkins jobs leaving workdirs, filling disk on integration-agent-docker-1041 - https://phabricator.wikimedia.org/T391060#10711562 (10hashar) > 4.9G ./quibble-composer-mysql-php83@2 > drwxr-xr-x 5 jenkins-deploy wikide... [06:55:11] 10Continuous-Integration-Infrastructure, 10MediaWiki-libs-Rdbms: Raise Quibble jobs' tested version of MariaDB to 10.6 - https://phabricator.wikimedia.org/T366646#10711601 (10hashar) In https://gerrit.wikimedia.org/r/c/integration/config/+/1133912 //Revert MardiaDB to 10.5, Wikimedia package lacks mysql_instal... [07:44:43] 10Beta-Cluster-Infrastructure, 10CirrusSearch, 10Data-Platform-SRE (2025.03.22 - 2025.04.11), 10Discovery-Search (2025.03.22 - 2025.04.11): Migrate deployment-prep elasticsearch cluster to opensearch - https://phabricator.wikimedia.org/T389971#10711702 (10Gehel) 05In progressβ†’03Resolved [08:28:24] 10Scap, 10Wikidata, 10Wikidata-Query-Service, 10Data-Platform-SRE (2025.03.22 - 2025.04.11), and 2 others: scap service restarts for WDQS are inconsistent - https://phabricator.wikimedia.org/T221709#10711857 (10Gehel) [08:51:09] (03PS1) 10Hashar: jjb: apply rebuild of revert to Mariadb 10.5 [integration/config] - 10https://gerrit.wikimedia.org/r/1134167 (https://phabricator.wikimedia.org/T366646) [08:52:51] (03CR) 10Hashar: [C:03+2] jjb: apply rebuild of revert to Mariadb 10.5 [integration/config] - 10https://gerrit.wikimedia.org/r/1134167 (https://phabricator.wikimedia.org/T366646) (owner: 10Hashar) [08:53:25] !log Updating Quibble jobs to catch up with latest image https://gerrit.wikimedia.org/r/c/integration/config/+/1134167 | T3666646 [08:53:26] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [08:54:05] (03Merged) 10jenkins-bot: jjb: apply rebuild of revert to Mariadb 10.5 [integration/config] - 10https://gerrit.wikimedia.org/r/1134167 (https://phabricator.wikimedia.org/T366646) (owner: 10Hashar) [09:06:46] (03PS1) 10Hashar: jjb: update tox jobs to default to python 3.9 [integration/config] - 10https://gerrit.wikimedia.org/r/1134168 [09:09:18] !log Update tox jobs to default to python 3.9 | https://gerrit.wikimedia.org/r/c/integration/config/+/1134168 [09:09:19] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:10:18] (03CR) 10Hashar: [C:03+2] jjb: update tox jobs to default to python 3.9 [integration/config] - 10https://gerrit.wikimedia.org/r/1134168 (owner: 10Hashar) [09:12:27] (03Merged) 10jenkins-bot: jjb: update tox jobs to default to python 3.9 [integration/config] - 10https://gerrit.wikimedia.org/r/1134168 (owner: 10Hashar) [09:20:13] (03open) 10aklapper: Log out Phab user when their SUL account is globally locked [repos/phabricator/phabricator] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/87 (https://phabricator.wikimedia.org/T338384) [09:22:04] 10Phabricator, 10Release-Engineering-Team (Doing 😎), 10MediaWiki-extensions-CentralAuth, 06Stewards-and-global-tools, 13Patch-For-Review: CentralAuth locks should disable linked Phabricator account - https://phabricator.wikimedia.org/T338384#10712004 (10Aklapper) a:03Aklapper [09:22:28] (03update) 10aklapper: Log out Phab user when their SUL account is globally locked [repos/phabricator/phabricator] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/87 (https://phabricator.wikimedia.org/T338384) [09:24:22] (03update) 10aklapper: Disable Phab user when their SUL account is globally locked [repos/phabricator/phabricator] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/87 (https://phabricator.wikimedia.org/T338384) [09:33:01] 10Phabricator, 10Release-Engineering-Team (Doing 😎), 10MediaWiki-extensions-CentralAuth, 06Stewards-and-global-tools, 13Patch-For-Review: CentralAuth locks should disable linked Phabricator account - https://phabricator.wikimedia.org/T338384#10712036 (10Bugreporter) Note we should also consider unlocks -... [09:38:24] I think we have some corrupted npm cache stuff again [09:38:29] seen at https://integration.wikimedia.org/ci/job/mwgate-node20/35782/consoleFull and https://integration.wikimedia.org/ci/job/mwgate-node20/35784/consoleFull [09:38:42] now if I could only remember what the fix for that was… [09:42:30] !log ssh integration-castor05.integration.eqiad1.wikimedia.cloud sudo -u jenkins-deploy rm -rf /srv/castor/castor-mw-ext-and-skins/master/mwgate-node20 # fix failure seen in mwgate-node20 35782 and 35784 [09:42:31] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:42:35] Lucas_WMDE: ouch :/ [09:42:50] (command based on https://phabricator.wikimedia.org/T370748#10006237 and https://sal.toolforge.org/log/WFGpY5MBFFSCpsJz1qlm) [09:42:55] 10Phabricator, 10Release-Engineering-Team (Doing 😎), 10MediaWiki-extensions-CentralAuth, 06Stewards-and-global-tools, 13Patch-For-Review: CentralAuth locks should disable linked Phabricator account - https://phabricator.wikimedia.org/T338384#10712078 (10Aklapper) Right, I was thinking of only logging out... [09:42:56] +1 [09:51:47] 10Phabricator, 10Release-Engineering-Team (Doing 😎), 10MediaWiki-extensions-CentralAuth, 06Stewards-and-global-tools, 13Patch-For-Review: CentralAuth locks should disable linked Phabricator account - https://phabricator.wikimedia.org/T338384#10712093 (10Bugreporter) Even if removal of global locks is not... [10:11:11] 10Continuous-Integration-Infrastructure, 07Jenkins: Manage provisioning and configuration of Jenkins agents using OpenTofu - https://phabricator.wikimedia.org/T389891#10712159 (10hashar) The runbook can probably be simplified a bit: * drop the local Puppetserver, I don't think it is needed anymore now that th... [10:18:02] 10Deployments, 06serviceops, 10Shellbox, 10Wikibase-Quality-Constraints, and 5 others: Burst of GuzzleHttp Exception for http://localhost:6025/call/constraint-regex-checker - https://phabricator.wikimedia.org/T371633#10712169 (10karapayneWMDE) 05Openβ†’03Resolved a:03karapayneWMDE [10:28:14] 10Phabricator, 10Release-Engineering-Team (Doing 😎), 10MediaWiki-extensions-CentralAuth, 06Stewards-and-global-tools, 13Patch-For-Review: CentralAuth locks should disable linked Phabricator account - https://phabricator.wikimedia.org/T338384#10712217 (10Aklapper) There is no way to store //why// a Phab a... [10:42:09] hashar: I'm working on fixing the issues and analysing the logs in an ether pad. It's mostly a small number of issue repeated [10:43:21] https://etherpad.wikimedia.org/p/ng8EYRhQ7SIXy7cLBexZ [10:44:20] BlueSpice is 50% of it. [10:44:53] then another 20-30% is stuff depending on Cite which in turn has a test for optional VE code but that VE-test is missing a "skip" for when VE is not installed. [10:44:56] I'll fix that next week [10:45:43] There's also a number of false positives (i.e. already failing on master branch) but that's hard to exclude [10:46:13] what's easier to exclude is cases where there was already no recursion, that means the failure must be unreealted [10:46:34] e.g. "TuleapSkin" has 2 depends which have either no deps or point to one of hte other, so before/after is identical. [11:00:47] 10Phabricator, 10Release-Engineering-Team (Doing 😎), 10MediaWiki-extensions-CentralAuth, 06Stewards-and-global-tools, 13Patch-For-Review: CentralAuth locks should disable linked Phabricator account - https://phabricator.wikimedia.org/T338384#10712306 (10Bugreporter) But at least it is known who is disabl... [11:01:12] 10Continuous-Integration-Infrastructure, 07Zuul: Provide a way for "bot" patch providers (like LibIp or lsc) to be low-priority, and for zuul to try to merge them later after human-C+2'ed patches - https://phabricator.wikimedia.org/T389535#10712311 (10hashar) 05Openβ†’03Declined This is not possible in Z... [11:10:58] 10Phabricator: Deactivate account @Wfan - https://phabricator.wikimedia.org/T390997#10712356 (10Aklapper) [11:14:33] 10Phabricator, 10Release-Engineering-Team (Doing 😎), 10MediaWiki-extensions-CentralAuth, 06Stewards-and-global-tools, 13Patch-For-Review: CentralAuth locks should disable linked Phabricator account - https://phabricator.wikimedia.org/T338384#10712366 (10Aklapper) Note that our [Antivandalism code](https:... [11:16:00] 10Phabricator, 10Release-Engineering-Team (Doing 😎), 10MediaWiki-extensions-CentralAuth, 06Stewards-and-global-tools, 13Patch-For-Review: CentralAuth locks should disable linked Phabricator account - https://phabricator.wikimedia.org/T338384#10712371 (10Bugreporter) Yeah, so accounts without such entries... [11:26:18] 10Phabricator, 10Release-Engineering-Team (Doing 😎), 10MediaWiki-extensions-CentralAuth, 06Stewards-and-global-tools, 13Patch-For-Review: CentralAuth locks should disable linked Phabricator account - https://phabricator.wikimedia.org/T338384#10712403 (10Bugreporter) >>! In T338384#10712366, @Aklapper wro... [11:41:40] 10Phabricator, 10Release-Engineering-Team (Doing 😎), 10MediaWiki-extensions-CentralAuth, 06Stewards-and-global-tools, 13Patch-For-Review: CentralAuth locks should disable linked Phabricator account - https://phabricator.wikimedia.org/T338384#10712480 (10Aklapper) Oh true, it is listed in the "global" fee... [12:34:00] (03CR) 10D3r1ck01: [C:03+1] commands: Fix casing in "Run QUnit tests" section label [integration/quibble] - 10https://gerrit.wikimedia.org/r/1134091 (owner: 10Krinkle) [13:19:57] (03CR) 10Jforrester: "Dupe of Ibd2ab881cea9ff70afac2114ae4e4024cf63a7be. I was waiting for the weekend when it wouldn't disrupt people as much." [integration/config] - 10https://gerrit.wikimedia.org/r/1134167 (https://phabricator.wikimedia.org/T366646) (owner: 10Hashar) [13:20:07] (03Abandoned) 10Jforrester: jjb: Upgrade Quibble bullseye jobs to latest (no-op) [integration/config] - 10https://gerrit.wikimedia.org/r/1133920 (owner: 10Jforrester) [13:38:35] (03update) 10jforrester: releases: Bump api-testing to 1.7.1 [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/70 (owner: 10daimona) [13:43:46] James_F: isn't it up-to-date already? Or otherwise I'm not sure what to do with it. [13:44:06] Daimona: Isn't what? [13:44:20] The MR just above [13:44:37] GitLab insists it's not and can't be merged. [13:44:43] * James_F refreshes. [13:44:47] Huh?! [13:44:52] Eurgh. Dear GitLab, you suck. [13:45:00] (03merge) 10jforrester: releases: Bump api-testing to 1.7.1 [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/70 (owner: 10daimona) [13:45:01] There. [13:45:17] Hah! Gerrit does that too btw. It can be confusing at times. [13:46:41] "that" being: when you go navigate to the previous page, and other scenarios that I haven't fully pinned down, it'll show an outdated version [13:47:14] I blame race conditions, which is the programmer way of saying 'ghosts'. [13:47:16] Refreshing updates it but oftentimes it's not immediately clear that you have to do it [14:26:57] 10Continuous-Integration-Infrastructure, 10Testing Support, 10ci-test-error (WMF-deployed Build Failure), 10MW-1.44-notes (1.44.0-wmf.23; 2025-04-01), and 2 others: Selenium timeouts can cause the job to remain stuck until the build times out - https://phabricator.wikimedia.org/T389536#10713035 (10zeljkofil... [15:04:19] (03CR) 10Hashar: [C:03+2] "Thank you!!! 😎" [integration/quibble] - 10https://gerrit.wikimedia.org/r/1134091 (owner: 10Krinkle) [15:19:37] (03open) 10swfrench: kubernetes: remove support for the deprecated debug boolean [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/741 (https://phabricator.wikimedia.org/T389499) [15:25:47] (03Merged) 10jenkins-bot: commands: Fix casing in "Run QUnit tests" section label [integration/quibble] - 10https://gerrit.wikimedia.org/r/1134091 (owner: 10Krinkle) [15:48:12] 10Continuous-Integration-Config: Jenkins 'test' pipeline is not running on rebase of patch set with CodeReview+2 - https://phabricator.wikimedia.org/T358900#10713463 (10Umherirrender) 05Openβ†’03Invalid The result of 5f18d3cd2dad5f65539d9919a1528496336ee3c2, but wanted behaviour (T357080) [16:43:56] (03CR) 10Hashar: "recheck after having changed tox default python from 3.7 to 3.9 - https://gerrit.wikimedia.org/r/c/integration/config/+/1134168?usp=dashbo" [integration/gerrit-admin] - 10https://gerrit.wikimedia.org/r/1134018 (https://phabricator.wikimedia.org/T390719) (owner: 10Hashar) [16:50:45] (03merge) 10dancy: kubernetes: remove support for the deprecated debug boolean [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/741 (https://phabricator.wikimedia.org/T389499) (owner: 10swfrench) [17:17:06] 10Continuous-Integration-Infrastructure, 07Jenkins: Manage provisioning and configuration of Jenkins agents using OpenTofu - https://phabricator.wikimedia.org/T389891#10713866 (10bd808) >>! In T389891#10712159, @hashar wrote: > I am not sure it is worth the investment given the #zuul-upgrade we lead to decommi... [17:34:38] 10Scap (SpiderPig πŸ•ΈοΈ): session cookie keeps being updated/ regenerated - https://phabricator.wikimedia.org/T390078#10713941 (10dancy) 05Openβ†’03Declined [17:57:53] 06Project-Admins: Create project tag for - https://phabricator.wikimedia.org/T391140 (10Eugene233) 03NEW [17:58:39] 06Project-Admins: Create project tag for WdTmCollab - https://phabricator.wikimedia.org/T391140#10714024 (10Eugene233) [18:02:53] 06Project-Admins: Create project tag for WdTmCollab - https://phabricator.wikimedia.org/T391140#10714035 (10Eugene233) [18:36:41] (03PS1) 10MΓ‘tΓ© SzabΓ³: zuul: Test IPInfo with MobileFrontend [integration/config] - 10https://gerrit.wikimedia.org/r/1134278 (https://phabricator.wikimedia.org/T345639) [20:03:14] 06Project-Admins: Create project tag for WdTmCollab - https://phabricator.wikimedia.org/T391140#10714313 (10xSavitar) Done: https://phabricator.wikimedia.org/project/view/7831/ Feel free to resolve the task if all looks good. [21:37:54] (03open) 10dancy: interaction.py: Downcase y/n response [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/742 (https://phabricator.wikimedia.org/T380924) [21:37:56] (03update) 10dancy: interaction.py: Downcase y/n response [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/742 (https://phabricator.wikimedia.org/T380924) [21:40:31] (03merge) 10dancy: interaction.py: Downcase y/n response [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/742 (https://phabricator.wikimedia.org/T380924) [21:44:10] (03open) 10dancy: kubernetes.py: Fix monitoring of deployment to fresh cluster [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/743 [21:44:13] (03update) 10dancy: kubernetes.py: Fix monitoring of deployment to fresh cluster [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/743 [21:44:35] (03update) 10dancy: kubernetes.py: Fix monitoring of deployment to fresh cluster [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/743 [21:50:46] 10Scap, 06serviceops: Migrate scap's maintenance script invocations to PHP 8.1 - https://phabricator.wikimedia.org/T390225#10714565 (10Scott_French) I reviewed the scripts today, and aside from `rebuildLocalisationCache.php`, these all appear to be fairly straightforward in their implementation and / or use ca... [21:59:53] (03approved) 10swfrench: kubernetes.py: Fix monitoring of deployment to fresh cluster [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/743 (owner: 10dancy) [22:16:09] 10Release-Engineering-Team (Doing 😎), 07OKR-Work, 10Test-Platform (Radar): [WE6.2.6] Create design document for Pretrain (nΓ©e Group -1) deployment - https://phabricator.wikimedia.org/T379683#10714639 (10bd808) 05In progressβ†’03Resolved a:03bd808 The final report on this hypothesis can be seen at htt... [23:23:57] (03approved) 10bd808: web: Use vite-plugin-vuetify [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/740 (owner: 10dancy) [23:25:03] maintenance-disconnect-full-disks build 690172 integration-agent-docker-1065 (/: 21%, /srv: 95%, /var/lib/docker: 48%): OFFLINE due to disk space [23:26:10] (03update) 10bd808: web: Use vite-plugin-vuetify [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/740 (owner: 10dancy) [23:29:08] (03merge) 10dancy: web: Use vite-plugin-vuetify [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/740 [23:30:03] maintenance-disconnect-full-disks build 690173 integration-agent-docker-1065 (/: 21%, /srv: 50%, /var/lib/docker: 48%): RECOVERY disk space OK [23:35:06] (03open) 10bd808: spiderpig: manually split vendor chunks [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/744 [23:35:09] (03update) 10bd808: spiderpig: manually split vendor chunks [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/744 [23:41:17] (03update) 10bd808: spiderpig: manually split vendor chunks [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/744 [23:54:08] (03update) 10bd808: spiderpig: manually split vendor chunks [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/744