[00:21:56] my CI extensions wiki is now passing tests on PHP 8.0, --testsuite=extensions [00:22:54] I haven't updated its extension set, it's pretty old, but there is a chance that PHP 8.0 extension tests will pass in CI now [00:54:45] hashar: this may or may not be another npm cache corruption :( https://integration.wikimedia.org/ci/job/mwext-node12-rundoc-docker/10446/console [00:54:52] I’ll try another gate-and-submit [00:54:57] (but might go to sleep before it finishes tbh) [00:55:48] that would be on integration-agent-docker-1009, in case that matters [00:56:05] oh wait, nevermind, this is ENOSPC. so hopefully no persistent corruption [00:57:57] 10Phabricator, 10Security-Team, 10SecTeam-Processed, 10Security: Audit members of acl*security for more than x duration of no activity (Jan 2022) - https://phabricator.wikimedia.org/T299400 (10Aklapper) a:03Dsharpe Thanks. I see that three accounts were removed. > Some accounts listed didn't show up und... [03:48:24] 10MediaWiki-Releasing: Provide machine readable information about MediaWiki versions - https://phabricator.wikimedia.org/T294746 (10Lens0021) How about reusing https://www.wikidata.org/wiki/Q83#P348 Wikidata statement or its JSON representation https://www.wikidata.org/wiki/Special:EntityData/Q83.json, though it... [07:09:56] (03CR) 10Awight: Split core npm and composer tests (032 comments) [integration/quibble] - 10https://gerrit.wikimedia.org/r/588087 (owner: 10Awight) [07:32:05] !log integration-castor03:/srv/jenkins-workspace/caches$ sudo rm -rf castor-mw-ext-and-skins/ [07:32:06] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [07:53:28] (03PS1) 10Brian Wolff: Make TimedMediaHandler depend on QuickInstantCommons for phan [integration/config] - 10https://gerrit.wikimedia.org/r/756924 [08:15:36] 10Beta-Cluster-Infrastructure, 10Parsoid, 10Wikimedia-Site-requests, 10Sustainability (Incident Followup): Enable pig latin variant on English Wikipedia beta cluster - https://phabricator.wikimedia.org/T299975 (10Peachey88) [08:22:41] (03CR) 10Kosta Harlan: "recheck" [integration/quibble] - 10https://gerrit.wikimedia.org/r/588087 (owner: 10Awight) [08:22:53] (03CR) 10Awight: "This change is ready for review." [integration/quibble] - 10https://gerrit.wikimedia.org/r/756946 (owner: 10Awight) [08:28:21] the world is slowly exploding [08:36:30] integration-agent-docker-1009 had a full disk [08:36:43] 7148 ./wmf-quibble-selenium-php72-docker@3 [08:36:51] 5170 ./mwext-node12-rundoc-docker@2 [08:37:13] and somehow the disk hasn't been wiped [08:57:20] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team: Cancelled Jenkins build do not run postbuild script - https://phabricator.wikimedia.org/T299995 (10hashar) [09:12:39] (03CR) 10Awight: "This change is ready for review." [integration/config] - 10https://gerrit.wikimedia.org/r/756949 (https://phabricator.wikimedia.org/T259690) (owner: 10Awight) [09:17:07] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: Puppet failure on integration-agent-qemu* - https://phabricator.wikimedia.org/T299996 (10hashar) [09:17:10] 10Phabricator, 10Release-Engineering-Team: Puppet failure on phabricator-stage-1001.devtools - https://phabricator.wikimedia.org/T299997 (10hashar) [09:17:59] (03CR) 10WMDE-Fisch: [C: 03+1] Remove outdated beta feature dependency for FileExporter [integration/config] - 10https://gerrit.wikimedia.org/r/756949 (https://phabricator.wikimedia.org/T259690) (owner: 10Awight) [09:18:43] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: Puppet failures on integration-agent-qemu hosts - https://phabricator.wikimedia.org/T299836 (10Majavah) [09:18:45] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: Puppet failure on integration-agent-qemu* - https://phabricator.wikimedia.org/T299996 (10Majavah) [09:22:57] !log unblocked beta again [09:22:57] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:30:31] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: Puppet failure on integration-agent-qemu* - https://phabricator.wikimedia.org/T299996 (10hashar) The instances are using: | Instance | Flavor |--|-- | integration-agent-qemu-1001 | g2.cores8.ram24.disk80 | integration-agent-qemu-1003 | g3.... [09:31:29] T300000 [09:31:29] T300000: Use capitallinkoverride for gadget namespace - https://phabricator.wikimedia.org/T300000 [09:33:03] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: Puppet failure on integration-agent-qemu* - https://phabricator.wikimedia.org/T299996 (10hashar) The puppet failure: ` (/Stage[main]/Profile::Ci::Dockervolume/Labs_lvm::Volume[docker]/Exec[available-space-docker]/returns) Less than 1.5G is av... [09:35:12] \o/ [09:43:57] !logintegration-agent-qemu-1003: nuked /dev/vd/second-local-disk and /srv to make room for a docker logical volume. That has fixed puppet T299996 [09:43:58] T299996: Puppet failure on integration-agent-qemu* - https://phabricator.wikimedia.org/T299996 [09:51:00] !log integration-agent-qemu-1003: nuked /dev/vd/second-local-disk and /srv to make room for a docker logical volume. That has fixed puppet T299996 [09:51:01] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:51:02] T299996: Puppet failure on integration-agent-qemu* - https://phabricator.wikimedia.org/T299996 [09:51:33] !log integration-agent-qemu-1001: resizing /dev/mapper/vd-second--local--disk (/srv) to 20G : `resize2fs -p /dev/mapper/vd-second--local--disk 20G` [09:51:34] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:59:18] !log integration-agent-qemu-1001: resizing /dev/mapper/vd-second--local--disk (/srv) to 20G : `resize2fs -p /dev/mapper/vd-second--local--disk 20G` # T299996 [09:59:19] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:59:20] T299996: Puppet failure on integration-agent-qemu* - https://phabricator.wikimedia.org/T299996 [09:59:22] !log integration-agent-qemu-1001: resized /srv to 100% disk free: `lvextend -r -l +100%FREE /dev/mapper/vd-second--local--disk` # T299996 [09:59:24] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:59:58] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: Puppet failure on integration-agent-qemu* - https://phabricator.wikimedia.org/T299996 (10hashar) 05duplicate→03Resolved Solved on integration-agent-qemu-1001 as well: ` root@integration-agent-qemu-1001:~# vgs VG #PV #LV #SN Attr VSize... [10:00:00] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: Puppet failure on integration-agent-qemu* - https://phabricator.wikimedia.org/T299996 (10hashar) a:03hashar [10:02:05] !log integration: removing usage of `role::ci::slave::labs::docker::docker_lvm_volume` in Horizon following https://gerrit.wikimedia.org/r/c/operations/puppet/+/755948 . Docker role instances now always have a 24G partition for Docker [10:02:05] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:05:35] (03PS3) 10Majavah: Stop branching UserMerge [tools/release] - 10https://gerrit.wikimedia.org/r/755535 (https://phabricator.wikimedia.org/T216089) [10:05:51] ^ anyone kind enough to +2 that? [10:07:53] 10Release-Engineering-Team (Seen), 10MediaWiki-extensions-UserMerge, 10Stewards-and-global-tools, 10MW-1.38-notes (1.38.0-wmf.13; 2021-12-13), 10Patch-For-Review: Undeploy UserMerge Extension from WMF production - https://phabricator.wikimedia.org/T216089 (10Majavah) [10:08:45] (03CR) 10Reedy: [C: 03+2] "buh-bye" [tools/release] - 10https://gerrit.wikimedia.org/r/755535 (https://phabricator.wikimedia.org/T216089) (owner: 10Majavah) [10:08:50] thanks reedy [10:09:28] (03Merged) 10jenkins-bot: Stop branching UserMerge [tools/release] - 10https://gerrit.wikimedia.org/r/755535 (https://phabricator.wikimedia.org/T216089) (owner: 10Majavah) [10:10:29] T300000 \o/ [10:10:29] T300000: Use capitallinkoverride for gadget namespace - https://phabricator.wikimedia.org/T300000 [10:18:29] 10Release-Engineering-Team (Seen), 10MediaWiki-extensions-UserMerge, 10Stewards-and-global-tools, 10MW-1.38-notes (1.38.0-wmf.13; 2021-12-13), 10Patch-For-Review: Undeploy UserMerge Extension from WMF production - https://phabricator.wikimedia.org/T216089 (10Majavah) [10:21:33] 10Release-Engineering-Team (Seen), 10MediaWiki-extensions-UserMerge, 10Stewards-and-global-tools, 10MW-1.38-notes (1.38.0-wmf.13; 2021-12-13), 10Patch-For-Review: Undeploy UserMerge Extension from WMF production - https://phabricator.wikimedia.org/T216089 (10Majavah) 05Open→03Resolved This is complete. [10:31:41] (03CR) 10Kosta Harlan: [C: 03+1] Split core npm and composer tests (031 comment) [integration/quibble] - 10https://gerrit.wikimedia.org/r/588087 (owner: 10Awight) [10:37:49] (03CR) 10Kosta Harlan: Render an activity diagram for the main pipeline (031 comment) [integration/quibble] - 10https://gerrit.wikimedia.org/r/756946 (owner: 10Awight) [10:47:02] Lucas_WMDE: guten tag. The failure you had over night was integration-agent-docker-1009 having a full disk [10:47:21] we do wipe the workspace after the build has completed, but that apparently does not work when a build is cancelled by Zuul :-\\ [10:47:53] and well npm run doc on Wikibase takes ~ 2GB while a Wikibase wmf-quibble runs takes ~ 8G [10:48:08] so if they pill up on the same instance, that escaltes quickly :] [10:48:33] the problem is not solved and will happen again, but at least I have filed some tasks [10:51:13] (03CR) 10Kosta Harlan: "recheck" [integration/quibble] - 10https://gerrit.wikimedia.org/r/756543 (owner: 10Kosta Harlan) [10:52:15] hashar: ok, thanks [10:52:47] I guess I’ll avoid cancelling builds then, I saw it as a way to save resources but if the builds pile up it’s probably not a good idea [11:11:13] (03PS5) 10Kosta Harlan: GitClean: Default to background processing [integration/quibble] - 10https://gerrit.wikimedia.org/r/756543 [11:14:49] (03CR) 10jerkins-bot: [V: 04-1] GitClean: Default to background processing [integration/quibble] - 10https://gerrit.wikimedia.org/r/756543 (owner: 10Kosta Harlan) [11:15:18] 10Continuous-Integration-Config, 10Quibble: EEXIST: file already exists on npm install in Selenium job - https://phabricator.wikimedia.org/T300018 (10kostajh) [11:19:07] (03PS6) 10Kosta Harlan: GitClean: Default to background processing [integration/quibble] - 10https://gerrit.wikimedia.org/r/756543 [11:28:20] (03CR) 10jerkins-bot: [V: 04-1] GitClean: Default to background processing [integration/quibble] - 10https://gerrit.wikimedia.org/r/756543 (owner: 10Kosta Harlan) [12:44:30] Lucas_WMDE: Zuul cancels build by itself when some change ahead in the queue fails it will cancel builds for all other changes behind [12:44:38] so if you have change A > B > C all having jobs running [12:44:48] 😬 [12:44:53] if A is determinated to be failling, it is dropped from the queue which becomes B > C [12:44:59] and all the jobs triggered for B and C are cancelled [12:45:03] which Jenkins abort [12:45:08] so all those failed gate-and-submits I kept retrying last night probably didn’t help [12:45:17] and somehow Jenkins when aborting kill the whole process group [12:45:43] which end up killing (SIGKILL) the cleanup task being run in the post build script [12:46:07] so essentially it is a bit of an issue in Jenkins itself [12:46:32] does the next run have a chance to clean up the workspace or do they all get separate directories? [12:47:33] I’m trying to understand why this isn’t a much bigger issue than it seems to be, since Zuul-cancelled builds probably happen all the time then [12:48:13] hmm [12:48:44] the jobs create some directories on the host: cache log and src [12:49:05] with somthing like mkdir X && docker run --entrypoint=rm -fR X [12:49:25] so eventually the left over bits will be pruned whenever the same job happen to run on the same agent [12:49:35] hm [12:49:35] so if job npm-docker fails and leave stuff behind [12:49:47] the next build of npm-docker job on that agent will delete the material [12:49:59] then if two build of a job happens on the same agent, they will use different workspaces [12:50:11] Jenkins append a "@" to the directory name [12:50:27] so two builds of a job named npm-docker happening concurrently on the same agent will result in directories: [12:50:37] npm-docker and npm-docker@2 [12:51:00] if the build that created npm-docker@2 is cancelled, the bits are left behind [12:51:01] and that will only be cleaned up the next time two npm-docker’s run in parallel? [12:51:13] and will not be removed until npm-docker is run twice again on the same agent [12:51:16] yeah [12:51:17] damn [12:51:23] so previously [12:51:37] we spinned of an entirely fresh virtual machine for each new build [12:51:52] so we never cared about cleaning up cause at the end of the build the VM and all its data got entirely deleted [12:52:13] but nowadays we have that half backed system that has a bunch of leaks here and there :-\ [12:52:18] :/ [12:52:27] I imagine it’s faster, but yeah… [12:58:48] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team, 10Upstream: Cancelled Jenkins build do not run postbuild script - https://phabricator.wikimedia.org/T299995 (10hashar) My fringe theory of how it happens is: Zuul cancels a build Jenkins sends SIGTERM to the process group whi... [12:58:57] Lucas_WMDE: I wrote a summary on the task [12:59:06] thanks [12:59:10] but I think the issue is that Jenkins core assumes it has to kill every single process in the build [12:59:33] while the Post Build Script plugin we use spawns some commands while Jenkins is killing them all [12:59:41] the sequence should probably be: [12:59:44] - Jenkins kill process [12:59:57] - once finished sending SIGTERM / SIGKILL hand off to plugins [13:00:04] - postbuild script plugin execute its commands [13:00:16] lot of fun [13:01:01] is there anything I can do to help when experiencing ENOSPC errors in a build? [13:01:14] so far I’ve usually rechecked and it worked, so I was assuming they resolved themselves [13:01:16] I am going to rebuild all instances to have larger disks [13:01:27] I did some step last week to have a larger partition which will help a bit [13:01:37] but the next step is to rebuild all agents with a bit more disk space [13:01:44] ok [13:01:54] it is pending on having another flavor added to openstack which should be done soonish [13:02:03] then well [13:02:25] create instances with the new flavor, add puppet to it, get a /srv that is a few gigabytes larger [13:02:44] and in parallel I guess I should fill a task for Jenkins to solve the issue [13:02:47] ;) [13:03:03] more or less related: I am not sure why Wikibase node modules take soooo much disk space [13:03:15] (which clearly CI should support properly nonetheless) [13:03:25] probably related to us having several Node projects in that git repo [13:03:27] I guess Wikibase material has a lot of dependencies [13:30:39] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Radar), 10Security-Team, 10serviceops, and 2 others: Setup GitLab Runner in trusted environment - https://phabricator.wikimedia.org/T295481 (10ops-monitoring-bot) Icinga downtime set by jelto@cumin1001 for 6:00:00 1 host(s) and their services with re... [13:30:58] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team, 10Upstream: Cancelled Jenkins build do not run postbuild script - https://phabricator.wikimedia.org/T299995 (10hashar) I have created a dummy job https://integration.wikimedia.org/ci/job/T2999995%20-%20cancel%20and%20postbuild/... [13:35:37] hashar: well, I managed to cut our non-nested node_modules almost in half https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/756977 [13:36:07] it was probably not the most significant part anyways, the nested Data Bridge etc. node_modules/ are probably much larger [13:36:09] but still [13:44:18] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team, 10Upstream: Cancelled Jenkins build do not run postbuild script - https://phabricator.wikimedia.org/T299995 (10hashar) The issue is more complicated in https://integration.wikimedia.org/ci/job/mwext-node12-rundoc-docker/10357/... [13:44:48] !log Jenkins CI: added Logger https://integration.wikimedia.org/ci/log/ProcessTree%20-%20T299995/ to watch `hudson.util.ProcessTree` for T299995 [13:44:53] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:44:53] T299995: Cancelled Jenkins build do not run postbuild script - https://phabricator.wikimedia.org/T299995 [13:45:52] Lucas_WMDE: OH NICE [13:47:19] Lucas_WMDE: though removing the dependency drop a lot of mega bytes from node_modules so I guess plotly.js has a lot of dependencies? [13:47:24] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Radar), 10Security-Team, 10serviceops, and 2 others: Setup GitLab Runner in trusted environment - https://phabricator.wikimedia.org/T295481 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by jelto@cumin1001 for hosts: `gitlab-runner1... [13:47:31] but your script only download /1.58.5/plotly.min.js so it would miss all the dependencies? [13:50:56] well, we never used the dependencies to begin with [13:51:07] the old version of the script copied a single JS file out of node_modules [13:51:35] (but never *sourced* any of the related JS files) [13:51:45] we were just using npm as a downloader basically [14:10:57] (03PS1) 10Majavah: Review access change [puppet] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/756989 [14:15:00] (03PS2) 10Majavah: Review access change [puppet] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/756989 [14:16:01] (03CR) 10Majavah: "This access change lets members of the NDA and WMF groups (who have permissions to launch manual Jenkins jobs which lets them use the `pcc" [puppet] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/756989 (owner: 10Majavah) [14:19:01] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team, 10Upstream: Cancelled Jenkins build do not run postbuild script - https://phabricator.wikimedia.org/T299995 (10hashar) Ah related investigation from 2018 is at T176747. It referred to https://github.com/jenkinsci/jenkins/commit... [14:19:58] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team, 10Upstream: Cancelled Jenkins build do not run postbuild script - https://phabricator.wikimedia.org/T299995 (10hashar) Will revisit later. It is a bit lof a long rabbit hole to find out what is wrong between Jenkins, docker cli... [14:20:15] Lucas_WMDE: well that is surely a nice improvement :] [14:20:51] 00:12:59.376 [0-9] 2022-01-25T13:49:03.307Z ERROR @wdio/runner: Error: connect ECONNREFUSED 127.0.0.1:43429 [14:20:51] 00:13:15.960 [0-9] at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1148:16) [14:20:51] 00:13:15.960 [0-9] FAILED in chrome - /tests/selenium/specs/watchstar.js [14:20:52] pff [14:23:40] (03PS3) 10Majavah: Review access change [puppet] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/756989 [14:41:24] (03CR) 10Jforrester: "🎉" [tools/release] - 10https://gerrit.wikimedia.org/r/755535 (https://phabricator.wikimedia.org/T216089) (owner: 10Majavah) [14:46:51] it passed now, phew ^^ [15:00:02] polished the change a bit to also hash the file while downloading it [15:37:29] Hello everyone. [15:46:07] (03PS1) 10Ahmon Dancy: Release 4.2.0-1 [tools/scap] - 10https://gerrit.wikimedia.org/r/757019 [15:46:28] (03CR) 10Ahmon Dancy: [C: 03+2] Release 4.2.0-1 [tools/scap] - 10https://gerrit.wikimedia.org/r/757019 (owner: 10Ahmon Dancy) [15:49:14] (03CR) 10jerkins-bot: [V: 04-1] Release 4.2.0-1 [tools/scap] - 10https://gerrit.wikimedia.org/r/757019 (owner: 10Ahmon Dancy) [15:50:41] (03CR) 10Ahmon Dancy: [C: 03+1] "OK with me." [tools/release] - 10https://gerrit.wikimedia.org/r/756695 (owner: 10Thcipriani) [15:50:44] * James_F hashar: Hey, is now an OK time to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/715144 adding an extension to the gate? CI looks relatively clean. [15:54:05] (03PS4) 10Jforrester: Zuul: [mediawiki/extensions/Math] Add Math to the main gate [integration/config] - 10https://gerrit.wikimedia.org/r/715144 (https://phabricator.wikimedia.org/T232948) [15:54:14] (03CR) 10Ahmon Dancy: Release 4.2.0-1 [tools/scap] - 10https://gerrit.wikimedia.org/r/757019 (owner: 10Ahmon Dancy) [15:54:18] (03CR) 10Ahmon Dancy: [C: 03+2] Release 4.2.0-1 [tools/scap] - 10https://gerrit.wikimedia.org/r/757019 (owner: 10Ahmon Dancy) [15:55:04] (03CR) 10Jforrester: [C: 03+2] Zuul: [mediawiki/extensions/Math] Add Math to the main gate [integration/config] - 10https://gerrit.wikimedia.org/r/715144 (https://phabricator.wikimedia.org/T232948) (owner: 10Jforrester) [15:55:17] (03Merged) 10jenkins-bot: Release 4.2.0-1 [tools/scap] - 10https://gerrit.wikimedia.org/r/757019 (owner: 10Ahmon Dancy) [15:56:11] (03PS5) 10Jforrester: Bundle Math extension with MediaWiki [tools/release] - 10https://gerrit.wikimedia.org/r/715082 (https://phabricator.wikimedia.org/T232948) [15:57:17] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/Math] Add Math to the main gate [integration/config] - 10https://gerrit.wikimedia.org/r/715144 (https://phabricator.wikimedia.org/T232948) (owner: 10Jforrester) [15:57:38] !log Zuul: [mediawiki/extensions/Math] Add Math to the main gate for T232948 [15:57:41] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:57:41] T232948: Bundle Math extension with MediaWiki - https://phabricator.wikimedia.org/T232948 [15:58:36] (Queue (Jenkins jobs + Zuul functions) alert) firing: Queue (Jenkins jobs + Zuul functions) alert - https://alerts.wikimedia.org [15:58:56] TAR_ENTRY_ERROR ENOSPC errros. [15:59:21] On integration-agent-docker-1005 [16:01:05] 10Release-Engineering-Team, 10Scap, 10serviceops: Deploy Scap version 4.2.0 - https://phabricator.wikimedia.org/T300058 (10dancy) [16:03:36] (Queue (Jenkins jobs + Zuul functions) alert) firing: (2) Queue (Jenkins jobs + Zuul functions) alert - https://alerts.wikimedia.org [16:08:02] 10Release-Engineering-Team (Yak Shaving 🐃🪒), 10Infrastructure-Foundations, 10Puppet, 10User-brennen: logspam-watch: UTF-8 errors for some input - https://phabricator.wikimedia.org/T292246 (10brennen) 05Open→03Resolved [16:08:36] (Queue (Jenkins jobs + Zuul functions) alert) firing: (3) Queue (Jenkins jobs + Zuul functions) alert - https://alerts.wikimedia.org [16:12:02] (03CR) 10Jbond: [C: 03+1] "policy wise i think this is fine, however im not familiar with gerrit syntax so have added antonie" [puppet] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/756989 (owner: 10Majavah) [16:12:17] 10Release-Engineering-Team (Next), 10Release, 10Train Deployments, 10User-brennen: 1.38.0-wmf.19 deployment blockers - https://phabricator.wikimedia.org/T293960 (10brennen) [16:13:06] 10Release-Engineering-Team (Next), 10Release, 10Train Deployments, 10User-brennen: 1.38.0-wmf.19 deployment blockers - https://phabricator.wikimedia.org/T293960 (10brennen) [16:15:13] 10Release-Engineering-Team (Yak Shaving 🐃🪒), 10User-brennen: A tool for quickly answering what groups an extension is deployed to - https://phabricator.wikimedia.org/T296050 (10brennen) a:05brennen→03None [16:23:36] (Queue (Jenkins jobs + Zuul functions) alert) resolved: (2) Queue (Jenkins jobs + Zuul functions) alert - https://alerts.wikimedia.org [16:25:13] (03CR) 10Jforrester: [C: 03+2] Bundle Math extension with MediaWiki [tools/release] - 10https://gerrit.wikimedia.org/r/715082 (https://phabricator.wikimedia.org/T232948) (owner: 10Jforrester) [16:26:30] (03CR) 10Awight: [C: 03+2] BrowserTests: Rework npm parallel install using ParallelCommand [integration/quibble] - 10https://gerrit.wikimedia.org/r/754866 (owner: 10Kosta Harlan) [16:26:41] (03Merged) 10jenkins-bot: Bundle Math extension with MediaWiki [tools/release] - 10https://gerrit.wikimedia.org/r/715082 (https://phabricator.wikimedia.org/T232948) (owner: 10Jforrester) [16:29:35] (03PS4) 10Jforrester: parameter_functions: Math extension is now tarballed [integration/config] - 10https://gerrit.wikimedia.org/r/715085 (https://phabricator.wikimedia.org/T232948) [16:36:14] (03CR) 10Jforrester: [C: 03+2] parameter_functions: Math extension is now tarballed [integration/config] - 10https://gerrit.wikimedia.org/r/715085 (https://phabricator.wikimedia.org/T232948) (owner: 10Jforrester) [16:38:46] (03Merged) 10jenkins-bot: parameter_functions: Math extension is now tarballed [integration/config] - 10https://gerrit.wikimedia.org/r/715085 (https://phabricator.wikimedia.org/T232948) (owner: 10Jforrester) [16:39:08] !log Zuul: Mark Math extension as now tarballed in parameter_functions for T232948 [16:39:10] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:39:10] T232948: Bundle Math extension with MediaWiki - https://phabricator.wikimedia.org/T232948 [16:39:23] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10MediaWiki-Stakeholders-Group, 10Epic, 10MW-1.38-release: Expand the set of bundled extensions and skins in MediaWiki 1.38 - https://phabricator.wikimedia.org/T290934 (10Jdforrester-WMF) [16:39:29] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10MediaWiki-Stakeholders-Group, 10Epic, 10MW-1.37-release: Expand the set of bundled extensions and skins in MediaWiki 1.37 - https://phabricator.wikimedia.org/T279842 (10Jdforrester-WMF) [16:39:54] (03CR) 10Awight: "I really like where this is going, but don't feel good about the amount of tech debt required for the current implementation. Perhaps we " [integration/quibble] - 10https://gerrit.wikimedia.org/r/742200 (https://phabricator.wikimedia.org/T50217) (owner: 10Kosta Harlan) [16:40:23] (03CR) 10Awight: "recheck" [integration/quibble] - 10https://gerrit.wikimedia.org/r/756543 (owner: 10Kosta Harlan) [16:43:18] (03Merged) 10jenkins-bot: BrowserTests: Rework npm parallel install using ParallelCommand [integration/quibble] - 10https://gerrit.wikimedia.org/r/754866 (owner: 10Kosta Harlan) [16:43:45] (03PS13) 10Awight: Split core npm and composer tests [integration/quibble] - 10https://gerrit.wikimedia.org/r/588087 [16:45:08] (03CR) 10Awight: Split core npm and composer tests (031 comment) [integration/quibble] - 10https://gerrit.wikimedia.org/r/588087 (owner: 10Awight) [16:46:58] (03CR) 10jerkins-bot: [V: 04-1] Split core npm and composer tests [integration/quibble] - 10https://gerrit.wikimedia.org/r/588087 (owner: 10Awight) [16:56:34] (03PS1) 10Ahmon Dancy: Create /srv/patches on deploy server [tools/train-dev] - 10https://gerrit.wikimedia.org/r/757034 [17:02:09] (03CR) 10Ahmon Dancy: [C: 03+2] Create /srv/patches on deploy server [tools/train-dev] - 10https://gerrit.wikimedia.org/r/757034 (owner: 10Ahmon Dancy) [17:03:10] (03Merged) 10jenkins-bot: Create /srv/patches on deploy server [tools/train-dev] - 10https://gerrit.wikimedia.org/r/757034 (owner: 10Ahmon Dancy) [17:13:50] 10Release-Engineering-Team (Next), 10Release, 10Train Deployments, 10User-brennen: 1.38.0-wmf.19 deployment blockers - https://phabricator.wikimedia.org/T293960 (10brennen) [17:50:53] 10Phabricator, 10SecTeam-Processed, 10Security: Audit members of acl*security for more than x duration of no activity (Jan 2022) - https://phabricator.wikimedia.org/T299400 (10Dsharpe) 05Open→03Resolved Because I couldn't think of a better way to do it, I added a personal calendar entry for myself to do... [18:04:23] (03PS1) 10Elukey: Remove ORES from the calendar [tools/release] - 10https://gerrit.wikimedia.org/r/757048 [18:15:46] 10GitLab (Infrastructure), 10serviceops: Migrate gitlab-test instance to puppet - https://phabricator.wikimedia.org/T297411 (10Dzahn) @Jelto Thank you! I deployed your amended change. looked good now. compiled and deployed in production, confirmed noop. Then re-applied the gitlab role on our cloud instance.... [18:21:58] legoktm: the recountCategory.php upgrades you did were merged today; would it be possible to test the script on Beta and see how it does? [18:22:14] (for --mode all) [18:35:58] It's not clear if there is an appropriate ticket, but a job on integration-agent-docker-1008 is failing with `npm ERR! nospc ENOSPC: no space left on device` https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php72-phpunit-standalone-docker/1603/console [18:40:13] hashar: ^ [19:02:25] (03PS14) 10Awight: Split core npm and composer tests [integration/quibble] - 10https://gerrit.wikimedia.org/r/588087 [19:03:07] (03PS15) 10Awight: Split core npm and composer tests [integration/quibble] - 10https://gerrit.wikimedia.org/r/588087 [19:08:28] (03PS9) 10Awight: Clean-up: remove old parallel_run [integration/quibble] - 10https://gerrit.wikimedia.org/r/587890 [19:08:59] (03PS1) 10Majavah: Zuul: [mediawiki/extensions/CentralAuth] Drop UserMerge dependency [integration/config] - 10https://gerrit.wikimedia.org/r/757059 [19:32:02] (03CR) 10Accraze: [C: 03+1] Remove ORES from the calendar [tools/release] - 10https://gerrit.wikimedia.org/r/757048 (owner: 10Elukey) [19:34:11] (03PS1) 10Majavah: updateinterwikicache: Do not commit+sync [tools/scap] - 10https://gerrit.wikimedia.org/r/757062 (https://phabricator.wikimedia.org/T247107) [19:37:22] Thanks Majavah! [19:48:00] 10Deployments, 10Release-Engineering-Team (Radar), 10serviceops, 10Sustainability (Incident Followup), 10User-jijiki: Remove provisioning for 'mwscript', 'foreachwikiindblist' etc from deployment host - https://phabricator.wikimedia.org/T253822 (10dancy) I'm in favor of removing /srv/mediawiki entirely f... [19:52:59] (03PS1) 10Ahmon Dancy: Make sync-world, etc work without /srv/mediawiki on deploy server [tools/scap] - 10https://gerrit.wikimedia.org/r/757070 (https://phabricator.wikimedia.org/T253822) [19:53:17] (03CR) 10Kosta Harlan: [C: 03+2] Split core npm and composer tests [integration/quibble] - 10https://gerrit.wikimedia.org/r/588087 (owner: 10Awight) [19:53:39] (03CR) 10jerkins-bot: [V: 04-1] Make sync-world, etc work without /srv/mediawiki on deploy server [tools/scap] - 10https://gerrit.wikimedia.org/r/757070 (https://phabricator.wikimedia.org/T253822) (owner: 10Ahmon Dancy) [19:55:31] (03PS2) 10Ahmon Dancy: Make sync-world, etc work without /srv/mediawiki on deploy server [tools/scap] - 10https://gerrit.wikimedia.org/r/757070 (https://phabricator.wikimedia.org/T253822) [20:01:34] (03CR) 10Jforrester: [C: 03+2] Zuul: [mediawiki/extensions/CentralAuth] Drop UserMerge dependency [integration/config] - 10https://gerrit.wikimedia.org/r/757059 (owner: 10Majavah) [20:09:47] (03Merged) 10jenkins-bot: Split core npm and composer tests [integration/quibble] - 10https://gerrit.wikimedia.org/r/588087 (owner: 10Awight) [20:14:21] (03CR) 10Jforrester: [C: 03+2] "…" [integration/config] - 10https://gerrit.wikimedia.org/r/757059 (owner: 10Majavah) [20:17:27] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/CentralAuth] Drop UserMerge dependency [integration/config] - 10https://gerrit.wikimedia.org/r/757059 (owner: 10Majavah) [20:17:39] !log Zuul: [mediawiki/extensions/CentralAuth] Drop UserMerge dependency [20:17:40] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:18:24] (03CR) 10Jforrester: [C: 03+1] "Let's ship it!" [integration/config] - 10https://gerrit.wikimedia.org/r/756047 (owner: 10Hashar) [21:24:24] 10GitLab (CI & Job Runners), 10serviceops: upgrade gitlab-runners to bullseye - https://phabricator.wikimedia.org/T297659 (10brennen) [21:25:00] 10GitLab (Infrastructure): Document and test failover for GitLab and GitLab Replica - https://phabricator.wikimedia.org/T296713 (10brennen) [21:25:24] 10GitLab (Infrastructure), 10GitLab-Test, 10Release-Engineering-Team (Radar), 10serviceops-radar, 10Cloud-VPS (Quota-requests): Request increased quota for devtools Cloud VPS project - https://phabricator.wikimedia.org/T299561 (10brennen) [21:25:42] 10GitLab (Project Migration), 10Striker, 10Tools: Replace Diffusion integration with Gitlab integration in Striker (toolsadmin) - https://phabricator.wikimedia.org/T296893 (10brennen) [21:27:06] 10GitLab (Project Migration), 10Release-Engineering-Team: Create new GitLab project group: research - https://phabricator.wikimedia.org/T300074 (10fab) [21:49:10] 10GitLab (Project Migration), 10Release-Engineering-Team (Doing), 10User-brennen, 10Voice & Tone: Rename mainline Git branch from "master" to "main" on all WMF-hosted repositories during GitLab migration - https://phabricator.wikimedia.org/T281593 (10bd808) [21:53:35] dduvall: gentle nudge for your review/feedback on https://gerrit.wikimedia.org/r/c/blubber/+/749569 (feature: build-time arguments for lives & runs user config) [22:02:12] 10GitLab (CI & Job Runners), 10Security Team AppSec, 10Security-Team, 10SecTeam-Processed, 10Security: Create a simple tool to check for bidirectional unicode characters - https://phabricator.wikimedia.org/T295333 (10sbassett) 05Open→03Resolved a:03sbassett Per previous, we'll just use the semgrep... [22:02:24] 10GitLab (CI & Job Runners), 10Security Team AppSec, 10Security-Team, 10Security: Create initial proof of concept application security pipeline repository - https://phabricator.wikimedia.org/T289293 (10sbassett) [22:03:31] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Yak Shaving 🐃🪒), 10Security Team AppSec, 10Security-Team, and 2 others: Confirm with releng which docker images make the most sense to use - https://phabricator.wikimedia.org/T296805 (10sbassett) Ping @thcipriani. Does the strategy I've outlined wit... [22:08:25] 10GitLab (CI & Job Runners), 10Security Team AppSec, 10Security-Team, 10Security: Investigate SAST template options now included with Gitlab CE and formulate use-cases and documentation - https://phabricator.wikimedia.org/T294312 (10sbassett) Initially documented here: https://www.mediawiki.org/wiki/Securi... [22:09:30] > 21:41:38 ArgumentCountError: Too few arguments to function GrowthExperiments\Homepage\SiteNoticeGenerator::__construct(), 2 passed in /workspace/src/extensions/GrowthExperiments/tests/phpunit/unit/SiteNoticeGeneratorTest.php on line 112 and exactly 3 expected [22:09:38] looks like that's failing on a bunch of different repos [22:10:18] hm.. or not [22:10:21] 10GitLab (CI & Job Runners), 10Security Team AppSec, 10Security-Team, 10SecTeam-Processed, 10Security: Create best practices / guidelines documentation for Gitlab application security ci templates - https://phabricator.wikimedia.org/T295374 (10sbassett) In progress here: https://www.mediawiki.org/wiki/Se... [22:39:58] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Yak Shaving 🐃🪒), 10Security Team AppSec, 10Security-Team, and 2 others: Confirm with releng which docker images make the most sense to use - https://phabricator.wikimedia.org/T296805 (10thcipriani) hey @sbassett thanks for the ping, sorry for the str... [23:45:01] 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review, 10Release, 10Train Deployments: 1.38.0-wmf.7 deployment blockers - https://phabricator.wikimedia.org/T293948 (10Stang)