[00:30:25] hasharAway: :-) [00:41:54] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen), 10observability, 06SRE, 13Patch-For-Review: Export zuul metrics to Prometheus - https://phabricator.wikimedia.org/T233089#10414621 (10colewhite) 05In progress→03Resolved Zuul is effectively migrated at this point and the... [08:06:20] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen), 10observability, 06SRE, 13Patch-For-Review: Export zuul metrics to Prometheus - https://phabricator.wikimedia.org/T233089#10414905 (10Volans) FYI the links at the bottom of https://integration.wikimedia.org/zuul/ ( Job Stats... [09:15:05] 10GitLab: GitLab Private Repository Request for: Trust and Safety Product team - https://phabricator.wikimedia.org/T382405#10414990 (10kostajh) It would be nice if TSP engineers could have access to create public repos under `repos/trust-and-safety-product`, is that possible to set up? [11:18:55] !log lucaswerkmeister-wmde@deployment-deploy04:~$ mwscript createAndPromote metawiki 'Lucas Werkmeister (WMDE)' --force --bureaucrat # testing T244019 [11:18:57] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [11:18:58] T244019: Inconsistent user permissions for users who were recently added to a new group - https://phabricator.wikimedia.org/T244019 [11:37:43] (03PS1) 10Máté Szabó: zuul: Test ReportIncident with CommunityConfiguration [integration/config] - 10https://gerrit.wikimedia.org/r/1105679 (https://phabricator.wikimedia.org/T374113) [12:31:05] 06Release-Engineering-Team, 06Data Products, 06Data-Platform-SRE, 10Dumps-Generation, and 4 others: WE 5.4 KR - Hypothesis 5.4.4 - Q3 FY24/55 - Migrate current-generation dumps to run on kubernetes - https://phabricator.wikimedia.org/T352650#10415363 (10BTullis) [13:15:27] 06Release-Engineering-Team, 06Data Products, 06Data-Platform-SRE, 10Dumps-Generation, and 4 others: WE 5.4 KR - Hypothesis 5.4.4 - Q3 FY24/55 - Migrate current-generation dumps to run on kubernetes - https://phabricator.wikimedia.org/T352650#10415473 (10BTullis) [13:21:37] 06Release-Engineering-Team, 06Data Products, 06Data-Platform-SRE, 10Dumps-Generation, and 4 others: WE 5.4 KR - Hypothesis 5.4.4 - Q3 FY24/55 - Migrate current-generation dumps to run on kubernetes - https://phabricator.wikimedia.org/T352650#10415499 (10BTullis) [13:29:53] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen), 06cloud-services-team, 10Cloud-VPS, 10ci-test-error (WMF-deployed Build Failure): Various CI jobs running in the integration Cloud VPS project failing due to transient DNS lookup... - https://phabricator.wikimedia.org/T374830#10415523 [13:39:42] (03CR) 10Kosta Harlan: [C:03+1] zuul: Test ReportIncident with CommunityConfiguration [integration/config] - 10https://gerrit.wikimedia.org/r/1105679 (https://phabricator.wikimedia.org/T374113) (owner: 10Máté Szabó) [13:56:54] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen), 06cloud-services-team, 10Cloud-VPS, 10ci-test-error (WMF-deployed Build Failure): Various CI jobs running in the integration Cloud VPS project failing due to transient DNS lookup... - https://phabricator.wikimedia.org/T374830#10415630 [14:06:40] (03PS1) 10Hslater: Use bluespice template for mediawiki/extensions/NotifyMe [integration/config] - 10https://gerrit.wikimedia.org/r/1105715 [14:13:02] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen), 06cloud-services-team, 10Cloud-VPS, 10ci-test-error (WMF-deployed Build Failure): Various CI jobs running in the integration Cloud VPS project failing due to transient DNS lookup... - https://phabricator.wikimedia.org/T374830#10415684 [14:58:34] (03CR) 10Harroyo-wmf: [C:03+1] zuul: Test ReportIncident with CommunityConfiguration [integration/config] - 10https://gerrit.wikimedia.org/r/1105679 (https://phabricator.wikimedia.org/T374113) (owner: 10Máté Szabó) [15:48:43] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen), 10observability, 06SRE, 13Patch-For-Review: Export zuul metrics to Prometheus - https://phabricator.wikimedia.org/T233089#10416007 (10colewhite) 05Resolved→03Open [16:10:06] 10Fresh, 07ARM support, 07Upstream: ECONNREFUSED error when running Selenium tests on M1 Mac - https://phabricator.wikimedia.org/T308889#10416188 (10zeljkofilipin) [16:51:51] TFW you are wishing for easy to digest test failure reports in Jenkins and then you find out that a lot of work was done to get rid of the data files that would have allowed them. :/ (T256402) [16:51:52] T256402: Remove JUnit artefacts from Quibble jobs - https://phabricator.wikimedia.org/T256402 [16:53:12] Scanning a couple of megabytes of log output to try and find the error message is not my idea of a good time. [16:53:25] (03PS3) 10Jforrester: jjb: Switch codehealth jobs to sonar-scanner images with Node 20 [integration/config] - 10https://gerrit.wikimedia.org/r/1104711 [16:53:34] (03CR) 10Jforrester: [C:03+2] jjb: Switch codehealth jobs to sonar-scanner images with Node 20 [integration/config] - 10https://gerrit.wikimedia.org/r/1104711 (owner: 10Jforrester) [16:54:58] (03Merged) 10jenkins-bot: jjb: Switch codehealth jobs to sonar-scanner images with Node 20 [integration/config] - 10https://gerrit.wikimedia.org/r/1104711 (owner: 10Jforrester) [16:55:01] bd808: I also miss that file, I was talking with dduvall yesterday about the possibility of checking how many times we run a test for a particular piece of code by having a tool parse that file and dump it somewhere we could query. [17:01:08] <_joe_> hey releng [17:01:21] <_joe_> CI for mediawiki extensions seems to be broken [17:01:34] <_joe_> we have a production fix that is a one liner [17:01:36] <_joe_> https://gerrit.wikimedia.org/r/c/mediawiki/extensions/EventBus/+/1105757 [17:01:42] <_joe_> is it ok if I V+2 it? [17:01:56] <_joe_> the error is with *npm* not being able to download something [17:02:17] <_joe_> so clearly not related to the patch, which also doesn't affect the website being on a maintenance script [17:02:21] <_joe_> thcipriani: ^^ [17:02:34] _joe_: yeah I can't imagine the selenium tests are relevant [17:04:51] _joe_: (in a meeting) let's see if the recheck works unless it's something actively causing user problems, V+2 will confuse zuul and it'll take a while flailing to recover cc brennen dancy (relengers I've seen today) [17:05:18] <_joe_> AIUI it already failed [17:05:54] Looks like a recheck is in progress now. [17:06:55] The previous error was a DNS resolution problem with "download.cypress.io". [17:07:08] presumably if we need to V+2 this, the backports will require the same unless the underlying issue is fixed, correct? [17:07:50] the underlying issue is flakey DNS most likely [17:08:24] in general, we'd try to fix the issue and not V+2 anything that's not production critical because zuul's speculative future state has to do a lot of thrashing to recover otherwise [17:08:25] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen), 06cloud-services-team, 10Cloud-VPS, 10ci-test-error (WMF-deployed Build Failure): Various CI jobs running in the integration Cloud VPS project failing due to transient DNS lookup... - https://phabricator.wikimedia.org/T374830#10416581 [17:08:51] then go back and C+2 everything again (or recheck) [17:09:06] Tests passed this time [17:09:16] No need for hacks. [17:09:33] \o/ [17:09:39] T374830 is the current tracking task for DNS poops itself [17:09:39] T374830: Various CI jobs running in the integration Cloud VPS project failing due to transient DNS lookup failures, often for our own hosts such as gerrit.wikimedia.org - https://phabricator.wikimedia.org/T374830 [17:12:03] bd808: thanks for the pointer [17:12:04] <_joe_> sorry, the gerrit UI confused me [17:12:15] <_joe_> and yes, thanks for the assistance [17:23:33] (03PS2) 10Hslater: Use bluespice template for extensions NotifyMe & CognitiveProcessDesigner [integration/config] - 10https://gerrit.wikimedia.org/r/1105715 [17:29:36] 10Release-Engineering-Team (Doing 😎), 07OKR-Work: [WE6.2.6] Create design document for Group -1 deployment - https://phabricator.wikimedia.org/T379683#10416682 (10zeljkofilipin) [17:51:44] (03PS1) 10Cwhite: update dashboard links to new panels [integration/docroot] - 10https://gerrit.wikimedia.org/r/1105779 (https://phabricator.wikimedia.org/T233089) [17:56:56] (03CR) 10Jforrester: [C:03+1] "LGTM. Want me to deploy it?" [integration/docroot] - 10https://gerrit.wikimedia.org/r/1105779 (https://phabricator.wikimedia.org/T233089) (owner: 10Cwhite) [17:59:16] (03PS1) 10Jforrester: build: Updating mediawiki/mediawiki-phan-config to 0.15.0 [integration/docroot] - 10https://gerrit.wikimedia.org/r/1105785 [18:21:13] (03CR) 10Cwhite: "That would be awesome, thank you!" [integration/docroot] - 10https://gerrit.wikimedia.org/r/1105779 (https://phabricator.wikimedia.org/T233089) (owner: 10Cwhite) [18:25:03] maintenance-disconnect-full-disks build 659589 integration-agent-docker-1054 (/: 28%, /srv: 99%, /var/lib/docker: 37%): OFFLINE due to disk space [18:30:02] maintenance-disconnect-full-disks build 659590 integration-agent-docker-1054 (/: 28%, /srv: 42%, /var/lib/docker: 35%): RECOVERY disk space OK [19:08:16] (03open) 10toyofuku: Add Web Team Deploy window [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/140 (https://phabricator.wikimedia.org/T381541) [19:09:30] (03update) 10toyofuku: Add Web Team Deploy window [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/140 (https://phabricator.wikimedia.org/T381541) [19:14:39] (03merge) 10thcipriani: Add Web Team Deploy window [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/140 (https://phabricator.wikimedia.org/T381541) (owner: 10toyofuku) [19:49:14] (03CR) 10Jforrester: [C:03+2] "I'll land it now, as I'm landing other things anyway." [integration/docroot] - 10https://gerrit.wikimedia.org/r/1099256 (owner: 10Libraryupgrader) [19:49:16] (03CR) 10Jforrester: [C:03+2] build: Updating mediawiki/mediawiki-phan-config to 0.15.0 [integration/docroot] - 10https://gerrit.wikimedia.org/r/1105785 (owner: 10Jforrester) [19:49:19] (03CR) 10Jforrester: [C:03+2] update dashboard links to new panels [integration/docroot] - 10https://gerrit.wikimedia.org/r/1105779 (https://phabricator.wikimedia.org/T233089) (owner: 10Cwhite) [19:49:54] (03Merged) 10jenkins-bot: build: Updating mediawiki/mediawiki-phan-config to 0.15.0 [integration/docroot] - 10https://gerrit.wikimedia.org/r/1105785 (owner: 10Jforrester) [19:50:08] (03Merged) 10jenkins-bot: update dashboard links to new panels [integration/docroot] - 10https://gerrit.wikimedia.org/r/1105779 (https://phabricator.wikimedia.org/T233089) (owner: 10Cwhite) [19:52:58] (03CR) 10Jforrester: [C:03+2] "Deployed; caching may take a little while." [integration/docroot] - 10https://gerrit.wikimedia.org/r/1105779 (https://phabricator.wikimedia.org/T233089) (owner: 10Cwhite) [20:21:27] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen), 10observability, 06SRE, 13Patch-For-Review: Export zuul metrics to Prometheus - https://phabricator.wikimedia.org/T233089#10417126 (10colewhite) 05Open→03Resolved Thanks @Volans for pointing those out! With the latest de... [22:16:20] (03update) 10thcipriani: branch.py: ensure we're tracking skins/* changes [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/139 [22:16:24] (03update) 10thcipriani: branch.py: ensure we're tracking skins/* changes [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/139 [22:17:19] (03update) 10thcipriani: branch.py: ensure we're tracking skins/* changes [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/139 [22:17:57] (03update) 10thcipriani: branch.py: ensure we're tracking skins/* changes [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/139 [22:26:15] 10Release-Engineering-Team (Doing 😎), 05Release, 05Train Deployments: 1.44.0-wmf.8 deployment blockers - https://phabricator.wikimedia.org/T375667#10417496 (10thcipriani) 05Open→03Resolved Last train 2024 is now complete! [22:28:45] 10Release-Engineering-Team (Doing 😎), 05Release, 05Train Deployments: 1.44.0-wmf.11 deployment blockers - https://phabricator.wikimedia.org/T382362#10417521 (10thcipriani) p:05Triage→03Medium a:03dduvall [22:29:11] 10Continuous-Integration-Config: BotPassword file for FLOSSbot - https://phabricator.wikimedia.org/T145331#10417530 (10Pppery)