[00:28:58] 10Release-Engineering-Team (Bonus Level 🕹ī¸), 10Patch-For-Review, 10Release, 10Train Deployments: 1.40.0-wmf.1 deployment blockers - https://phabricator.wikimedia.org/T314190 (10Zabe) [05:58:51] 10Phabricator, 10Release-Engineering-Team (Bonus Level 🕹ī¸), 10serviceops, 10serviceops-collab, 10Patch-For-Review: move phabricator to new hardware generation - https://phabricator.wikimedia.org/T280597 (10Marostegui) [05:59:09] 10Phabricator, 10Release-Engineering-Team (Bonus Level 🕹ī¸), 10serviceops, 10serviceops-collab, 10Patch-For-Review: sort out mysql privileges for phab1004/phab2002 - https://phabricator.wikimedia.org/T315713 (10Marostegui) 05Open→03Resolved The problem wasn't the grants from what I can see, as there's... [08:52:48] Project beta-scap-sync-world build #68658: 04FAILURE in 7 min 38 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/68658/ [09:02:30] (03CR) 10Jelto: "looks mostly good, one comment in-line" [integration/config] - 10https://gerrit.wikimedia.org/r/832374 (https://phabricator.wikimedia.org/T308271) (owner: 10Dduvall) [09:11:07] Project beta-scap-sync-world build #68659: 04STILL FAILING in 16 min: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/68659/ [09:30:33] Project beta-scap-sync-world build #68660: 04STILL FAILING in 15 min: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/68660/ [09:41:42] Project beta-scap-sync-world build #68661: 04STILL FAILING in 9 min 18 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/68661/ [09:50:19] Project beta-scap-sync-world build #68662: 04STILL FAILING in 4 min 52 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/68662/ [09:55:50] 10Project-Admins: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706 (10PWaigi-WMF) Hi, I'm a Product Manager; I'd like to create projects, sub-projects and milestones for the Inuka team. [10:01:30] Project beta-scap-sync-world build #68663: 04STILL FAILING in 6 min 43 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/68663/ [10:09:16] Project beta-scap-sync-world build #68664: 04STILL FAILING in 4 min 24 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/68664/ [10:19:02] Project beta-scap-sync-world build #68665: 04STILL FAILING in 4 min 11 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/68665/ [10:29:49] Project beta-scap-sync-world build #68666: 04STILL FAILING in 5 min 1 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/68666/ [10:38:59] Project beta-scap-sync-world build #68667: 04STILL FAILING in 4 min 5 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/68667/ [10:56:57] Project beta-scap-sync-world build #68668: 04STILL FAILING in 12 min: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/68668/ [11:02:33] Project beta-scap-sync-world build #68669: 15ABORTED in 3 min 59 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/68669/ [11:03:09] !log soft reboot deployment-parsoid12, unresponsive [11:03:10] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [11:04:16] 10Beta-Cluster-Infrastructure: beta-scap-sync-world failing - https://phabricator.wikimedia.org/T317759 (10TheresNoTime) p:05Triage→03High Repeatedly failing for the last few hours with ` sudo -u mwdeploy -n -- /usr/bin/scap cdb-rebuild (ran as mwdeploy@deployment-parsoid12.deployment-prep.eqiad1.wikimedia.c... [11:05:52] Project beta-scap-sync-world build #68670: 15ABORTED in 59 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/68670/ [11:10:19] Project beta-scap-sync-world build #68671: 04STILL FAILING in 4 min 21 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/68671/ [11:15:29] 10Beta-Cluster-Infrastructure: beta-scap-sync-world failing - https://phabricator.wikimedia.org/T317759 (10TheresNoTime) Managed to SSH in after a reboot, looks like it's OOMing? Job still failing with the above "not responding" error, and I can't open another SSH session. ` top - 11:13:30 up 8 min, 2 users,... [11:18:22] Yippee, build fixed! [11:18:23] Project beta-scap-sync-world build #68672: 09FIXED in 3 min 29 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/68672/ [11:25:30] "throw more resources at it" might fix T317759, but seeing as it feels like a fairly new problem something must have "gone wrong" lately and I don't want to mask that with just a bigger instance.. :/ [11:25:31] T317759: beta-scap-sync-world failing - https://phabricator.wikimedia.org/T317759 [11:27:14] Project beta-scap-sync-world build #68673: 04FAILURE in 2 min 24 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/68673/ [11:29:32] 10Beta-Cluster-Infrastructure: beta-scap-sync-world failing - https://phabricator.wikimedia.org/T317759 (10TheresNoTime) >>! In T317759#8238922, @TheresNoTime wrote: > Managed to SSH in after a reboot, looks like it's OOMing? `rsync: fork failed in do_recv: Cannot allocate memory (12)` — so that's a yes..? Sho... [11:31:39] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Radar): beta-scap-sync-world failing - https://phabricator.wikimedia.org/T317759 (10TheresNoTime) [11:38:26] Yippee, build fixed! [11:38:26] Project beta-scap-sync-world build #68674: 09FIXED in 3 min 33 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/68674/ [11:46:46] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Radar), 10Parsoid: beta-scap-sync-world failing - https://phabricator.wikimedia.org/T317759 (10hashar) What is possible is the process uses half of the available memory and when invoking `fork()` the Kernel reject its. I have encountered that ages ag... [11:47:04] TheresNoTime: looks like some parsoid process ends up at 100% CPU with lot of IO [11:47:13] well I assume it is parsoid, that could be anything else really [11:49:13] Thanks for the comments hashar :) I don't know enough about parsoid to effectively figure out the root cause, but clearly something is "broke" [11:51:52] maybe there are some logs somewhere that could lead to the process [11:51:53] ah yeah [11:51:57] so I spotted that back in June [11:52:03] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Radar), 10Parsoid: beta-scap-sync-world failing - https://phabricator.wikimedia.org/T317759 (10hashar) From Logstash it looks like there are a few process parsing a large json file https://en.wikipedia.beta.wmflabs.org/wiki/Data:DutchMuni-json Which... [11:52:04] That iowait is o_o [11:52:07] it is https://phabricator.wikimedia.org/T310069 [11:52:24] which is the same symptoms [11:52:29] which leads to https://phabricator.wikimedia.org/T288889 [11:52:48] that is why I cookie licked you task TheresNoTime , the error sounded familiar :] [11:54:18] I am guessing we can empty up that article with a comment linking to the few tasks above [11:54:32] it got created in 2017 so maybe it is no more used anywhere [11:56:14] 100% empty that file, is that really "all" that's breaking things? o.O [11:59:39] * TheresNoTime blanked it [12:00:41] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Radar), 10Parsoid: beta-scap-sync-world failing - https://phabricator.wikimedia.org/T317759 (10TheresNoTime) >>! In T317759#8238995, @hashar wrote: > From Logstash it looks like there are a few process parsing a large json file https://en.wikipedia.b... [12:01:34] 10Phabricator: New Herald rule: add `design-systems-team` tag to any new tasks tagged with `codex` - https://phabricator.wikimedia.org/T317801 (10Aklapper) 05Open→03Resolved a:03Aklapper Created H408: * When all of these conditions are met: ** Project tags include any of #Codex * Take these actions the fir... [12:01:47] * TheresNoTime needs to add a new sort of check to https://www.isbetabroken.com ... [12:03:56] TheresNoTime: just have it return `

yes!

` [12:05:09] I was so *so* tempted :p [12:07:03] :-] [12:33:07] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Radar), 10Parsoid: beta-scap-sync-world failing - https://phabricator.wikimedia.org/T317759 (10TheresNoTime) p:05Unbreak!→03Triage Dropping from //UBN!//, immediate issue is resolved (though there should probably be a follow-up to figure out why... [13:07:20] 10Continuous-Integration-Infrastructure, 10User-zeljkofilipin: selenium-daily Jenkins jobs fail sometimes with: "Failed to read test report file" - https://phabricator.wikimedia.org/T317878 (10zeljkofilipin) [13:11:36] 10Continuous-Integration-Infrastructure, 10User-zeljkofilipin: selenium-daily Jenkins jobs fail sometimes with: "Failed to read test report file" - https://phabricator.wikimedia.org/T317878 (10zeljkofilipin) [13:11:53] 10Continuous-Integration-Infrastructure, 10User-zeljkofilipin: selenium-daily Jenkins jobs fail sometimes with: "Failed to read test report file" - https://phabricator.wikimedia.org/T317878 (10zeljkofilipin) [13:17:37] 10Continuous-Integration-Infrastructure, 10User-zeljkofilipin: selenium-daily Jenkins jobs fail sometimes with: "Failed to read test report file" - https://phabricator.wikimedia.org/T317878 (10zeljkofilipin) [13:26:59] 10Continuous-Integration-Infrastructure, 10MediaWiki-Core-Tests, 10Browser-Tests, 10User-zeljkofilipin: ffmpeg stderr Capture area 1920x1080 at position 0.0 outside the screen size 1280x1024 - https://phabricator.wikimedia.org/T317879 (10zeljkofilipin) [13:30:01] 10Continuous-Integration-Infrastructure, 10MediaWiki-Core-Tests, 10Browser-Tests, 10User-zeljkofilipin: ffmpeg stderr Capture area 1920x1080 at position 0.0 outside the screen size 1280x1024 - https://phabricator.wikimedia.org/T317879 (10zeljkofilipin) [13:41:08] 10Continuous-Integration-Infrastructure, 10MediaWiki-Core-Tests, 10Browser-Tests, 10User-zeljkofilipin: ffmpeg stderr Capture area 1920x1080 at position 0.0 outside the screen size 1280x1024 - https://phabricator.wikimedia.org/T317879 (10zeljkofilipin) [13:44:14] 10Continuous-Integration-Infrastructure, 10MediaWiki-Core-Tests, 10Browser-Tests, 10User-zeljkofilipin: ffmpeg stderr Capture area 1920x1080 at position 0.0 outside the screen size 1280x1024 - https://phabricator.wikimedia.org/T317879 (10zeljkofilipin) [13:47:16] 10Continuous-Integration-Infrastructure, 10MediaWiki-Core-Tests, 10Browser-Tests, 10User-zeljkofilipin: ffmpeg stderr Capture area 1920x1080 at position 0.0 outside the screen size 1280x1024 - https://phabricator.wikimedia.org/T317879 (10zeljkofilipin) [14:04:31] 10Phabricator: New Herald rule: add `design-systems-team` tag to any new tasks tagged with `codex` - https://phabricator.wikimedia.org/T317801 (10ldelench_wmf) Awesome, thank you! [14:30:34] 10Release-Engineering-Team (Bonus Level 🕹ī¸), 10Patch-For-Review, 10Release, 10Train Deployments: 1.40.0-wmf.1 deployment blockers - https://phabricator.wikimedia.org/T314190 (10Jdlrobson) [14:43:45] 10Phabricator, 10Release-Engineering-Team (Bonus Level 🕹ī¸), 10serviceops, 10serviceops-collab, 10Patch-For-Review: sort out mysql privileges for phab1004/phab2002 - https://phabricator.wikimedia.org/T315713 (10Dzahn) Oh, I assumed that part (that it's not using a dbproxy in this case) was meant to be. T... [15:44:37] (03PS2) 10Dduvall: dockerfiles: Create and chown home directory for buildctl user [integration/config] - 10https://gerrit.wikimedia.org/r/832374 (https://phabricator.wikimedia.org/T308271) [15:45:03] (03CR) 10Dduvall: dockerfiles: Create and chown home directory for buildctl user (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/832374 (https://phabricator.wikimedia.org/T308271) (owner: 10Dduvall) [16:17:53] (03PS1) 10Hashar: Use gerrit-deploy for deployment on devtools [software/gerrit] (deploy/wmf/stable-3.4) - 10https://gerrit.wikimedia.org/r/832518 (https://phabricator.wikimedia.org/T317412) [16:26:38] 10Gerrit, 10Release-Engineering-Team (Priority Backlog đŸ“Ĩ), 10Scap, 10Patch-For-Review: Automate Gerrit deployment steps - https://phabricator.wikimedia.org/T317412 (10hashar) a:03hashar I have changed the deployment user on devtools project from `gerrit2` to `gerrit-deployer` and we will need a bunch of... [16:33:58] What are the main differences with quibble-vendor-mysql-php62-phpunit-standalone-docker from the other test runners? CirrusSearch started failing some tests in only that runner yesterday due to differences in serialization precision. I have a fix up, but trying to understand why it only effects that runner and why it doesn't seem to effect other repos that run Cirrus tests. [16:34:06] s/62/72/ [16:37:46] (03CR) 10Ahmon Dancy: Replace Lock with TimeoutLock (037 comments) [tools/scap] - 10https://gerrit.wikimedia.org/r/828075 (https://phabricator.wikimedia.org/T315531) (owner: 10Jeena Huneidi) [16:50:36] 10Release-Engineering-Team (Bonus Level 🕹ī¸), 10Patch-For-Review, 10Release, 10Train Deployments: 1.40.0-wmf.1 deployment blockers - https://phabricator.wikimedia.org/T314190 (10dancy) @Jdlrobson Acknowledging the new train blocker. I will wait to hear from you before advancing the train today. [16:54:07] (03CR) 10Ahmon Dancy: dockerfiles: Create and chown home directory for buildctl user (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/832374 (https://phabricator.wikimedia.org/T308271) (owner: 10Dduvall) [16:55:47] (03CR) 10Dduvall: dockerfiles: Create and chown home directory for buildctl user (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/832374 (https://phabricator.wikimedia.org/T308271) (owner: 10Dduvall) [16:57:58] (03PS3) 10Dduvall: dockerfiles: Create and chown home directory for buildctl user [integration/config] - 10https://gerrit.wikimedia.org/r/832374 (https://phabricator.wikimedia.org/T308271) [16:58:09] (03CR) 10Dduvall: dockerfiles: Create and chown home directory for buildctl user (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/832374 (https://phabricator.wikimedia.org/T308271) (owner: 10Dduvall) [16:59:32] (03CR) 10Ahmon Dancy: dockerfiles: Create and chown home directory for buildctl user (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/832374 (https://phabricator.wikimedia.org/T308271) (owner: 10Dduvall) [16:59:34] (03PS4) 10Dduvall: dockerfiles: Create the buildctl user home directory [integration/config] - 10https://gerrit.wikimedia.org/r/832374 (https://phabricator.wikimedia.org/T308271) [17:00:02] (03PS5) 10Dduvall: dockerfiles: Create the buildctl user home directory [integration/config] - 10https://gerrit.wikimedia.org/r/832374 (https://phabricator.wikimedia.org/T308271) [17:00:10] (03CR) 10Dduvall: dockerfiles: Create the buildctl user home directory (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/832374 (https://phabricator.wikimedia.org/T308271) (owner: 10Dduvall) [17:00:26] (03CR) 10Ahmon Dancy: [C: 03+2] "Looks legit" [integration/config] - 10https://gerrit.wikimedia.org/r/832374 (https://phabricator.wikimedia.org/T308271) (owner: 10Dduvall) [17:02:20] (03Merged) 10jenkins-bot: dockerfiles: Create the buildctl user home directory [integration/config] - 10https://gerrit.wikimedia.org/r/832374 (https://phabricator.wikimedia.org/T308271) (owner: 10Dduvall) [17:04:26] 10Release-Engineering-Team (Bonus Level 🕹ī¸), 10Patch-For-Review, 10Release, 10Train Deployments: 1.40.0-wmf.1 deployment blockers - https://phabricator.wikimedia.org/T314190 (10cscott) There's a patch for T317857 which has been merged onto master. We just need to cherry-pick it to the train branch before... [17:10:05] dancy: thanks for the review! will you build/publish the buildctl image or shall i? [17:11:24] Leaving it to you. [17:22:44] 10Release-Engineering-Team (Bonus Level 🕹ī¸), 10Patch-For-Review, 10Release, 10Train Deployments: 1.40.0-wmf.1 deployment blockers - https://phabricator.wikimedia.org/T314190 (10Jdlrobson) @dancy I can be available to test https://gerrit.wikimedia.org/r/832547. Ping me on IRC when you are doing this. [17:23:41] dancy: 👍đŸŊ [17:24:18] !log Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/832374 [17:24:19] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:49:49] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Priority Backlog đŸ“Ĩ): Remove use custom network for GitLab runners and buildkitd in favor of a fixed IP - https://phabricator.wikimedia.org/T317904 (10dduvall) [17:50:49] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Priority Backlog đŸ“Ĩ): Remove use custom network for GitLab runners and buildkitd in favor of a fixed IP - https://phabricator.wikimedia.org/T317904 (10dduvall) [18:28:27] hi relengers! Could someone please publish the latest fundraising images from dev-images to the docker repository? [18:28:55] I think brennen was going to do it the other day, but I don't see them at https://docker-registry.wikimedia.org/v2/_catalog [18:29:39] specifically we need fundraising-mediawiki-bullseye-php74-apache2:1.0.0 [18:30:36] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Priority Backlog đŸ“Ĩ): Remove use custom network for GitLab runners and buildkitd in favor of a fixed IP - https://phabricator.wikimedia.org/T317904 (10dduvall) Well, I seem to be running into a catch 22 here: Fixed IPs are only supported when using a cust... [18:31:40] ooh, is all this buildctl stuff intended to make the repo auto-publish? [18:42:07] ejegg: I can take a stab at the fundraising image issue [18:45:42] ejegg: `docker pull docker-registry.wikimedia.org/releng/fundraising-mediawiki-bullseye-php74-apache2:1.0.0` works for me. [18:47:17] https://docker-registry.wikimedia.org/releng/fundraising-mediawiki-bullseye-php74-apache2/tags/ [18:50:03] Regarding buildctl, we're experimenting with using buildkitd/buildctl for building container images in Gitlab CI [18:55:12] 10Release-Engineering-Team (Bonus Level 🕹ī¸), 10Patch-For-Review, 10Release, 10Train Deployments: 1.40.0-wmf.1 deployment blockers - https://phabricator.wikimedia.org/T314190 (10cscott) [19:03:28] dancy: ohhh, we had been looking for it under https://docker-registry.wikimedia.org/ *dev* /fundraising-mediawiki-bullseye-php74-apache2:1.0.0 [19:04:53] would it be possible to publish it under the /dev/ namespace where we had our older dev-images? [19:05:11] or should everything be under /releng/ from now on? [19:07:05] (03PS9) 10Jeena Huneidi: Replace Lock with TimeoutLock [tools/scap] - 10https://gerrit.wikimedia.org/r/828075 (https://phabricator.wikimedia.org/T315531) [19:08:21] (03CR) 10Jeena Huneidi: "fixed some other bugs" [tools/scap] - 10https://gerrit.wikimedia.org/r/828075 (https://phabricator.wikimedia.org/T315531) (owner: 10Jeena Huneidi) [19:28:06] dancy or brennen, would it be possible to move (or republish) the fundraising*bullseye* images under docker-registry.wikimedia.org/dev rather than docker-registry.wikimedia.org/releng ? [19:34:28] we could move it (/dev is a different repo than /releng although I'd prefer not to move it), but what's the problem you're having? [19:36:20] (03PS10) 10Jeena Huneidi: Replace Lock with TimeoutLock [tools/scap] - 10https://gerrit.wikimedia.org/r/828075 (https://phabricator.wikimedia.org/T315531) [19:40:34] ejegg: oh, I see what you're talking about now! [19:40:46] it's normally in the /dev namespace [19:50:05] ejegg: I found the problem! [19:56:44] !log Updating development images on contint primary [19:56:45] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:33:41] thanks, thcipriani ! [20:39:55] (03CR) 10Ahmon Dancy: Replace Lock with TimeoutLock (033 comments) [tools/scap] - 10https://gerrit.wikimedia.org/r/828075 (https://phabricator.wikimedia.org/T315531) (owner: 10Jeena Huneidi) [20:55:57] 10Continuous-Integration-Infrastructure, 10MediaWiki-Core-Tests, 10Browser-Tests, 10User-zeljkofilipin: ffmpeg stderr Capture area 1920x1080 at position 0.0 outside the screen size 1280x1024 - https://phabricator.wikimedia.org/T317879 (10hashar) Well done on spotting the root cause ( https://gerrit.wikimed... [20:59:32] 10Continuous-Integration-Infrastructure, 10User-zeljkofilipin: selenium-daily Jenkins jobs fail sometimes with: "Failed to read test report file" - https://phabricator.wikimedia.org/T317878 (10hashar) [21:00:42] 10Continuous-Integration-Infrastructure, 10User-zeljkofilipin: selenium-daily Jenkins jobs fail sometimes with: "Failed to read test report file" - https://phabricator.wikimedia.org/T317878 (10hashar) The XML has: ` \t\t\t\treturn false;\n\t\t\t}\n\n\t\t\tconst api = new mw.Api();\n\n\t\t\tapi.ge... [21:01:07] 10Continuous-Integration-Infrastructure, 10User-zeljkofilipin: selenium-daily Jenkins jobs fail sometimes with: "Failed to read test report file" - https://phabricator.wikimedia.org/T317878 (10hashar) [21:08:43] > Successfully published image docker-registry.discovery.wmnet/dev/fundraising-mediawiki-bullseye-php74-apache2:1.0.0 [21:08:49] ^ ejegg [21:09:20] (which means it should be at docker-registry.wikimedia.org/dev/fundraising-mediawiki-bullseye-php74-apache2:1.0.0) [21:15:16] (03CR) 10Ahmon Dancy: Replace Lock with TimeoutLock (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/828075 (https://phabricator.wikimedia.org/T315531) (owner: 10Jeena Huneidi) [21:21:02] Hi all, do you have any idea why zuul do not have this https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/831618 running? Thanks~ [21:22:18] wfan: in the error log at https://integration.wikimedia.org/ci/job/wikimedia-fundraising-civicrm-docker/8462/console I see there are things related to "mysql failed to start" [21:22:51] maybe this is related: 19:14:08 ERROR 1269 (HY000) at line 1: Can't revoke all privileges for one or more of the requested users [21:24:14] yea, doing a "recheck" first is a good idea! [21:24:27] (03PS1) 10Ahmon Dancy: scap backport: Allow URLs to have a trailing slash [tools/scap] - 10https://gerrit.wikimedia.org/r/832574 [21:25:55] (03PS2) 10Ahmon Dancy: scap backport: Allow URLs to have a trailing slash [tools/scap] - 10https://gerrit.wikimedia.org/r/832574 [21:28:14] mutante : thanks while do a recheck is not working, and the "mysql failed to start" error was there before, which did not block zuul, maybe not related? [21:29:53] wfan: I see. Oh, yea, it's like it stopped responding or we are still waiting for the results after PS5, I am afraid I dont know much more [21:37:15] I don't see it on https://phabricator.wikimedia.org/T317637#8240882 [21:37:19] Nope [21:37:26] https://integration.wikimedia.org/zuul/ [21:37:34] Stupid clipboard [21:44:56] new fave unit test failure [21:45:04] Failed asserting that 'zsHLlLHdJMjmck4saLSdmHbBIqbhDPV3AExAvf0TY9A=.PpJlVn8YVlTqpEmz6XCKXA==.N+/GlQ==' is not equal to true. [21:45:35] At first glance you are like, wtf, why would that be equal to true. At second glance you are like, well of course its truthy, how come this test doesn't fail all the time [21:47:09] I like the word "truthy". it's actually in enwikt https://en.wiktionary.org/wiki/truthy [21:48:44] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Priority Backlog đŸ“Ĩ): Remove use custom network for GitLab runners and buildkitd in favor of a fixed IP - https://phabricator.wikimedia.org/T317904 (10dduvall) [21:49:23] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Priority Backlog đŸ“Ĩ): Explicitly config buildkitd with internal DNS nameserver - https://phabricator.wikimedia.org/T317904 (10dduvall) [21:50:17] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Priority Backlog đŸ“Ĩ): Explicitly config buildkitd with internal DNS nameserver - https://phabricator.wikimedia.org/T317904 (10dduvall) I've updated the task as abandoning the custom network no longer seems feasible, and there appears to be a way to config... [21:50:40] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Priority Backlog đŸ“Ĩ): Explicitly config buildkitd with internal DNS nameserver - https://phabricator.wikimedia.org/T317904 (10dduvall) p:05Triage→03Medium [22:15:27] wfan: given that CI has somehow completely stopped on your repo (the -1 on it is 4 PSes ago) it would be best if you can make a phab ticket if possible [22:16:16] mutante: ok, thanks for your help :) [22:19:18] what tag is appropriate for this ticket? [22:20:29] wfan: try Continuous-Integration-Infrastructure and release-engineering-team. people will fix it if needed [22:25:54] Hash.ar will likely be the one in the eu morning [22:28:07] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Blocking 🧱): CI has somehow completely stopped for a pr at wikimedia/fundraising/crm master - https://phabricator.wikimedia.org/T317928 (10AnnWF) [22:28:29] Thanks again~ mutante: [22:29:08] no problem [22:36:38] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Blocking 🧱): CI has somehow completely stopped for a pr at wikimedia/fundraising/crm master - https://phabricator.wikimedia.org/T317928 (10dancy) The error in https://integration.wikimedia.org/ci/job/wikimedia-fundraising-civicrm-docker/8462/... [22:36:40] 10Release-Engineering-Team (Blocking 🧱), 10Data Pipelines (Sprint 01): Increase maximum artifacts size on Gitlab Registry to accommodate files >1GB - https://phabricator.wikimedia.org/T317555 (10thcipriani) >>! In T317555#8237522, @Antoine_Quhen wrote: > Sorry, your change didn't make it work. > > But I can n... [22:42:50] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Blocking 🧱), 10Wikimedia-Fundraising-CiviCRM: CI has somehow completely stopped for a pr at wikimedia/fundraising/crm master - https://phabricator.wikimedia.org/T317928 (10dancy) [22:44:49] Project beta-scap-sync-world build #68739: 04FAILURE in 9 min 50 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/68739/ [22:50:39] ^ Looking into that. [22:51:13] deployment-mwmaint02's root filesystem is full. [22:51:16] Project beta-scap-sync-world build #68740: 04STILL FAILING in 2 min 7 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/68740/ [22:56:37] Project beta-scap-sync-world build #68741: 04STILL FAILING in 1 min 45 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/68741/ [23:01:52] 10Beta-Cluster-Infrastructure: deployment-mwmaint02.deployment-prep.eqiad1.wikimedia.cloud filesystem is full - https://phabricator.wikimedia.org/T317931 (10dancy) [23:03:40] 10Beta-Cluster-Infrastructure: deployment-mwmaint02.deployment-prep.eqiad1.wikimedia.cloud filesystem is full - https://phabricator.wikimedia.org/T317931 (10dancy) I deleted `/srv/mediawiki/php-master/cache/l10n/upstream/.~tmp~`. ` root@deployment-mwmaint02:~# df -h / Filesystem Size Used Avail Use% Mou... [23:07:23] Project beta-scap-sync-world build #68742: 04STILL FAILING in 1 min 48 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/68742/ [23:19:07] 10Beta-Cluster-Infrastructure: deployment-mwmaint02.deployment-prep.eqiad1.wikimedia.cloud filesystem is full - https://phabricator.wikimedia.org/T317931 (10dancy) That wasn't sufficient so next I deleted archived log files in /var/log/mediawiki that are older than 5 days. ` root@deployment-mwmaint02:/var/log/me... [23:19:43] Yippee, build fixed! [23:19:43] Project beta-scap-sync-world build #68743: 09FIXED in 4 min 45 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/68743/