[01:07:38] (03PS4) 10Catrope: Deploy Codex docs to doc.wikimedia.org [integration/config] - 10https://gerrit.wikimedia.org/r/736101 (https://phabricator.wikimedia.org/T293704) [01:07:55] (03PS5) 10Catrope: Deploy Codex docs to doc.wikimedia.org [integration/config] - 10https://gerrit.wikimedia.org/r/736101 (https://phabricator.wikimedia.org/T293704) [01:09:07] (03CR) 10Catrope: Deploy Codex docs to doc.wikimedia.org (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/736101 (https://phabricator.wikimedia.org/T293704) (owner: 10Catrope) [09:36:17] (03CR) 10Hashar: [C: 03+2] "Great thank you Roan for going thorough all the madness. I have deployed the job, will manually trigger it against the last merged change" [integration/config] - 10https://gerrit.wikimedia.org/r/736101 (https://phabricator.wikimedia.org/T293704) (owner: 10Catrope) [09:38:06] (03Merged) 10jenkins-bot: Deploy Codex docs to doc.wikimedia.org [integration/config] - 10https://gerrit.wikimedia.org/r/736101 (https://phabricator.wikimedia.org/T293704) (owner: 10Catrope) [09:40:53] you know your doc is outdated when it still refers to `REL1_24` :D [09:57:47] !log Reloading Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/736101 [09:57:49] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:10:06] (03PS1) 10Hashar: Fix Codec docsrc [integration/config] - 10https://gerrit.wikimedia.org/r/736996 (https://phabricator.wikimedia.org/T293704) [10:12:30] (03CR) 10Hashar: [C: 03+2] "The docsrc path was a bit off. I have adjusted the change and some doc is now properly copied at https://doc.wikimedia.org/codex/main/ I" [integration/config] - 10https://gerrit.wikimedia.org/r/736996 (https://phabricator.wikimedia.org/T293704) (owner: 10Hashar) [10:14:22] (03Merged) 10jenkins-bot: Fix Codec docsrc [integration/config] - 10https://gerrit.wikimedia.org/r/736996 (https://phabricator.wikimedia.org/T293704) (owner: 10Hashar) [10:20:46] 10Continuous-Integration-Config, 10Codex, 10Design-Systems-team (Design Systems Team FY2021-22 Kanban Board), 10Patch-For-Review: Deploy current main branch's docs site to doc.wikimedia.org - https://phabricator.wikimedia.org/T293704 (10hashar) I have manually triggered the `postmerge` pipeline against the... [12:04:30] !log shutdown deployment-logstash cluster [12:04:32] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:32:13] (03PS1) 10Hashar: tests: replace execfile() with imp.load_source() [integration/config] - 10https://gerrit.wikimedia.org/r/737050 [15:04:32] majavah: Congrats!!! [15:04:50] ? [15:04:59] on shutting down the deployment-logstash cluster [15:05:04] ah, indeed :D [15:05:15] I guess that was a while ago. [15:07:24] (03CR) 10Jforrester: "Oops, thanks." [integration/config] - 10https://gerrit.wikimedia.org/r/736630 (https://phabricator.wikimedia.org/T295002) (owner: 10Legoktm) [15:28:14] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Radar): beta logstash servers run out of disk space - https://phabricator.wikimedia.org/T288989 (10dancy) Today @Majavah turned off deployment-logstash04, deployment-logstash05, and deployment-logstash06. [15:48:59] (03PS1) 10Hashar: zuul: move doc_destination to a YAML file [integration/config] - 10https://gerrit.wikimedia.org/r/737059 [15:49:45] (03CR) 10Hashar: [C: 04-1] zuul: move doc_destination to a YAML file [integration/config] - 10https://gerrit.wikimedia.org/r/737059 (owner: 10Hashar) [16:01:50] (03CR) 10Reedy: "https://gerrit.wikimedia.org/r/c/mediawiki/vendor/+/716462 should be landing very soon, so this can be reverted" [integration/config] - 10https://gerrit.wikimedia.org/r/731741 (https://phabricator.wikimedia.org/T290274) (owner: 10Jforrester) [16:02:15] majavah: Do you know how the DNS record `logstash.svc.deployment-prep.eqiad1.wikimedia.cloud` is configured? A lookup on it returns the IP addresses of deployment-logstash04/05/06. [16:02:56] dancy: I think I configured it manually on Horizon [16:03:03] thx.. taking a look [16:03:04] what broke this time? [16:03:06] lol [16:03:20] Nothing broken this time. I just want to make sure it gets cleaned up [16:03:36] ah, sounds good [16:03:43] There are references to that name in scap.cfg [16:04:33] hmm.. I'm not at all sure what to replace it with :/ [16:05:08] if scap wants to log something to logstash, it'd ideally use the standard pipeline (rsyslogd+kafka), but scap may very well be reading things and I don't know what to do with that [16:05:09] ok. I see the DNS config in horizon. Thanks for the pointer. [16:05:55] Scap has a bunch of logging options, including syslog and udp2log, both of which are enabled for beta at this time. [16:10:38] 10Continuous-Integration-Config, 10Fundraising-Backlog, 10MediaWiki-extensions-CLDR, 10ci-test-error (WMF-deployed Build Failure): quibble-fundraising-cldr-REL1_35-php73-docker is failing - https://phabricator.wikimedia.org/T295114 (10Reedy) >But it looks like a mistake that this job is run for the master... [16:54:50] !log creating deployment-mwmaint02 (buster) to replace 01 running stretch [16:54:52] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:55:44] Any docker expert around to tell me I'm an idiot? My docker volume mount is "working" but empty. Command `docker run --rm -it --name scratch --mount type=bind,source="$(pwd)",target=/foo --entrypoint /bin/bash --workdir /foo docker-registry.wikimedia.org/releng/composer-test-php72:latest` [16:56:15] (And same story if I do my old standard of `-v $(pwd):/foo`.) [16:57:58] hmmm [16:58:20] * dancy tries a similar command [16:58:42] Even worse, if I change the mount to "$(pwd).." I get a directory empty except for the name of the current directory, which is also blank. [16:59:21] Maybe something's changed about docker's permissions to read the filesystem somehow? [17:00:40] so that container image runs as user 'nobody' by default. [17:01:02] (id 65534). I think that means that the mounted directory needs to be readable by that id. [17:01:28] Does anything change if you pass `--user root` on the docker run command line? [17:01:59] Nope. [17:02:28] Shall we do a screen share to poke around? [17:02:34] Sure. [17:08:17] 10Project-Admins: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706 (10Aklapper) @Ciell, @KCVelaga: Sorry, this somehow got lost! I added you both now. //Usual disclaimer: Please follow [guidelines](https://www.mediawiki.org/wiki/Phabricator/Creating_a... [17:10:09] Project beta-scap-sync-world build #26341: 04FAILURE in 17 min: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/26341/ [17:11:30] 10Continuous-Integration-Config, 10MediaWiki-Releasing: Have at least one job running REL1_XX with it's appropriate vendor repo - https://phabricator.wikimedia.org/T295175 (10Reedy) [17:12:20] 10Continuous-Integration-Config, 10MediaWiki-Releasing: Have at least one job running REL1_XX with it's appropriate vendor repo - https://phabricator.wikimedia.org/T295175 (10Reedy) [17:13:03] 10MediaWiki-Releasing, 10MW-1.37-notes, 10MW-1.37-release: Release 1.37.0-rc.2 - https://phabricator.wikimedia.org/T295176 (10Reedy) [17:13:19] 10MediaWiki-Releasing, 10MW-1.37-notes, 10MW-1.37-release: Release 1.37.0-rc.1 - https://phabricator.wikimedia.org/T294951 (10Reedy) [17:13:21] 10MediaWiki-Releasing, 10MW-1.37-notes, 10MW-1.37-release: Release 1.37.0-rc.2 - https://phabricator.wikimedia.org/T295176 (10Reedy) [17:13:31] 10MediaWiki-Releasing, 10MW-1.37-notes, 10MW-1.37-release: Write release announcement for 1.37.0-rc.2 - https://phabricator.wikimedia.org/T295178 (10Reedy) [17:13:43] 10MediaWiki-Releasing, 10MW-1.37-notes, 10MW-1.37-release: Release 1.37.0-rc.2 - https://phabricator.wikimedia.org/T295176 (10Reedy) [17:22:56] 10Continuous-Integration-Config, 10Fundraising-Backlog, 10MediaWiki-extensions-CLDR, 10ci-test-error (WMF-deployed Build Failure): quibble-fundraising-cldr-REL1_35-php73-docker is failing - https://phabricator.wikimedia.org/T295114 (10Umherirrender) The job is using quibble and also affected by T295036 Th... [17:23:07] 10MediaWiki-Releasing, 10MW-1.37-notes, 10MW-1.37-release: Release 1.37.0-rc.2 - https://phabricator.wikimedia.org/T295176 (10Reedy) [17:23:30] (03CR) 10Dduvall: backport: Approve selected/given changes (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/736589 (https://phabricator.wikimedia.org/T294454) (owner: 10Dduvall) [17:27:22] Yippee, build fixed! [17:27:23] Project beta-scap-sync-world build #26342: 09FIXED in 15 min: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/26342/ [17:28:42] 10Continuous-Integration-Config, 10Fundraising-Backlog, 10MediaWiki-extensions-CLDR, 10ci-test-error (WMF-deployed Build Failure): quibble-fundraising-cldr-REL1_35-php73-docker is failing - https://phabricator.wikimedia.org/T295114 (10Umherirrender) First failing: https://integration.wikimedia.org/ci/job/q... [17:35:07] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Radar): Migrate deployment-prep away from Debian Stretch to Buster/Bullseye - https://phabricator.wikimedia.org/T278641 (10Majavah) [17:35:20] 10Beta-Cluster-Infrastructure, 10Patch-For-Review: Upgrade deployment-prep MediaWiki clusters to Buster - https://phabricator.wikimedia.org/T278664 (10Majavah) 05Open→03Resolved [17:46:04] 10MediaWiki-Releasing, 10MW-1.37-notes, 10MW-1.37-release: Release 1.37.0-rc.2 - https://phabricator.wikimedia.org/T295176 (10Jdforrester-WMF) [17:48:45] 10MediaWiki-Releasing, 10MW-1.37-notes, 10MW-1.37-release: Release 1.37.0-rc.1 - https://phabricator.wikimedia.org/T294951 (10Reedy) [18:05:07] (03PS2) 10Hashar: zuul: move doc_destination to a YAML file [integration/config] - 10https://gerrit.wikimedia.org/r/737059 [18:05:10] (03PS1) 10Jforrester: Zuul: [mediawiki/extensions/WikiLambda] Switch quibble back to prod vendor [integration/config] - 10https://gerrit.wikimedia.org/r/737111 (https://phabricator.wikimedia.org/T290274) [18:07:48] (03CR) 10jerkins-bot: [V: 04-1] zuul: move doc_destination to a YAML file [integration/config] - 10https://gerrit.wikimedia.org/r/737059 (owner: 10Hashar) [18:08:04] (03CR) 10Jforrester: [C: 03+2] Zuul: [mediawiki/extensions/WikiLambda] Switch quibble back to prod vendor [integration/config] - 10https://gerrit.wikimedia.org/r/737111 (https://phabricator.wikimedia.org/T290274) (owner: 10Jforrester) [18:10:00] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/WikiLambda] Switch quibble back to prod vendor [integration/config] - 10https://gerrit.wikimedia.org/r/737111 (https://phabricator.wikimedia.org/T290274) (owner: 10Jforrester) [18:12:07] (03PS3) 10Hashar: zuul: move doc_destination to a YAML file [integration/config] - 10https://gerrit.wikimedia.org/r/737059 [18:12:23] !log Zuul: [mediawiki/extensions/WikiLambda] Switch quibble back to prod vendor T290274 [18:12:25] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:12:26] T290274: Add any necessary dependencies for WikiLambda to production vendor once they've passed Security review - https://phabricator.wikimedia.org/T290274 [18:13:40] (03CR) 10Ahmon Dancy: stage-train: this should make Tuesdays one command (031 comment) [tools/release] - 10https://gerrit.wikimedia.org/r/736072 (owner: 10Thcipriani) [18:20:26] poor doc.wikimedia.org , the ci config is a bit challenging ;) [18:20:33] I had some hacking, I am off! [18:25:50] 10Release-Engineering-Team, 10Scap, 10serviceops: Deploy Scap version 4.0.3 - https://phabricator.wikimedia.org/T294966 (10Legoktm) 05Open→03Resolved All done. [18:25:52] 10Release-Engineering-Team (Doing), 10Scap, 10serviceops: RESTBase deployment fails with scap internal error - https://phabricator.wikimedia.org/T294936 (10Legoktm) [18:50:49] (03PS3) 10Dduvall: backport: Approve selected/given changes [tools/scap] - 10https://gerrit.wikimedia.org/r/736589 (https://phabricator.wikimedia.org/T294454) [18:51:24] (03CR) 10jerkins-bot: [V: 04-1] backport: Approve selected/given changes [tools/scap] - 10https://gerrit.wikimedia.org/r/736589 (https://phabricator.wikimedia.org/T294454) (owner: 10Dduvall) [18:53:35] (03CR) 10Dduvall: backport: Approve selected/given changes (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/736589 (https://phabricator.wikimedia.org/T294454) (owner: 10Dduvall) [18:54:23] (03PS4) 10Dduvall: backport: Approve selected/given changes [tools/scap] - 10https://gerrit.wikimedia.org/r/736589 (https://phabricator.wikimedia.org/T294454) [18:58:49] 10Beta-Cluster-Infrastructure, 10Abstract Wikipedia team, 10function-evaluator, 10function-orchestrator: Work out how we're going to have "production-like" versions of the wikifunctions evaluator and orchestrator services in Beta Cluster - https://phabricator.wikimedia.org/T289315 (10Jdforrester-WMF) OK, s... [18:58:57] 10Beta-Cluster-Infrastructure, 10Abstract Wikipedia team, 10function-evaluator, 10function-orchestrator: Work out how we're going to have "production-like" versions of the wikifunctions evaluator and orchestrator services in Beta Cluster - https://phabricator.wikimedia.org/T289315 (10Jdforrester-WMF) a:03... [19:02:01] 10Beta-Cluster-Infrastructure, 10Abstract Wikipedia team, 10function-evaluator, 10function-orchestrator: Work out how we're going to have "production-like" versions of the wikifunctions evaluator and orchestrator services in Beta Cluster - https://phabricator.wikimedia.org/T289315 (10Majavah) >>! In T28931... [19:27:55] 10Release-Engineering-Team, 10Scap, 10serviceops: Deploy Scap version 4.0.3 - https://phabricator.wikimedia.org/T294966 (10dancy) Thanks @Legoktm ! [19:30:30] 🎉 [19:40:34] (03PS1) 10Dduvall: Configure Gerrit canonical web URL [tools/train-dev] - 10https://gerrit.wikimedia.org/r/737119 [19:42:41] dduvall: I just marked https://phabricator.wikimedia.org/T295187 as UBN, I assume it's a regression from the train [19:43:13] legoktm: ok, thank you [19:43:31] I also don't know if it's being cached in the parser cache anymore, which will need more work to fix [19:43:34] is it something that warrants immediate rollback? [19:44:01] I think so, people are seeing stuff in the wrong language [19:44:11] ok, sounds good [19:44:15] I did not 100% establish that it's caused by something in the train though [19:55:01] legoktm: if it's been cached, that's gonna be horrid right to clean up? [19:55:05] legoktm: hmm... i'm feeling a little hesitant to rollback if that isn't established [19:55:16] i don't want to cause more churn [19:55:18] You'd have to reject based on each group [19:56:16] it seems to be https://gerrit.wikimedia.org/r/c/mediawiki/core/+/721115 [19:56:26] but if we can get verification from someone that it is related and not fixable soon, rollback and wait until monday for a re-deploy makes total sense [19:56:45] which is in the wmf.7 changelog [19:57:10] I don't really know more than that, I'm just trying to relay/flag this [19:57:36] got it. yeah thanks for doing that [19:57:50] is that reproducible locally? testing if that patch is the cause would be a good start [19:58:06] Jdlrobson: ^ any thoughts as to cause? [19:58:54] and very rough timeline on how long this might take to investigate/fix? [20:00:24] (03PS2) 10Ahmon Dancy: Further updates to release process [tools/scap] - 10https://gerrit.wikimedia.org/r/736550 [20:04:09] (03CR) 10Dduvall: [C: 03+2] Further updates to release process [tools/scap] - 10https://gerrit.wikimedia.org/r/736550 (owner: 10Ahmon Dancy) [20:04:48] (03Merged) 10jenkins-bot: Further updates to release process [tools/scap] - 10https://gerrit.wikimedia.org/r/736550 (owner: 10Ahmon Dancy) [20:09:02] 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review, 10Release, 10Train Deployments: 1.38.0-wmf.7 deployment blockers - https://phabricator.wikimedia.org/T293948 (10dduvall) 05Resolved→03Open [20:09:22] 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review, 10Release, 10Train Deployments: 1.38.0-wmf.7 deployment blockers - https://phabricator.wikimedia.org/T293948 (10dduvall) [20:49:03] (03CR) 10Ahmon Dancy: [C: 03+2] Normalize files indentation [tools/train-dev] - 10https://gerrit.wikimedia.org/r/736442 (owner: 10Hashar) [20:49:40] (03Merged) 10jenkins-bot: Normalize files indentation [tools/train-dev] - 10https://gerrit.wikimedia.org/r/736442 (owner: 10Hashar) [20:55:47] (03PS1) 10Krinkle: zuul: Update pattern to match address change in my Gerrit account [integration/config] - 10https://gerrit.wikimedia.org/r/737128 [20:56:24] (03CR) 10Krinkle: "Oh, right... Let me forge my previous gitconfig first." [integration/config] - 10https://gerrit.wikimedia.org/r/737128 (owner: 10Krinkle) [20:56:39] (03PS2) 10Krinkle: zuul: Update pattern to match address change in my Gerrit account [integration/config] - 10https://gerrit.wikimedia.org/r/737128 [20:57:48] (03PS3) 10Krinkle: zuul: Update pattern to match address change in my Gerrit account [integration/config] - 10https://gerrit.wikimedia.org/r/737128 [20:58:48] (03PS4) 10Krinkle: zuul: Update email pattern to new address for my Gerrit account [integration/config] - 10https://gerrit.wikimedia.org/r/737128 [20:59:07] (03CR) 10Krinkle: [C: 03+2] zuul: Update email pattern to new address for my Gerrit account [integration/config] - 10https://gerrit.wikimedia.org/r/737128 (owner: 10Krinkle) [21:00:53] (03Merged) 10jenkins-bot: zuul: Update email pattern to new address for my Gerrit account [integration/config] - 10https://gerrit.wikimedia.org/r/737128 (owner: 10Krinkle) [21:10:24] (03PS1) 10Ahmon Dancy: www: rm /var/run/apache2/apache2.pid on startup [tools/train-dev] - 10https://gerrit.wikimedia.org/r/737130 [21:10:59] (03CR) 10Ahmon Dancy: [C: 03+2] www: rm /var/run/apache2/apache2.pid on startup [tools/train-dev] - 10https://gerrit.wikimedia.org/r/737130 (owner: 10Ahmon Dancy) [21:11:25] (03Merged) 10jenkins-bot: www: rm /var/run/apache2/apache2.pid on startup [tools/train-dev] - 10https://gerrit.wikimedia.org/r/737130 (owner: 10Ahmon Dancy) [21:15:11] (03CR) 10Ahmon Dancy: [C: 04-1] Configure Gerrit canonical web URL (031 comment) [tools/train-dev] - 10https://gerrit.wikimedia.org/r/737119 (owner: 10Dduvall) [21:21:43] (03PS1) 10Ahmon Dancy: update-repos: Don't pass --prune to git remote update [tools/train-dev] - 10https://gerrit.wikimedia.org/r/737133 [22:59:01] 10Continuous-Integration-Config, 10Release-Engineering-Team (Seen), 10Scap, 10MW-on-K8s, and 5 others: Define variant Wikimedia production config in compiled, static files - https://phabricator.wikimedia.org/T223602 (10awight) Where has this proposal been all my life? I'd like to help but it's unclear wha... [23:11:09] (03CR) 10Catrope: "Thank you! It works great, the docs were published. The CSS is broken, but that's because I have to configure Vitepress to tell it that it" [integration/config] - 10https://gerrit.wikimedia.org/r/736101 (https://phabricator.wikimedia.org/T293704) (owner: 10Catrope) [23:14:41] (03PS2) 10Dduvall: Configure Gerrit canonical web URL [tools/train-dev] - 10https://gerrit.wikimedia.org/r/737119 [23:16:16] (03CR) 10Dduvall: Configure Gerrit canonical web URL (031 comment) [tools/train-dev] - 10https://gerrit.wikimedia.org/r/737119 (owner: 10Dduvall) [23:16:30] (03CR) 10Dduvall: Configure Gerrit canonical web URL (031 comment) [tools/train-dev] - 10https://gerrit.wikimedia.org/r/737119 (owner: 10Dduvall) [23:18:26] (03CR) 10Dduvall: Configure Gerrit canonical web URL (031 comment) [tools/train-dev] - 10https://gerrit.wikimedia.org/r/737119 (owner: 10Dduvall) [23:22:00] (03CR) 10Ahmon Dancy: [C: 03+2] Configure Gerrit canonical web URL [tools/train-dev] - 10https://gerrit.wikimedia.org/r/737119 (owner: 10Dduvall) [23:22:54] (03Merged) 10jenkins-bot: Configure Gerrit canonical web URL [tools/train-dev] - 10https://gerrit.wikimedia.org/r/737119 (owner: 10Dduvall)