[08:19:48] (03CR) 10Hashar: [C: 03+2] "Nice :]" [tools/scap] - 10https://gerrit.wikimedia.org/r/747959 (owner: 10Ahmon Dancy) [08:22:48] (03Merged) 10jenkins-bot: git.py: Implement remote_set in a less destructive way [tools/scap] - 10https://gerrit.wikimedia.org/r/747959 (owner: 10Ahmon Dancy) [08:43:32] <_joe_> thcipriani / hashar / others: I see no deploys this week but deployments are active next week (from the 27th) which seems suboptimal to me [08:49:52] _joe_: that's a mistake in the tool [08:49:56] I believe it's automated [08:50:18] <_joe_> RhinosF1: that doesn't change the fact that it needs to be edited [08:53:23] Done [09:08:24] (03CR) 10Thiemo Kreuz (WMDE): [C: 03+1] "Can you add a line to the commit message that explains what this is needed for? Otherwise looks good, as far as I can tell." [integration/quibble] - 10https://gerrit.wikimedia.org/r/693456 (owner: 10Awight) [09:12:11] (03CR) 10Thiemo Kreuz (WMDE): [C: 03+2] Expand redirection to capture logging (032 comments) [integration/quibble] - 10https://gerrit.wikimedia.org/r/693457 (owner: 10Awight) [09:14:01] 10Beta-Cluster-Infrastructure: Puppet broken on deployment-deploy03.deployment-prep.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T297853 (10Majavah) 05Openβ†’03Resolved a:03Majavah [09:25:20] (03CR) 10Awight: "> We run phpunit with SpeedTrapListener to warn about tests that take "too long" (currently a 50ms threshold for all kinds of phpunit test" [integration/quibble] - 10https://gerrit.wikimedia.org/r/587885 (https://phabricator.wikimedia.org/T235449) (owner: 10Awight) [10:42:50] 10Release-Engineering-Team, 10MW-on-K8s, 10serviceops: Build MediaWiki images for kubernetes on the deployment servers - https://phabricator.wikimedia.org/T297673 (10Joe) p:05Triageβ†’03Medium a:03Joe [11:20:40] (03PS5) 10Awight: Introduce utility to redirect stdout and stderr [integration/quibble] - 10https://gerrit.wikimedia.org/r/693456 [11:20:42] (03PS5) 10Awight: Expand redirection to capture logging [integration/quibble] - 10https://gerrit.wikimedia.org/r/693457 [11:20:45] (03PS4) 10Awight: Wrapper to pretty-print parallel job progress [integration/quibble] - 10https://gerrit.wikimedia.org/r/693458 [11:20:48] (03PS41) 10Awight: Parallelism as a command object [integration/quibble] - 10https://gerrit.wikimedia.org/r/587885 (https://phabricator.wikimedia.org/T235449) [11:23:33] (03CR) 10Awight: Parallelism as a command object (031 comment) [integration/quibble] - 10https://gerrit.wikimedia.org/r/587885 (https://phabricator.wikimedia.org/T235449) (owner: 10Awight) [11:23:38] (03CR) 10jerkins-bot: [V: 04-1] Expand redirection to capture logging [integration/quibble] - 10https://gerrit.wikimedia.org/r/693457 (owner: 10Awight) [11:23:59] (03CR) 10jerkins-bot: [V: 04-1] Wrapper to pretty-print parallel job progress [integration/quibble] - 10https://gerrit.wikimedia.org/r/693458 (owner: 10Awight) [11:24:05] (03CR) 10jerkins-bot: [V: 04-1] Parallelism as a command object [integration/quibble] - 10https://gerrit.wikimedia.org/r/587885 (https://phabricator.wikimedia.org/T235449) (owner: 10Awight) [11:24:25] (03CR) 10jerkins-bot: [V: 04-1] Introduce utility to redirect stdout and stderr [integration/quibble] - 10https://gerrit.wikimedia.org/r/693456 (owner: 10Awight) [11:25:06] (03CR) 10Awight: Introduce utility to redirect stdout and stderr (031 comment) [integration/quibble] - 10https://gerrit.wikimedia.org/r/693456 (owner: 10Awight) [11:26:49] (03PS6) 10Awight: Introduce utility to redirect stdout and stderr [integration/quibble] - 10https://gerrit.wikimedia.org/r/693456 [11:26:51] (03PS6) 10Awight: Expand redirection to capture logging [integration/quibble] - 10https://gerrit.wikimedia.org/r/693457 [11:26:54] (03PS5) 10Awight: Wrapper to pretty-print parallel job progress [integration/quibble] - 10https://gerrit.wikimedia.org/r/693458 [11:26:57] (03PS42) 10Awight: Parallelism as a command object [integration/quibble] - 10https://gerrit.wikimedia.org/r/587885 (https://phabricator.wikimedia.org/T235449) [12:42:19] 10Project-Admins: Create OgvJS project as subproject of TimedMediaHandler - https://phabricator.wikimedia.org/T298016 (10Peachey88) Just a reminder, when you use Subprojects, It changes how group memberships are [[ https://secure.phabricator.com/book/phabricator/article/projects/#subprojects-and-milestones | man... [12:46:03] (03PS7) 10Awight: Introduce utility to redirect stdout and stderr [integration/quibble] - 10https://gerrit.wikimedia.org/r/693456 [12:59:04] (03PS8) 10Awight: Introduce utility to redirect stdout and stderr [integration/quibble] - 10https://gerrit.wikimedia.org/r/693456 [12:59:10] (03PS7) 10Awight: Expand redirection to capture logging [integration/quibble] - 10https://gerrit.wikimedia.org/r/693457 [12:59:16] (03PS6) 10Awight: Wrapper to pretty-print parallel job progress [integration/quibble] - 10https://gerrit.wikimedia.org/r/693458 [12:59:22] (03PS43) 10Awight: Parallelism as a command object [integration/quibble] - 10https://gerrit.wikimedia.org/r/587885 (https://phabricator.wikimedia.org/T235449) [12:59:32] (03CR) 10Awight: Expand redirection to capture logging (032 comments) [integration/quibble] - 10https://gerrit.wikimedia.org/r/693457 (owner: 10Awight) [13:09:10] 10Continuous-Integration-Config, 10Shellbox: Upgrade to php-ast 1.0.7 in CI - https://phabricator.wikimedia.org/T298008 (10Daimona) I don't know precisely how that job works, but it seems to be using the [[https://gerrit.wikimedia.org/r/plugins/gitiles/integration/config/+/ad06eff51df944d1d9dd888b4478fb50bd2bd... [13:09:45] 10Continuous-Integration-Config, 10Shellbox: shellbox-pipeline-test jobs fail because they use php-ast 0.1.6 - https://phabricator.wikimedia.org/T298008 (10Daimona) [14:44:58] I got into a pinch with scap on a wmcloud cluster (maps-experiments). It always wants to pull from deployment-deploy01.deployment-prep... even though I've run puppet and made sure that /etc/scap.cfg references deploy03, the new instance name. [14:46:53] why is it trying to pull from deployment-prep in the first place if it's outside the deployment-prep project? [14:50:10] I'm not entirely sure, but the arrangement seems to be something like the mw-debug servers. Deployment happens normally on the beta cluster, then this external box fetches updated code using `scap deploy-local` [14:50:26] In case it helps, https://wikitech.wikimedia.org/wiki/Maps/Maps_experiments [14:57:52] I'll ask the team involved, maybe it's possible to migrate back to deployment-prep now or later. [14:58:42] the commands on that wiki page seem to work fine for me? [14:59:44] majavah: Try the next repo that needs updating, s/kartotherian/tilerator/ in those lines... [14:59:49] Thanks for taking a look! [15:00:56] you're hit by T197470, /srv/deployment/tilerator/deploy-cache/.config still has the old name [15:00:56] T197470: find a way to systematically update the deployment server name across all repos - https://phabricator.wikimedia.org/T197470 [15:01:51] Ah--I did try to edit those configs by hand, to bootstrap it. [15:02:12] anyhow please try to get rid of the cross-project scap use (especially the ugly hack like that), it's a pain since the people doing maintenance on deployment-prep (which is rare enough since no-one "owns" it) don't always have access outside the project [15:02:37] +1 I'll suggest this [15:08:38] root@maps-master01:/srv/deployment/tilerator# grep -Hrn git_server . [15:08:38] ./deploy-cache/.config:201:git_server: deployment-deploy01.deployment-prep.eqiad1.wikimedia.cloud [15:08:46] awight: I think that's your problem [15:09:33] yeah, it works after fixing it [15:10:49] although `promote` fails on "Port 6534 not up. Waiting 3.00s" [15:17:45] hrm, I changed that manually too and it's reset to deploy01 now. Then I deleted the /srv/deployment/tilerator/deploy directory and it isn't recreated by puppet. [15:18:59] Without dragging you too much further down this rabbithole, do you have a recommendation about how to this external cluster should be set up? Is it problematic that deployment-prep would call out to another cluster (for serving maps to beta users)? [15:24:06] not an ideal end-of-day question... I'll come back to this tomorrow :-) [15:26:08] awight: I'd recommend setting up a local scap master and using that [16:07:18] thanks _joe_ and RhinosF1 <3 [16:07:52] Np thcipriani [16:11:52] <_joe_> thcipriani: uhhh what did I do? [16:12:56] _joe_: sorry for the context-less thank. Thank you for poking about the deployment calendar being innacurate. And thanks RhinosF.1 for fixing the calendar for the week. [16:21:06] 10Beta-Cluster-Infrastructure: Puppet broken on deployment-deploy03.deployment-prep.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T297853 (10dancy) Thank you! [16:49:33] 10Release-Engineering-Team (Radar), 10serviceops, 10Datacenter-Switchover: Switch deployment server to codfw (July 2021) - https://phabricator.wikimedia.org/T285820 (10Majavah) 05Openβ†’03Declined This was not done this time. [17:06:48] (03PS2) 10Ahmon Dancy: scap apply-patches: Make output more compact [tools/scap] - 10https://gerrit.wikimedia.org/r/748226 [17:17:32] (03CR) 1020after4: [C: 03+2] "I have no problem with this change, so +2, however:" [tools/scap] - 10https://gerrit.wikimedia.org/r/748226 (owner: 10Ahmon Dancy) [17:18:35] (03Merged) 10jenkins-bot: scap apply-patches: Make output more compact [tools/scap] - 10https://gerrit.wikimedia.org/r/748226 (owner: 10Ahmon Dancy) [17:29:03] 10Release-Engineering-Team (Next), 10Release, 10Train Deployments: 1.38.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T293957 (10Esanders) [17:37:41] (03PS1) 10Urbanecm: [Layout] Bump PHP version for wikimedia-cz/web-* [integration/config] - 10https://gerrit.wikimedia.org/r/748762 [18:45:16] 10Phabricator, 10Domains, 10SRE, 10Traffic: The phab.wiki domain redirect suddenly outputs "404, this domain is not configured" - https://phabricator.wikimedia.org/T298041 (10Dzahn) [18:46:19] 10Phabricator, 10Domains, 10SRE, 10Traffic: The phab.wiki domain redirect suddenly outputs "404, this domain is not configured" - https://phabricator.wikimedia.org/T298041 (10MarcoAurelio) @Dzahn Searching through my inbox, the first occurence of phab.wiki is from 2019, so it has been around for a short wh... [18:48:04] 10Phabricator, 10Domains, 10SRE, 10Traffic: The phab.wiki domain redirect suddenly outputs "404, this domain is not configured" - https://phabricator.wikimedia.org/T298041 (10Dzahn) Let me try to find the original old list of all the .wiki domains WMF once had. Will get back to you. [18:56:41] 10Phabricator, 10Domains, 10SRE, 10Traffic: The phab.wiki domain redirect suddenly outputs "404, this domain is not configured" - https://phabricator.wikimedia.org/T298041 (10Dzahn) @MarcoAurelio I still have an email from 2014 (!sic) that talks about .wiki domains and has a list that WMF was going to get.... [19:03:04] PROBLEM - SSH on contint1001.mgmt is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [19:03:42] (03PS1) 10Majavah: zuul: Make OAuth depend on AbuseFilter [integration/config] - 10https://gerrit.wikimedia.org/r/748788 [19:05:27] 10Phabricator, 10Domains, 10SRE, 10Traffic: The phab.wiki domain redirect suddenly outputs "404, this domain is not configured" - https://phabricator.wikimedia.org/T298041 (10Aklapper) I could neither find nor remember any trace of the domain `phab.wiki` either here... [19:08:50] 10Phabricator, 10Domains, 10SRE, 10Traffic: The phab.wiki domain redirect suddenly outputs "404, this domain is not configured" - https://phabricator.wikimedia.org/T298041 (10Dzahn) But I can confirm it does pop up in Google search results. maybe it's possible Phacility owns/owned that and pointed to us as... [19:09:47] (03CR) 10Zabe: zuul: Make OAuth depend on AbuseFilter (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/748788 (owner: 10Majavah) [19:10:53] (03PS2) 10Majavah: zuul: Make OAuth depend on AbuseFilter [integration/config] - 10https://gerrit.wikimedia.org/r/748788 [19:11:42] (03CR) 10Majavah: zuul: Make OAuth depend on AbuseFilter (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/748788 (owner: 10Majavah) [19:13:16] 10Phabricator, 10Domains, 10SRE, 10Traffic: The phab.wiki domain redirect suddenly outputs "404, this domain is not configured" - https://phabricator.wikimedia.org/T298041 (10Zabe) Using the global search tool I found https://wikitech.wikimedia.org/wiki/User:Revi/phab.wiki. [19:15:43] (03CR) 10Zabe: [C: 03+1] zuul: Make OAuth depend on AbuseFilter (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/748788 (owner: 10Majavah) [19:20:33] 10Phabricator, 10Domains, 10SRE, 10Traffic: The phab.wiki domain redirect suddenly outputs "404, this domain is not configured" - https://phabricator.wikimedia.org/T298041 (10Dzahn) Great detective work @Zabe, Thanks! added @revi Hi @Revi :) [19:21:06] (03CR) 10Ahmon Dancy: [C: 03+1] "Code looks good. I will test in train-dev before +2'ing." [tools/scap] - 10https://gerrit.wikimedia.org/r/743272 (https://phabricator.wikimedia.org/T294453) (owner: 10Jeena Huneidi) [19:22:27] 10Continuous-Integration-Infrastructure: Revisit how CI works for released branches - https://phabricator.wikimedia.org/T298043 (10Jdlrobson) [19:23:25] 10Phabricator, 10Domains, 10SRE, 10Traffic: The phab.wiki domain redirect suddenly outputs "404, this domain is not configured" - https://phabricator.wikimedia.org/T298041 (10Dzahn) https://wikitech.wikimedia.org/wiki/User_talk:Revi/phab.wiki [19:26:22] 10Phabricator, 10Domains, 10SRE, 10Traffic: The phab.wiki domain redirect suddenly outputs "404, this domain is not configured" - https://phabricator.wikimedia.org/T298041 (10Sunpriat2) @revi timestamps: https://web.archive.org/web/*/https://phab.wiki/* 1 https://web.archive.org/web/2019*/https://phab.wiki... [19:33:48] 10Phabricator, 10Domains, 10SRE: The phab.wiki domain redirect suddenly outputs "404, this domain is not configured" - https://phabricator.wikimedia.org/T298041 (10Dzahn) [19:34:14] 10Phabricator, 10Domains, 10SRE, 10serviceops-radar: The phab.wiki domain redirect suddenly outputs "404, this domain is not configured" - https://phabricator.wikimedia.org/T298041 (10Dzahn) [19:42:21] (03CR) 10Thcipriani: [C: 03+2] "LGTM" [tools/scap] - 10https://gerrit.wikimedia.org/r/748229 (owner: 10Ahmon Dancy) [19:43:01] (03Merged) 10jenkins-bot: change active_wikiversions interface [tools/scap] - 10https://gerrit.wikimedia.org/r/748229 (owner: 10Ahmon Dancy) [19:45:04] (03CR) 10Ahmon Dancy: change active_wikiversions interface (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/748229 (owner: 10Ahmon Dancy) [19:47:06] (03CR) 10Thcipriani: change active_wikiversions interface (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/748229 (owner: 10Ahmon Dancy) [19:54:28] (03PS14) 10Ahmon Dancy: scap backport: validate changes [tools/scap] - 10https://gerrit.wikimedia.org/r/743272 (https://phabricator.wikimedia.org/T294453) (owner: 10Jeena Huneidi) [19:55:02] (03CR) 10Ahmon Dancy: "rebased on top of https://gerrit.wikimedia.org/r/c/mediawiki/tools/scap/+/748229" [tools/scap] - 10https://gerrit.wikimedia.org/r/743272 (https://phabricator.wikimedia.org/T294453) (owner: 10Jeena Huneidi) [20:01:30] (03CR) 10Ahmon Dancy: [C: 03+2] scap backport: validate changes (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/743272 (https://phabricator.wikimedia.org/T294453) (owner: 10Jeena Huneidi) [20:02:18] 10GitLab (Project Migration), 10Release-Engineering-Team (Doing), 10User-brennen, 10Voice & Tone: Rename mainline Git branch from "master" to "main" on all WMF-hosted repositories during GitLab migration - https://phabricator.wikimedia.org/T281593 (10thcipriani) [20:02:24] 10GitLab (Project Migration), 10Release-Engineering-Team (Yak Shaving πŸƒπŸͺ’), 10User-brennen, 10Voice & Tone: Delete temporary main branch in mediawiki/core.git until rename from master to main is ready - https://phabricator.wikimedia.org/T296205 (10thcipriani) 05Openβ†’03Resolved [[ https://gerrit.wikimedi... [20:02:29] (03Merged) 10jenkins-bot: scap backport: validate changes [tools/scap] - 10https://gerrit.wikimedia.org/r/743272 (https://phabricator.wikimedia.org/T294453) (owner: 10Jeena Huneidi) [20:03:16] RECOVERY - SSH on contint1001.mgmt is OK: SSH OK - OpenSSH_6.6 (protocol 2.0) https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [20:08:50] (03PS1) 10Ahmon Dancy: scap backport: Wait for changes to be merged [tools/scap] - 10https://gerrit.wikimedia.org/r/748796 (https://phabricator.wikimedia.org/T295494) [20:09:24] (03CR) 10jerkins-bot: [V: 04-1] scap backport: Wait for changes to be merged [tools/scap] - 10https://gerrit.wikimedia.org/r/748796 (https://phabricator.wikimedia.org/T295494) (owner: 10Ahmon Dancy) [20:09:44] (03PS2) 10Ahmon Dancy: scap backport: Wait for changes to be merged [tools/scap] - 10https://gerrit.wikimedia.org/r/748796 (https://phabricator.wikimedia.org/T295494) [20:10:17] (03CR) 10jerkins-bot: [V: 04-1] scap backport: Wait for changes to be merged [tools/scap] - 10https://gerrit.wikimedia.org/r/748796 (https://phabricator.wikimedia.org/T295494) (owner: 10Ahmon Dancy) [20:16:05] (03PS3) 10Ahmon Dancy: scap backport: Wait for changes to be merged [tools/scap] - 10https://gerrit.wikimedia.org/r/748796 (https://phabricator.wikimedia.org/T295494) [20:19:44] 10Release-Engineering-Team (Doing), 10MW-on-K8s, 10Release Pipeline, 10User-brennen: Scap backport change_url command - https://phabricator.wikimedia.org/T287042 (10dancy) [20:20:26] 10Release-Engineering-Team (Done by Wed 24 Nov 🧟), 10Scap, 10Patch-For-Review: Scap backport change_url: Implement β€œwait for my commits to be merged” - https://phabricator.wikimedia.org/T295494 (10dancy) 05Openβ†’03In progress a:03dancy [20:31:22] (03PS1) 10Ahmon Dancy: git.py:update_submodules add checkout and force parameters [tools/scap] - 10https://gerrit.wikimedia.org/r/748800 [20:31:30] (03PS1) 10Ahmon Dancy: scap/plugins/patches.py: Very minor reuse of a variable [tools/scap] - 10https://gerrit.wikimedia.org/r/748801 [20:31:42] (03PS1) 10Ahmon Dancy: utils.py: add dir_is_empty function [tools/scap] - 10https://gerrit.wikimedia.org/r/748802 [21:51:23] (03CR) 10Thcipriani: [C: 03+2] "Does what it says on the tin" [tools/scap] - 10https://gerrit.wikimedia.org/r/748802 (owner: 10Ahmon Dancy) [21:52:15] (03CR) 10Thcipriani: [C: 03+2] scap/plugins/patches.py: Very minor reuse of a variable [tools/scap] - 10https://gerrit.wikimedia.org/r/748801 (owner: 10Ahmon Dancy) [21:54:18] (03CR) 10Thcipriani: [C: 03+2] git.py:update_submodules add checkout and force parameters [tools/scap] - 10https://gerrit.wikimedia.org/r/748800 (owner: 10Ahmon Dancy) [21:54:38] (03Merged) 10jenkins-bot: utils.py: add dir_is_empty function [tools/scap] - 10https://gerrit.wikimedia.org/r/748802 (owner: 10Ahmon Dancy) [21:54:40] (03Merged) 10jenkins-bot: scap/plugins/patches.py: Very minor reuse of a variable [tools/scap] - 10https://gerrit.wikimedia.org/r/748801 (owner: 10Ahmon Dancy) [21:58:44] (03Merged) 10jenkins-bot: git.py:update_submodules add checkout and force parameters [tools/scap] - 10https://gerrit.wikimedia.org/r/748800 (owner: 10Ahmon Dancy) [22:03:06] 10Release-Engineering-Team (Doing), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.38.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T293953 (10thcipriani) 05Openβ†’03Resolved [22:09:03] (03CR) 10Hashar: [C: 03+2] [Layout] Bump PHP version for wikimedia-cz/web-* [integration/config] - 10https://gerrit.wikimedia.org/r/748762 (owner: 10Urbanecm) [22:09:20] thanks hashar! [22:09:24] (and good evening) [22:11:06] (03Merged) 10jenkins-bot: [Layout] Bump PHP version for wikimedia-cz/web-* [integration/config] - 10https://gerrit.wikimedia.org/r/748762 (owner: 10Urbanecm) [22:11:57] (03CR) 10Hashar: "deployed!" [integration/config] - 10https://gerrit.wikimedia.org/r/748762 (owner: 10Urbanecm) [22:18:58] (03CR) 10Urbanecm: [Layout] Bump PHP version for wikimedia-cz/web-* (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/748762 (owner: 10Urbanecm) [23:01:13] 10Continuous-Integration-Config, 10phan: Add a phan job to gated extensions/skins and core testing each change with all of them - https://phabricator.wikimedia.org/T298053 (10Jdforrester-WMF) [23:56:58] 10Continuous-Integration-Config, 10phan: Add a phan job to gated extensions/skins and core testing each change with all of them - https://phabricator.wikimedia.org/T298053 (10BilalShirwani)