[06:00:39] jbond: `[RelEng] FAIL: train-presync` emphasis on the subject having FAIL ;) [06:01:12] your patch enhancing systemd timer email is the largest time productivity gain of the year [06:15:39] 10Deployments, 10Release-Engineering-Team: Deployment server permissions are broken preventing MediaWiki deployment - https://phabricator.wikimedia.org/T331378 (10hashar) [06:16:54] 10Deployments, 10Release-Engineering-Team: Deployment server permissions are broken preventing MediaWiki deployment - https://phabricator.wikimedia.org/T331378 (10hashar) p:05Triageβ†’03Unbreak! That is the first time the train runs since we did the datacenter switch over. The steps used to move the deployme... [06:17:19] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10Patch-For-Review, 10Release, 10Train Deployments: 1.40.0-wmf.26 deployment blockers - https://phabricator.wikimedia.org/T330204 (10hashar) [06:17:21] 10Deployments, 10Release-Engineering-Team: Deployment server permissions are broken preventing MediaWiki deployment - https://phabricator.wikimedia.org/T331378 (10hashar) [06:27:31] 10Deployments, 10Release-Engineering-Team: Deployment server permissions are broken preventing MediaWiki deployment - https://phabricator.wikimedia.org/T331378 (10hashar) At first I have suspected a `rsync` issue when we switched over cause some files are owned by UID `498` which has no name associated with it... [06:40:41] 10Deployments, 10Release-Engineering-Team: Deployment server permissions are broken preventing MediaWiki deployment - https://phabricator.wikimedia.org/T331378 (10hashar) Things I have checked we do have the profile script to set umask: ` name=/etc/profile.d/umask-wikidev.sh,lang=sh # this file is managed by p... [07:17:31] 10Deployments, 10Release-Engineering-Team: Deployment server permissions are broken preventing MediaWiki deployment - https://phabricator.wikimedia.org/T331378 (10Joe) The problem seems to be common to both servers - it is impossible to git commit as mwpresync in general on either deployment server anymore. I... [07:18:36] 10Deployments, 10Release-Engineering-Team: Deployment server permissions are broken preventing MediaWiki deployment - https://phabricator.wikimedia.org/T331378 (10Joe) One useful attempt would be to restore a backup of `/srv/mediawiki-staging` from deploy1002 dating before the switchover, and check permissions... [07:47:07] 10Phabricator, 10wikitech.wikimedia.org: Remove Phabricator 2FA from account DMartin-WMF - https://phabricator.wikimedia.org/T331170 (10Aklapper) 05Openβ†’03Resolved a:03Aklapper Stripped 2FA from user `DMartin-WMF` in Phab. Please feel encouraged to add it again at https://phabricator.wikimedia.org/settin... [07:55:46] 10Phabricator, 10DBA, 10Patch-For-Review: Switchover m3 master db1159 -> db1101 - https://phabricator.wikimedia.org/T331384 (10Marostegui) [08:00:28] 10Phabricator, 10DBA, 10Patch-For-Review: Switchover m3 master db1159 -> db1101 - https://phabricator.wikimedia.org/T331384 (10Marostegui) [08:01:09] 10Phabricator, 10DBA, 10Patch-For-Review: Switchover m3 master db1159 -> db1101 - https://phabricator.wikimedia.org/T331384 (10Marostegui) [08:03:03] 10Beta-Cluster-Infrastructure, 10Growth-Team, 10PageTriage, 10Wikimedia-maintenance-script-run, and 2 others: Install and test PageTriage for zhwiki on Beta Cluster - https://phabricator.wikimedia.org/T323378 (10Shizhao) >>! 在T323378#8413476中，@Stangε†™ι“οΌš > - i18n: 87% completed for [[ https://translatewiki.n... [08:03:10] 10Phabricator, 10DBA, 10Patch-For-Review: Switchover m3 master db1159 -> db1101 - https://phabricator.wikimedia.org/T331384 (10Marostegui) [08:05:51] 10Deployments, 10Release-Engineering-Team: Deployment server permissions are broken preventing MediaWiki deployment - https://phabricator.wikimedia.org/T331378 (10hashar) I have removed the git staged files from `/srv/mediawiki-staging` restoring its state. >>! In T331378#8671320, @Joe wrote: > One useful at... [08:11:14] 10Phabricator, 10DBA, 10Patch-For-Review: Switchover m3 master db1159 -> db1101 - https://phabricator.wikimedia.org/T331384 (10Marostegui) [08:16:45] 10Phabricator, 10DBA, 10Patch-For-Review: Switchover m3 master db1159 -> db1101 - https://phabricator.wikimedia.org/T331384 (10Marostegui) [08:17:08] 10Phabricator, 10DBA, 10Patch-For-Review: Switchover m3 master db1159 -> db1101 - https://phabricator.wikimedia.org/T331384 (10Marostegui) [08:17:13] !log Updating Jenkins jobs for JJB updgrade from 4.1.0 to 4.2.0 | https://gerrit.wikimedia.org/r/c/integration/config/+/893765 [08:17:16] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [08:18:01] (03CR) 10Hashar: [C: 03+2] "Thanks James! I have deployed the jobs ;)" [integration/config] - 10https://gerrit.wikimedia.org/r/893765 (owner: 10Hashar) [08:18:23] 10Phabricator, 10DBA, 10Patch-For-Review: Switchover m3 master db1159 -> db1101 - https://phabricator.wikimedia.org/T331384 (10Marostegui) [08:19:22] (03Merged) 10jenkins-bot: Upgrade JJB from 4.1.0 to 4.2.0 [integration/config] - 10https://gerrit.wikimedia.org/r/893765 (owner: 10Hashar) [08:21:25] 10Phabricator, 10DBA, 10Patch-For-Review: Switchover m3 master db1159 -> db1101 - https://phabricator.wikimedia.org/T331384 (10Marostegui) Test [08:22:09] (03PS4) 10Hashar: Upgrade JJB from 4.2.0 to 4.3.0 [integration/config] - 10https://gerrit.wikimedia.org/r/893786 [08:22:25] 10Phabricator, 10DBA, 10Patch-For-Review: Switchover m3 master db1159 -> db1101 - https://phabricator.wikimedia.org/T331384 (10Marostegui) [08:26:27] 10Phabricator, 10DBA, 10Patch-For-Review: Switchover m3 master db1159 -> db1101 - https://phabricator.wikimedia.org/T331384 (10Marostegui) [08:27:15] 10Phabricator, 10DBA, 10Patch-For-Review: Switchover m3 master db1159 -> db1101 - https://phabricator.wikimedia.org/T331384 (10Marostegui) [08:27:29] 10Phabricator, 10DBA, 10Patch-For-Review: Switchover m3 master db1159 -> db1101 - https://phabricator.wikimedia.org/T331384 (10Marostegui) 05Openβ†’03Resolved a:03Marostegui Done [08:37:22] 10Deployments, 10Release-Engineering-Team: Deployment server permissions are broken preventing MediaWiki deployment - https://phabricator.wikimedia.org/T331378 (10hashar) Doing further digging, all the files which are not group writable under `/srv/mediawiki-staging` are in the `wikidev` group. I think they sh... [09:14:03] 10Beta-Cluster-Infrastructure, 10Growth-Team, 10PageTriage, 10Wikimedia-maintenance-script-run, and 2 others: Install and test PageTriage for zhwiki on Beta Cluster - https://phabricator.wikimedia.org/T323378 (10kostajh) >>! In T323378#8668647, @Xiplus wrote: > wgPageTriageEnableEnglishWikipediaFeatures ne... [09:18:01] (03CR) 10Hashar: [C: 03+2] "Another noop in jjb diff :)" [integration/config] - 10https://gerrit.wikimedia.org/r/893786 (owner: 10Hashar) [09:19:08] (03Merged) 10jenkins-bot: Upgrade JJB from 4.2.0 to 4.3.0 [integration/config] - 10https://gerrit.wikimedia.org/r/893786 (owner: 10Hashar) [10:37:11] 10Phabricator, 10Release-Engineering-Team (Blocking 🧱), 10Security-Team, 10User-AKlapper, 10user-sbassett: Establish a workflow that scales for requesting Phab 2FA resets - https://phabricator.wikimedia.org/T306708 (10Aklapper) @sbassett: Thanks for the update! @Bmueller: Please share how to best proceed... [10:40:13] (03PS1) 10Hashar: jjb: rename Quibble fullrun job [integration/config] - 10https://gerrit.wikimedia.org/r/895142 [10:40:15] (03PS1) 10Hashar: jjb: use a job template for Quibble fullrun jobs [integration/config] - 10https://gerrit.wikimedia.org/r/895143 [11:04:04] hashar: always the simple things :) [11:05:39] jbond: that got praised by our team :] [11:08:16] :D [11:12:31] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Radar), 10SRE, 10Patch-For-Review: git: detected dubious ownership in repository at '/srv/mediawiki-staging' - https://phabricator.wikimedia.org/T325128 (10hashar) >>! In T325128#8667625, @MatthewVernon wrote: > @hashar are there still things that... [12:22:23] 10Phabricator, 10DBA: Switchover m3 master db1101 -> db1159 - https://phabricator.wikimedia.org/T331387 (10Marostegui) [13:26:11] 10GitLab (Infrastructure), 10serviceops-collab: Create cumin host in devtools project - https://phabricator.wikimedia.org/T331296 (10Jelto) Adding @Volans to get some feedback about running cookbooks in WMCS. I'm not sure if creating a dedicated cumin master helps us running cookbooks in WMCS. Is it possible... [13:26:26] 10Phabricator, 10serviceops-collab: create aphlict2001 (Phabricator realtime notifications codfw) - https://phabricator.wikimedia.org/T322369 (10eoghan) I split out the phabricator config into a separate module from the main phabricator module in https://gerrit.wikimedia.org/r/c/operations/puppet/+/891841 - ne... [13:35:59] 10GitLab (Infrastructure), 10serviceops-collab: Create cumin host in devtools project - https://phabricator.wikimedia.org/T331296 (10Volans) @Jelto the TL;DR is that there is no real support for Spicerack and cookbooks in WMCS for the use case you're looking for simply because there are too many differences be... [14:40:53] 10GitLab (Infrastructure), 10serviceops-collab: Define future design of GitLab backups - https://phabricator.wikimedia.org/T330172 (10Jelto) The above estimates consider just the current GitLab usage. At some point we may migrate all projects from gerrit to GitLab. The sizes on `gerrit1001` are: Repositories... [14:52:44] 10Release-Engineering-Team (GitLab V: Event Horizon πŸŒ„): Run docker-gc on deploy servers - https://phabricator.wikimedia.org/T329678 (10jnuche) 05Openβ†’03In progress [15:30:43] 10GitLab (Infrastructure), 10serviceops-collab, 10Patch-For-Review: Add safeguard flag to gitlab-restore.sh script - https://phabricator.wikimedia.org/T331295 (10Jelto) p:05Triageβ†’03Medium a:03Jelto [15:49:02] 10GitLab (Infrastructure), 10serviceops-collab: Create cumin host in devtools project - https://phabricator.wikimedia.org/T331296 (10Jelto) >>! In T331296#8672432, @Volans wrote: > @Jelto the TL;DR is that there is no real support for Spicerack and cookbooks in WMCS for the use case you're looking for simply b... [15:56:50] 10Deployments, 10Release-Engineering-Team: Deployment server permissions are broken preventing MediaWiki deployment - https://phabricator.wikimedia.org/T331378 (10hnowlan) There are UID differences between the two hosts: ` hnowlan@deploy1002:~$ id trebuchet uid=497(trebuchet) gid=498(trebuchet) groups=498(tre... [16:05:08] 10Project-Admins: Creation of a new project "All-and-every-Wikibooks" - https://phabricator.wikimedia.org/T330600 (10Aklapper) Could you clarify who else (apart from you) across Wikibooks communities plans to actively use (and maintain) this project tag? I'm still torn on this, obviously... [16:20:39] (03PS1) 10Krinkle: zuul: Enable CI jobs for labs/tools/intuition [integration/config] - 10https://gerrit.wikimedia.org/r/895321 [16:21:33] (03CR) 10Krinkle: [C: 03+2] zuul: Enable CI jobs for labs/tools/intuition [integration/config] - 10https://gerrit.wikimedia.org/r/895321 (owner: 10Krinkle) [16:23:22] (03Merged) 10jenkins-bot: zuul: Enable CI jobs for labs/tools/intuition [integration/config] - 10https://gerrit.wikimedia.org/r/895321 (owner: 10Krinkle) [16:25:06] (03PS1) 10Krinkle: zuul: Enable code coverage for intuition.git [integration/config] - 10https://gerrit.wikimedia.org/r/895322 [16:25:19] (03CR) 10Krinkle: [C: 03+2] zuul: Enable code coverage for intuition.git [integration/config] - 10https://gerrit.wikimedia.org/r/895322 (owner: 10Krinkle) [16:26:43] (03Merged) 10jenkins-bot: zuul: Enable code coverage for intuition.git [integration/config] - 10https://gerrit.wikimedia.org/r/895322 (owner: 10Krinkle) [16:27:26] 10Deployments, 10Release-Engineering-Team: Deployment server permissions are broken preventing MediaWiki deployment - https://phabricator.wikimedia.org/T331378 (10hashar) Nice. `trebuchet` is the previous deployment system which has been entirely superseded by scap more than a few years ago. I guess we kept th... [16:34:59] 10Deployments, 10Release-Engineering-Team: Deployment server permissions are broken preventing MediaWiki deployment - https://phabricator.wikimedia.org/T331378 (10hashar) I have been confused by the group mismatching and the files permission/umask. Here is a summary of my investigation. Humans have files cr... [16:35:38] 10Project-Admins: Creation of a new project "All-and-every-Wikibooks" - https://phabricator.wikimedia.org/T330600 (10JackPotte) Hello, actually I had posted this idea on: - https://en.wikibooks.org/wiki/Wikibooks:Reading_room/Proposals#Relation_with_Phabricator - https://fr.wikibooks.org/wiki/Wikilivres:Le_Bist... [16:35:49] 10Phabricator, 10wikitech.wikimedia.org: Remove Phabricator 2FA from account DMartin-WMF - https://phabricator.wikimedia.org/T331170 (10DMartin-WMF) My thanks, also - @Aklapper and all! [16:52:01] 10GitLab (Infrastructure), 10serviceops-collab: Create cumin host in devtools project - https://phabricator.wikimedia.org/T331296 (10Jelto) 05Openβ†’03Declined I deleted the new host `cumin-master-1001.devtools.eqiad1.wikimedia.cloud`. As discussed above we can not use cookbooks in WMCS currently. So I'm cl... [17:06:56] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10MW-on-K8s, 10serviceops, 10Patch-For-Review: Build MediaWiki images for kubernetes on the deployment servers - https://phabricator.wikimedia.org/T297673 (10thcipriani) 05In progressβ†’03Resolved I believe this is happening now. Just noticed that this tas... [17:21:09] (03PS1) 10Pwangai: Zuul: [mediawiki/extensions/Gadgets] Enable Sonar Codehealth [integration/config] - 10https://gerrit.wikimedia.org/r/895329 (https://phabricator.wikimedia.org/T321837) [17:34:28] (03PS1) 10Pwangai: Zuul: [mediawiki/extensions/Wikistories] Enable Sonar Codehealth [integration/config] - 10https://gerrit.wikimedia.org/r/895333 (https://phabricator.wikimedia.org/T321837) [17:48:38] !log (deployment-prep) `samtar@deployment-mwmaint02:~$ mwscript maintenance/refreshLinks.php --wiki enwiki --category='Pages that use Phonos'` for T326163 [17:48:42] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:48:43] T326163: Add page properties for Phonos usage data - https://phabricator.wikimedia.org/T326163 [17:52:04] !log (deployment-prep) Ctrl+C'd `mwscript maintenance/refreshLinks.php --wiki enwiki --category='Pages that use Phonos'`, taking "a long time", saw `GlobalVarConfig::get: undefined option: 'PhonosStoreFilesAsMp3'`, T326163 [17:52:08] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:52:44] (03CR) 10Dduvall: [C: 03+2] Allow more BuildKit frontend image names [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/893028 (https://phabricator.wikimedia.org/T329553) (owner: 10Lucas Werkmeister (WMDE)) [17:52:54] (03CR) 10Dduvall: [C: 03+2] "Thanks for this!" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/893028 (https://phabricator.wikimedia.org/T329553) (owner: 10Lucas Werkmeister (WMDE)) [17:54:04] (03Merged) 10jenkins-bot: Allow more BuildKit frontend image names [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/893028 (https://phabricator.wikimedia.org/T329553) (owner: 10Lucas Werkmeister (WMDE)) [17:54:10] \o/ [17:55:42] be nice if `maintenance/refreshLinks.php` could be run `--verbose`ly [18:21:36] 10Deployments, 10Release-Engineering-Team: Deployment server permissions are broken preventing MediaWiki deployment - https://phabricator.wikimedia.org/T331378 (10dancy) Here is my assessment of the problem: ` error: insufficient permission for adding an object to repository database .git/objects error: insuff... [18:28:34] 10Deployments, 10Release-Engineering-Team: Deployment server permissions are broken preventing MediaWiki deployment - https://phabricator.wikimedia.org/T331378 (10dancy) >>! In T331378#8671563, @hashar wrote: > find /srv/mediawiki-staging/.git -type d -not -perm g+s > > I am guessing we should normalize a... [18:37:22] (03PS1) 10Krinkle: zuul: Enable CI jobs for intuition-web [integration/config] - 10https://gerrit.wikimedia.org/r/895343 [18:37:35] (03CR) 10Krinkle: [C: 03+2] zuul: Enable CI jobs for intuition-web [integration/config] - 10https://gerrit.wikimedia.org/r/895343 (owner: 10Krinkle) [18:38:54] (03Merged) 10jenkins-bot: zuul: Enable CI jobs for intuition-web [integration/config] - 10https://gerrit.wikimedia.org/r/895343 (owner: 10Krinkle) [18:48:32] 10Deployments, 10Release-Engineering-Team: Deployment server permissions are broken preventing MediaWiki deployment - https://phabricator.wikimedia.org/T331378 (10dancy) I changed the permissions/group accordingly. [19:05:25] TheresNoTime: patches welcome ;) [19:08:19] * TheresNoTime grumbles [19:53:01] bd808: want to pay the "patches welcome" tax and add https://gerrit.wikimedia.org/r/c/mediawiki/core/+/895348 to your list? :D /half-joking.. [20:05:36] :D [20:09:44] TheresNoTime: +2 given [20:09:49] thank you :D [20:11:05] I wonder what the average number of days between me giving a +2 in mediawiki/core is? I imagine it is trending quite large since I ran away to hide in Python code. [20:12:49] poor you :> [20:13:03] (Python has grown on me a *lot* tbf) [21:40:13] 10Deployments, 10Release-Engineering-Team: Deployment server permissions are broken preventing MediaWiki deployment - https://phabricator.wikimedia.org/T331378 (10dancy) p:05Unbreak!β†’03Medium [23:35:34] (03PS1) 10Krinkle: zuul: Switch intuition-web node16 job to node16-browser-docker [integration/config] - 10https://gerrit.wikimedia.org/r/895371 [23:56:04] (03CR) 10Krinkle: [C: 03+2] zuul: Switch intuition-web node16 job to node16-browser-docker [integration/config] - 10https://gerrit.wikimedia.org/r/895371 (owner: 10Krinkle) [23:57:14] (03Merged) 10jenkins-bot: zuul: Switch intuition-web node16 job to node16-browser-docker [integration/config] - 10https://gerrit.wikimedia.org/r/895371 (owner: 10Krinkle)