[06:26:28] 10Phabricator, 10Security: Enable 2FA for your Phabricator account with access to Security tasks (July 2022) - https://phabricator.wikimedia.org/T313378 (10Aklapper) [09:42:43] 10GitLab (Project Migration), 10serviceops: import subversion repos from Phabricator into Gitlab - https://phabricator.wikimedia.org/T308061 (10Aklapper) (Whoever recently created the saved Diffusion query called "svn-repos" at https://phabricator.wikimedia.org/diffusion/query/6WUBLfM9eS2R/ , it was the "defau... [11:02:32] 10GitLab (Project Migration), 10Release-Engineering-Team: Create new GitLab project group: structured-data - https://phabricator.wikimedia.org/T312037 (10mfossati) @thcipriani , would it be possible to expedite this? As I'm working on {T311745}, I'd like to enable continuous integration on the code repository,... [11:09:05] 10GitLab (Project Migration), 10serviceops: import subversion repos from Phabricator into Gitlab - https://phabricator.wikimedia.org/T308061 (10hashar) Subversion got removed from Phabricator by mistake in https://gerrit.wikimedia.org/r/c/operations/puppet/+/785149 which has lead to T307889 and now this task t... [11:47:34] 10Phabricator, 10Release-Engineering-Team (The Decommission Mission πŸ’€), 10serviceops, 10serviceops-collab: Setup rsync for phab data on disk - https://phabricator.wikimedia.org/T313360 (10LSobanski) @thcipriani if that particular bash history is critical, should we persist it somewhere more resilient? [11:55:44] Project beta-scap-sync-world build #60504: 04FAILURE in 54 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/60504/ [12:05:47] Project beta-scap-sync-world build #60505: 04STILL FAILING in 54 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/60505/ [12:16:11] Yippee, build fixed! [12:16:11] Project beta-scap-sync-world build #60506: 09FIXED in 1 min 14 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/60506/ [12:19:40] 10GitLab (Project Migration), 10serviceops: import subversion repos from Phabricator into Gitlab - https://phabricator.wikimedia.org/T308061 (10Aklapper) I propose to decline this task, and not to care about these SVN repos. Not every ancient code base must be archived //by us// forever.`{{Citation needed}}`... [13:07:51] 10GitLab (CI & Job Runners), 10serviceops, 10serviceops-collab: DNS/networking not working on Trusted Runners - https://phabricator.wikimedia.org/T311241 (10Jelto) DNS issues on Trusted Runners should be fixed now. Mocked/test Trusted Runners in WMCS devtools still have DNS issues. When buildkitd was added... [13:15:40] (03PS2) 10Jforrester: Zuul: [mediawiki/extensions/VueTest] Add extension-codehealth pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/811368 (owner: 10Kosta Harlan) [13:15:45] (03PS3) 10Jforrester: Zuul: [mediawiki/extensions/VueTest] Add extension-codehealth pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/811368 (owner: 10Kosta Harlan) [13:15:48] (03CR) 10Jforrester: [C: 03+2] Zuul: [mediawiki/extensions/VueTest] Add extension-codehealth pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/811368 (owner: 10Kosta Harlan) [13:18:44] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/VueTest] Add extension-codehealth pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/811368 (owner: 10Kosta Harlan) [13:19:29] !log Zuul: [mediawiki/extensions/VueTest] Add extension-codehealth pipeline [13:19:31] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:22:17] jnuche: I have some late night meeting tonight so I guess I can sync up with daniel [13:22:27] maybe we can migrate the replica tonight or tomorrow [13:22:48] hashar: that'd be awesome [14:12:18] 10Beta-Cluster-Infrastructure, 10Abstract Wikipedia team (Phase ΞΈ – Throttling), 10Patch-For-Review: function-orchestrator down in Beta Cluster due to a disk full of docker images - https://phabricator.wikimedia.org/T313334 (10ori) 05Openβ†’03Resolved a:03ori After running `docker image prune`: ` $ df -... [14:45:38] 10Release-Engineering-Team, 10Scap, 10Patch-For-Review: Scap pool counter error while backporting - https://phabricator.wikimedia.org/T310835 (10Joe) >>! In T310835#8085147, @dancy wrote: > I looked through operations/puppet/modules/conftool/files/safe-service-restart.py and I don't see any code for falling... [14:47:18] 10GitLab (Project Migration), 10Release-Engineering-Team: Create new GitLab project group: structured-data - https://phabricator.wikimedia.org/T312037 (10brennen) @mfossati - sorry this slipped through the cracks, will add momentarily. [14:54:32] 10Release-Engineering-Team, 10Wikimedia-GitHub: Add Matthew Vernon (@mcv21) to Wikimedia github - https://phabricator.wikimedia.org/T288350 (10thcipriani) 05Openβ†’03Resolved [14:57:51] 10GitLab (Project Migration), 10Release-Engineering-Team: Create new GitLab project group: structured-data - https://phabricator.wikimedia.org/T312037 (10brennen) 05Openβ†’03Resolved a:03brennen Created https://gitlab.wikimedia.org/repos/structured-data Added @mfossati and cparle as owners; other users wi... [15:06:28] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Radar), 10Discovery, 10Discovery-Search (Current work), 10Patch-For-Review: Deploy new bullseye elastic cluster nodes on deployment-prep - https://phabricator.wikimedia.org/T299797 (10Gehel) [15:06:31] 10Scap: scap sync-file message during php-fpm-restarts about restart-php-fmp-all being called with an empty host list - https://phabricator.wikimedia.org/T313417 (10Lucas_Werkmeister_WMDE) [15:13:51] 10Scap: scap sync-file message during php-fpm-restarts about restart-php-fmp-all being called with an empty host list - https://phabricator.wikimedia.org/T313417 (10dancy) a:03dancy It is safe to ignore that message. I will work on eliminating it. [15:14:21] 10Scap: scap sync-file message during php-fpm-restarts about restart-php-fmp-all being called with an empty host list - https://phabricator.wikimedia.org/T313417 (10dancy) [15:14:47] 10Scap: scap sync-file message during php-fpm-restarts about restart-php-fmp-all being called with an empty host list - https://phabricator.wikimedia.org/T313417 (10Lucas_Werkmeister_WMDE) Good to hear, thanks! :) [15:27:23] 10Phabricator, 10Release-Engineering-Team (The Decommission Mission πŸ’€), 10serviceops, 10serviceops-collab: Setup rsync for phab data on disk - https://phabricator.wikimedia.org/T313360 (10thcipriani) >>! In T313360#8090919, @LSobanski wrote: > @thcipriani if that particular bash history is critical, should... [15:31:06] (03PS1) 10Ahmon Dancy: _restart_php: Exclude empty host lists [tools/scap] - 10https://gerrit.wikimedia.org/r/815756 (https://phabricator.wikimedia.org/T313417) [15:37:57] (03CR) 10Ahmon Dancy: [C: 03+2] _restart_php: Exclude empty host lists [tools/scap] - 10https://gerrit.wikimedia.org/r/815756 (https://phabricator.wikimedia.org/T313417) (owner: 10Ahmon Dancy) [15:42:28] (03Merged) 10jenkins-bot: _restart_php: Exclude empty host lists [tools/scap] - 10https://gerrit.wikimedia.org/r/815756 (https://phabricator.wikimedia.org/T313417) (owner: 10Ahmon Dancy) [15:43:57] !log Upgrading scap to 4.11.1-1+0~20220720154238.348~1.gbp94de82 in beta cluster [15:43:58] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:47:03] (03PS1) 10Ahmon Dancy: Release 4.11.2-1 [tools/scap] - 10https://gerrit.wikimedia.org/r/815758 [15:47:05] (03CR) 10Ahmon Dancy: [C: 03+2] Release 4.11.2-1 [tools/scap] - 10https://gerrit.wikimedia.org/r/815758 (owner: 10Ahmon Dancy) [15:55:33] (03Merged) 10jenkins-bot: Release 4.11.2-1 [tools/scap] - 10https://gerrit.wikimedia.org/r/815758 (owner: 10Ahmon Dancy) [15:57:20] 10Gerrit, 10Release-Engineering-Team (The Decommission Mission πŸ’€), 10Patch-For-Review: Bring up Gerrit2002 - https://phabricator.wikimedia.org/T313250 (10hashar) >>! In T313250#8089675, @Dzahn wrote: >>>! In T313250#8089191, @hashar wrote: >> we can switch the DNS entry for gerrit-replica.wikimedia.org from... [15:59:04] 10Scap, 10Patch-For-Review: scap sync-file message during php-fpm-restarts about restart-php-fmp-all being called with an empty host list - https://phabricator.wikimedia.org/T313417 (10dancy) 05Openβ†’03Resolved Scap 4.11.2 has been deployed with the fix. [16:09:29] 10Scap, 10Patch-For-Review: scap sync-file message during php-fpm-restarts about restart-php-fpm-all being called with an empty host list - https://phabricator.wikimedia.org/T313417 (10dancy) [16:09:59] 10Gerrit, 10Release-Engineering-Team (The Decommission Mission πŸ’€), 10Patch-For-Review: Bring up Gerrit2002 - https://phabricator.wikimedia.org/T313250 (10hashar) Ahmon has a nice improvement suggestion for ssh_known_hosts at https://gerrit.wikimedia.org/r/c/operations/puppet/+/815400/ which I have replied to... [16:14:50] 10Gerrit, 10Release-Engineering-Team (The Decommission Mission πŸ’€), 10Patch-For-Review: Bring up Gerrit2002 - https://phabricator.wikimedia.org/T313250 (10hashar) Some deployment notes which can probably be moved to the task description: Once Gerrit is up on the new host and Prometheus has polled it, it shou... [16:42:02] 10GitLab (CI & Job Runners), 10serviceops, 10serviceops-collab: DNS/networking not working on Trusted Runners - https://phabricator.wikimedia.org/T311241 (10dduvall) >>! In T311241#8086793, @Jelto wrote: >>>! In T311241#8065350, @Jelto wrote: >> [...] >> Would it be possible to use the docker default network... [16:48:30] 10GitLab (CI & Job Runners), 10Release-Engineering-Team, 10serviceops, 10serviceops-collab, 10User-dduvall: Changes to modules/gitlab_runner/templates/config-template.toml.erb have no effect on existing runners - https://phabricator.wikimedia.org/T311746 (10brennen) [17:17:08] 10GitLab (Project Migration), 10Release-Engineering-Team (The Decommission Mission πŸ’€), 10Striker, 10Tools: Figure out workflow for programatically adding GitLab users - https://phabricator.wikimedia.org/T313366 (10thcipriani) [17:19:26] 10GitLab (Project Migration), 10Release-Engineering-Team (The Decommission Mission πŸ’€), 10Striker, 10Tools: Figure out workflow for programatically adding GitLab users - https://phabricator.wikimedia.org/T313366 (10thcipriani) [17:28:29] 10Phabricator, 10Release-Engineering-Team (The Decommission Mission πŸ’€), 10serviceops, 10serviceops-collab: Setup rsync for phab data on disk - https://phabricator.wikimedia.org/T313360 (10hashar) Earlier this week I have mentioned `rsync` issue about username/group names being mixed up. That caused us a lo... [17:29:56] 10Phabricator, 10Release-Engineering-Team (The Decommission Mission πŸ’€), 10serviceops, 10serviceops-collab: Setup rsync for phab data on disk - https://phabricator.wikimedia.org/T313360 (10thcipriani) [17:34:06] 10Phabricator, 10Release-Engineering-Team (The Decommission Mission πŸ’€), 10serviceops, 10serviceops-collab: Email tool maintainers about git-ssh deprecation on phabricator - https://phabricator.wikimedia.org/T313359 (10thcipriani) >>! In T313359#8089535, @bd808 wrote: >>>! In T313359#8089476, @thcipriani wr... [17:36:35] 10Phabricator, 10Release-Engineering-Team (The Decommission Mission πŸ’€), 10serviceops, 10serviceops-collab: Email tool maintainers about git-ssh deprecation on phabricator - https://phabricator.wikimedia.org/T313359 (10hashar) For git over http one can issue a 301 redirect and avoid breaking the world. For... [17:40:39] 10Phabricator, 10Release-Engineering-Team (The Decommission Mission πŸ’€), 10User-brennen: Deploy Phabricator with scap - https://phabricator.wikimedia.org/T313259 (10brennen) a:03brennen [17:44:50] 10GitLab (Project Migration), 10serviceops: import subversion repos from Phabricator into Gitlab - https://phabricator.wikimedia.org/T308061 (10demon) I don't think we need the Subversion data itself -- definitely don't need to go through the hassle of porting it all over to Gitlab. I proposed we just tar+gzip... [17:54:58] 10Gerrit, 10Release-Engineering-Team (The Decommission Mission πŸ’€), 10Patch-For-Review: Bring up Gerrit2002 - https://phabricator.wikimedia.org/T313250 (10thcipriani) [18:36:57] 10Phabricator, 10Release-Engineering-Team (The Decommission Mission πŸ’€), 10serviceops, 10serviceops-collab: Email tool maintainers about git-ssh deprecation on phabricator - https://phabricator.wikimedia.org/T313359 (10Aklapper) > do you have thoughts about how easy this query would be? Čüş, not really...... [18:41:19] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10Patch-For-Review, 10Release, 10Train Deployments: 1.39.0-wmf.21 deployment blockers - https://phabricator.wikimedia.org/T308074 (10jeena) [18:48:58] (03Abandoned) 10Ebernhardson: Stop weekly release branching ApiFeatureUsage [tools/release] - 10https://gerrit.wikimedia.org/r/814872 (https://phabricator.wikimedia.org/T313248) (owner: 10Ebernhardson) [19:42:06] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10Patch-For-Review, 10Release, 10Train Deployments: 1.39.0-wmf.21 deployment blockers - https://phabricator.wikimedia.org/T308074 (10taavi) [20:03:25] 10GitLab (Project Migration), 10Striker, 10Tools, 10Goal: Replace Diffusion integration with Gitlab integration in Striker (toolsadmin) - https://phabricator.wikimedia.org/T296893 (10bd808) p:05Triageβ†’03High [20:31:17] /awya [20:31:25] meant /away [20:32:46] 10Release-Engineering-Team (Radar), 10Scap, 10Patch-For-Review, 10User-jijiki: Update Scap to perform rolling restart for all MW deploy - https://phabricator.wikimedia.org/T266055 (10dancy) [20:33:05] 10Scap, 10Performance-Team, 10serviceops: MW wmf-config tmp cache stays outdated after Scap deploy (opcache revalidation is off) - https://phabricator.wikimedia.org/T311788 (10dancy) 05Resolvedβ†’03Open The same situation happened today during a deployment, so we're not out of the woods yet. [21:04:34] 10GitLab (Project Migration), 10serviceops: import subversion repos from Phabricator into Gitlab - https://phabricator.wikimedia.org/T308061 (10Dzahn) I would also say we can make a .tar.gz and then I would suggest to put it here, next to these: https://dumps.wikimedia.org/other/misc/ .oh..look..there is alr... [21:15:50] 10Scap, 10Performance-Team, 10serviceops: MW wmf-config tmp cache stays outdated after Scap deploy (opcache revalidation is off) - https://phabricator.wikimedia.org/T311788 (10dancy) Clarifying for the record: Opcache revalidation is turned off by setting `opcache.validate_timestamps` to 0. `opcache.revali... [21:48:46] 10GitLab (Project Migration), 10serviceops: import subversion repos from Phabricator into Gitlab - https://phabricator.wikimedia.org/T308061 (10hashar) There are some dumps yes. For toolserver I previously traced the history to T60801. My point above was that `subversion` was erroneously removed and I would... [22:13:36] 10GitLab (Project Migration), 10serviceops: import subversion repos from Phabricator into Gitlab - https://phabricator.wikimedia.org/T308061 (10RhinosF1) The removal wasn't an accident or erroneous. It just didn't quite consider that some docs for old config variables and stuff doesn't point to SVN. I'm not q... [22:38:00] 10Phabricator, 10Release-Engineering-Team (The Decommission Mission πŸ’€), 10serviceops, 10serviceops-collab: Email tool maintainers about git-ssh deprecation on phabricator - https://phabricator.wikimedia.org/T313359 (10Dzahn) >>! In T313359#8092203, @Aklapper wrote: > Čüş https://en.wiktionary.org/wiki/%C4... [23:34:10] 10Phabricator, 10Release-Engineering-Team (The Decommission Mission πŸ’€), 10serviceops, 10serviceops-collab: Email tool maintainers about git-ssh deprecation on phabricator - https://phabricator.wikimedia.org/T313359 (10Dzahn) When I used the "metrics_user" I only got a limited set of phabricator databases b... [23:41:55] 10Phabricator, 10Release-Engineering-Team (The Decommission Mission πŸ’€), 10serviceops, 10serviceops-collab: Email tool maintainers about git-ssh deprecation on phabricator - https://phabricator.wikimedia.org/T313359 (10Dzahn) Now in this table there are also "viewPolicy", "editPolicy" and "pushPolicy" field... [23:45:34] is there actually a Phab deployment happening? [23:48:16] mutante: nope [23:48:22] maybe i should just scrap that event [23:48:37] causes more confusion than anything, and we can add to the calendar when necessary. [23:50:07] i just used the pattern mukunda & i had going where we basically ignored the event except when we didn't, but on reflection there's probably no need to keep it on peoples' calendars. [23:54:18] 10Phabricator, 10Release-Engineering-Team (The Decommission Mission πŸ’€), 10serviceops, 10serviceops-collab: Email tool maintainers about git-ssh deprecation on phabricator - https://phabricator.wikimedia.org/T313359 (10Dzahn) total git repos starting with tool- : 445 repos without commits, starting with t...