[01:21:49] 10GitLab: Updating Wikimedia Developer Account email does not propagate to Gitlab - https://phabricator.wikimedia.org/T323880 (10Soda) [05:20:04] 10Phabricator, 10Release-Engineering-Team (Bonus Level 🕹️), 10serviceops, 10serviceops-collab, 10Patch-For-Review: Deprecate git-ssh service on phabricator.wikimedia.org - https://phabricator.wikimedia.org/T296022 (10abi_) [05:21:31] 10Phabricator, 10Release-Engineering-Team, 10translatewiki.net, 10Language-Team (Language-2022-October-December), and 2 others: Phabricator translations can no more be received due to decommissioning of git-ssh - https://phabricator.wikimedia.org/T321350 (10abi_) 05Open→03In progress a:03hashar [08:22:41] Question for scap backport. If I deploy some config patch using scap backport and found issue once it is synced to canaries. Do I need to just stop there or need to do scap backport --revert? [08:23:14] urbanecm and Amir1 ^^ (as you deploy so frequently!) [08:25:24] kart_: I know we had the issue that stopping it didn't clean it from canaries kart_ [08:25:36] Don't know if that's fixed now [08:26:21] OK. I did scap backport --revert -- which seems OK. [08:26:41] Yeah. Better safe than sorry [08:26:51] Thanks! [08:27:04] PS: There are undeployed changes in mediawiki-config. [08:27:05] 10Continuous-Integration-Config, 10Release-Engineering-Team (Priority Backlog 📥), 10MediaWiki-Core-Tests, 10Code-Health, and 8 others: Reduce runtime of MW shared gate Jenkins jobs to 5 min - https://phabricator.wikimedia.org/T225730 (10hashar) >>! In T225730#8370890, @Krinkle wrote: > @hashar Afaik for PH... [08:32:50] jnuche: I am still struggling with (docker|podman) cgroup(v1|v2) (!root|root) [08:33:02] but I might have made some progress :D [08:49:12] 10Continuous-Integration-Config, 10Release-Engineering-Team (Priority Backlog 📥), 10MediaWiki-Core-Tests, 10Code-Health, and 8 others: Reduce runtime of MW shared gate Jenkins jobs to 5 min - https://phabricator.wikimedia.org/T225730 (10Krinkle) >>! In T225730#8423575, @hashar wrote: >>>>! In T225730#83703... [08:57:13] hashar: let me know if I can help with anything! [09:06:07] it is a mess :D [09:06:37] jnuche: I can't have a network attached running podman rootless (aka with my own user) [09:06:52] apparently that is in podman 3.2 but Debian ships 3.0 ;) [09:07:21] 10Phabricator, 10SRE, 10Traffic, 10Wikimedia-Incident: Phabricator was logging out users repeatedly (2022-08-26) - https://phabricator.wikimedia.org/T316337 (10Marostegui) What should we do with this task? Anything left? [09:07:22] and I have managed to completely break my Docker setup somehow [09:12:46] hashar: ah damn, did you try in the end the systemd workaround to use v1 cgroups? [09:13:09] nop [09:13:26] I guess I am taking that as an excuse to finally switch entirely to podman/buildah [09:13:31] I am not that far ;) [09:18:42] 10Phabricator, 10SRE, 10Traffic, 10Wikimedia-Incident: Phabricator was logging out users repeatedly (2022-08-26) - https://phabricator.wikimedia.org/T316337 (10jcrespo) As soon as I finish the wikitech description I intend to resolve it. [09:40:22] 10Phabricator, 10SRE, 10Traffic, 10Wikimedia-Incident: Phabricator was logging out users repeatedly (2022-08-26) - https://phabricator.wikimedia.org/T316337 (10jcrespo) [09:40:48] 10Phabricator, 10SRE, 10Traffic, 10Wikimedia-Incident: Phabricator was logging out users repeatedly (2022-08-26) - https://phabricator.wikimedia.org/T316337 (10jcrespo) 05Open→03Resolved a:03Vgutierrez @hashar @Vgutierrez Please review my summary of the incident at: https://wikitech.wikimedia.org/wik... [12:30:29] 10Release-Engineering-Team (Seen), 10Wikidata, 10wdwb-tech, 10MW-1.35-notes (1.35.0-wmf.16; 2020-01-21), 10Technical-Debt: Stop using jsonlint and instead use eslint-plugin-json for the linting - https://phabricator.wikimedia.org/T220036 (10Volker_E) [12:57:32] 10Release-Engineering-Team (GitLab III: GitLab in LA 🪃): CasC configuration for Releasing Jenkins instance - https://phabricator.wikimedia.org/T323909 (10jnuche) 05Open→03In progress [12:58:03] 10Release-Engineering-Team (GitLab III: GitLab in LA 🪃): CasC configuration for Releasing Jenkins instance - https://phabricator.wikimedia.org/T323909 (10jnuche) [12:58:05] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team (GitLab III: GitLab in LA 🪃): Automate integration Jenkins deployment and config changes - https://phabricator.wikimedia.org/T319406 (10jnuche) [13:07:42] 10Continuous-Integration-Config, 10WikiLambda, 10Abstract Wikipedia team (Phase κ – Clean-up), 10MW-1.39-notes (1.39.0-wmf.22; 2022-07-25), and 2 others: Run some full end-to-end browser integration tests on Beta Cluster Wikifunctions now it exists - https://phabricator.wikimedia.org/T294388 (10zeljkofilipi... [13:30:08] 10Continuous-Integration-Infrastructure, 10Quibble, 10Developer Productivity: Provide early feedback when a patch has job failures - https://phabricator.wikimedia.org/T323750 (10kostajh) >>! In T323750#8419328, @hashar wrote: > Ideally Zuul would report immediately when it knows that a change is not going to... [13:31:32] 10Continuous-Integration-Config, 10GrowthExperiments, 10Quibble, 10Growth-Team (Current Sprint), and 3 others: api-testing tests break for GrowthExperiments - https://phabricator.wikimedia.org/T320935 (10kostajh) 05Open→03Resolved >>! In T320935#8418795, @EUdoh-WMF wrote: > I see it, thanks a lot. The... [15:48:16] 10Project-Admins, 10Abstract Wikipedia team, 10WikiLambda: Rename "WikiLambda" and "WikiLambda Front-End" Phabricator tags to reflect the new project name: WikiFunction - https://phabricator.wikimedia.org/T323759 (10Jdforrester-WMF) Having an additional project for Wikifunctions sounds reasonable, but it sho... [15:54:54] (03PS1) 10QChris: Allow “Gerrit Managers” to import history [extensions/PagePermissions] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/861407 [15:54:56] (03CR) 10QChris: [V: 03+2 C: 03+2] Allow “Gerrit Managers” to import history [extensions/PagePermissions] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/861407 (owner: 10QChris) [15:54:58] (03PS1) 10QChris: Import done. Revoke import grants [extensions/PagePermissions] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/861408 [15:55:00] (03CR) 10QChris: [V: 03+2 C: 03+2] Import done. Revoke import grants [extensions/PagePermissions] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/861408 (owner: 10QChris) [15:59:14] (03PS1) 10QChris: Allow “Gerrit Managers” to import history [extensions/WatchAnalytics] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/861414 [15:59:16] (03CR) 10QChris: [V: 03+2 C: 03+2] Allow “Gerrit Managers” to import history [extensions/WatchAnalytics] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/861414 (owner: 10QChris) [15:59:25] (03PS1) 10QChris: Import done. Revoke import grants [extensions/WatchAnalytics] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/861415 [15:59:27] (03CR) 10QChris: [V: 03+2 C: 03+2] Import done. Revoke import grants [extensions/WatchAnalytics] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/861415 (owner: 10QChris) [16:05:52] 10Beta-Cluster-Infrastructure, 10Abstract Wikipedia team, 10Wikimedia-Site-requests: Update favicon and CentralAuthLoginIcon for wikifunctionswiki - https://phabricator.wikimedia.org/T323627 (10Jdforrester-WMF) Thank you! [16:14:43] 10Phabricator, 10decommission-hardware, 10serviceops-collab, 10Patch-For-Review: decommission phab1001.eqiad.wmnet - https://phabricator.wikimedia.org/T323418 (10LSobanski) 05Stalled→03Open [16:16:14] 10Continuous-Integration-Config, 10Abstract Wikipedia team: Explore impact of adding WikiLambda to the general MediaWiki CI gate - https://phabricator.wikimedia.org/T289741 (10Jdforrester-WMF) [16:16:51] (03PS1) 10Jforrester: Zuul: [mediawiki/extensions/WikiLambda] Add to the extension gate [integration/config] - 10https://gerrit.wikimedia.org/r/861422 (https://phabricator.wikimedia.org/T289741) [16:18:39] (03Abandoned) 10Anne Tomasevich: [WIP] Add redirect from /codex to /codex/latest [integration/docroot] - 10https://gerrit.wikimedia.org/r/859130 (https://phabricator.wikimedia.org/T311097) (owner: 10Anne Tomasevich) [16:20:40] 10GitLab (Infrastructure), 10serviceops-collab: Evaluate a high available GitLab architecture - https://phabricator.wikimedia.org/T323201 (10LSobanski) p:05Triage→03Medium [16:21:17] 10Phabricator, 10serviceops-collab: make Phabricator work with readonly DB - https://phabricator.wikimedia.org/T323312 (10LSobanski) p:05Triage→03Medium [16:26:17] 10Gerrit, 10Release-Engineering-Team (GitLab III: GitLab in LA 🪃), 10SRE-OnFire, 10serviceops-collab, and 2 others: gerrit1001 running out of space on / - https://phabricator.wikimedia.org/T323262 (10LSobanski) a:03LSobanski [16:28:21] 10Project-Admins, 10Abstract Wikipedia team, 10WikiLambda: Rename "WikiLambda" and "WikiLambda Front-End" Phabricator tags to reflect the new project name: WikiFunction - https://phabricator.wikimedia.org/T323759 (10Teleosteen) @Jdforrester-WMF OK, so from your message, I think I understood something: WikiB... [16:30:14] (03CR) 10Ahmon Dancy: Makefile: drop reference to blubber (031 comment) [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/859988 (owner: 10Hashar) [16:34:53] (03CR) 10Ahmon Dancy: [C: 03+2] Use Blubber BuildKit frontend instead of blubberoid (032 comments) [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/842543 (https://phabricator.wikimedia.org/T318290) (owner: 10Dduvall) [16:49:08] (03CR) 10Ahmon Dancy: [C: 03+2] Send Gerrit review with tag autogenerated:pipelinelib [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/859989 (owner: 10Hashar) [16:55:59] 10GitLab, 10serviceops-collab: automatically check for new gitlab releases and send notifications - https://phabricator.wikimedia.org/T323932 (10Dzahn) [16:56:21] 10GitLab, 10serviceops-collab: automatically check for new gitlab releases and send notifications - https://phabricator.wikimedia.org/T323932 (10Dzahn) [17:09:23] 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Debian Stretch Deprecation): Cloud VPS "deployment-prep" project Stretch deprecation - https://phabricator.wikimedia.org/T306068 (10Jgiannelos) I prepare a new instance ` deployment-wikifeeds02 ` for wikifeeds thats based on bullseye and updated the webproxy to point... [17:09:58] 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Debian Stretch Deprecation): Cloud VPS "deployment-prep" project Stretch deprecation - https://phabricator.wikimedia.org/T306068 (10Jgiannelos) [17:16:25] 10Release-Engineering-Team, 10Scap, 10User-brennen: Prompted for sudo password during deployment - https://phabricator.wikimedia.org/T323735 (10brennen) 05Open→03Resolved a:03brennen Seems like this ought to be fixed. [18:05:14] 10Release-Engineering-Team, 10Scap: Save K8s image build logs - https://phabricator.wikimedia.org/T323939 (10jnuche) [18:06:03] 10Release-Engineering-Team, 10Scap: Save K8s image build logs - https://phabricator.wikimedia.org/T323939 (10jnuche) [18:06:05] 10Release-Engineering-Team, 10Scap, 10MW-on-K8s: Scap Mediawiki K8s deployments - https://phabricator.wikimedia.org/T318536 (10jnuche) [18:44:29] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10GitLab (CI & Job Runners), 10Jenkins, 10Math: Can one add an additional test runner in Jenkins (or the new CI)? - https://phabricator.wikimedia.org/T323942 (10Physikerwelt) [19:39:40] jnuche: I have managed to build the scap3-dev images using podman! :] [19:40:22] then after scap get installed and a deploy is done it fails cause it can not resolve the hostnames of the targets (`jenkins-ci` and `jenkins-rel` ) [19:41:38] Nice work [19:42:51] somehow it mounts/copy my host `/etc/resolv.conf` and `/etc/hosts` [19:43:37] dancy: ;) [19:43:54] thank you for the `autogenerated:pipeline` patch [19:44:37] I just pressed the button! (after cherry picking and running `make test`) of course [19:45:48] I need a linter for GNU make [19:46:37] and since I have removed docker entirely I can't test it :D [19:47:12] haha.. I'll test for you [19:47:22] When your next patchset is ready. [19:47:39] well [20:05:43] (03PS2) 10Hashar: Makefile: drop reference to blubber [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/859988 [20:05:45] (03PS3) 10Hashar: Send Gerrit commits with tag autogenerated:pipelinelib [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/859989 [20:06:00] dancy: done, there was an extra closing parenthesis [20:06:30] (03CR) 10Hashar: Makefile: drop reference to blubber (031 comment) [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/859988 (owner: 10Hashar) [20:09:05] (03CR) 10Ahmon Dancy: [C: 03+2] Makefile: drop reference to blubber (031 comment) [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/859988 (owner: 10Hashar) [20:10:07] (03Merged) 10jenkins-bot: Makefile: drop reference to blubber [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/859988 (owner: 10Hashar) [20:20:32] 10Release-Engineering-Team (GitLab III: GitLab in LA 🪃), 10Scap, 10Developer Productivity: Add support for "gerrit/r/:changenum" URLs to scap backport command - https://phabricator.wikimedia.org/T323320 (10dancy) [20:27:46] 10Release-Engineering-Team (GitLab III: GitLab in LA 🪃), 10Scap, 10Developer Productivity: Add support for "gerrit/r/:changenum" URLs to scap backport command - https://phabricator.wikimedia.org/T323320 (10dancy) a:03dancy [20:33:54] Hi there, could someone change the "Code review" link on https://phabricator.wikimedia.org/project/profile/1449? I thought scap has been moved to gitlab now [20:41:45] cirno: it's complicated. code review is still on gerrit for many things. It should not link to Phabricator Diffusion though. scap is the deployment tool, not the review tool and still in use. [20:42:02] oh, code review for changes in scap itself. sorry [20:42:13] it took me a moment to get it [20:43:41] would this be the right replacement link? [20:43:43] https://gitlab.wikimedia.org/repos/releng/scap3-dev/-/merge_requests [20:44:18] I thought it should be this one: https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests [20:44:25] https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests [20:44:34] right! ack [20:45:17] I made the edit on the Phab dashboard [20:45:18] ** Open: https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests?scope=all&state=opened [20:45:21] ** All: https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests?scope=all&state=all [20:45:36] thanks! [20:45:47] yw [20:46:17] I love moving things away from diffusion [21:07:09] I'm looking to schedule a deployment window this week (ideally 29 Nov 23:00 UTC/14:00PT) for a configuration change (https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/860946/), is anyone around who can help me with that? [21:11:25] wugapodes: is there a reason why that can't go through a normal backport window? [21:12:57] Based on my reading of https://wikitech.wikimedia.org/wiki/Deployments/Inclusion_criteria I was of the understanding that this needed its own window because it is "Adding a new extension (either cluster-wide or per wiki)". [21:15:29] Although it literally meets that rule, I suspect that ContactPage isn't really what they had in mind by that [21:15:33] hmm. no-one follows that in practice [21:17:54] I wonder if by per-wiki they meant the first time an extension is added to any wiki [21:29:05] I'll add it to a backport window then, thanks for the feedback [21:57:38] 10GitLab (Project Migration), 10Release-Engineering-Team: Create new GitLab project group: - https://phabricator.wikimedia.org/T323957 (10SDunlap) [21:57:56] 10GitLab (Project Migration), 10Release-Engineering-Team: Create new GitLab project group: abstract - https://phabricator.wikimedia.org/T323957 (10SDunlap) [22:06:11] noting here that we're doing a phabricator server migration over in #wikimedia-operations. [22:18:00] ^ thx for topic change [22:30:49] * thcipriani doffs cap [22:31:57] theobjectivedad: operations only has one n ;) [22:32:47] unless d z has gone rouge and created yet another irc channel [22:40:59] bah [22:41:08] mosh + weechat glitch [23:27:48] 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥), 10serviceops-collab, 10Patch-For-Review: move phabricator to new hardware generation - https://phabricator.wikimedia.org/T280597 (10Dzahn) phabricator switched to phab1004 [23:42:57] 10Phabricator, 10decommission-hardware, 10serviceops-collab, 10Patch-For-Review: decommission phab1001.eqiad.wmnet - https://phabricator.wikimedia.org/T323418 (10Dzahn) [23:43:00] 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥), 10serviceops-collab, 10Patch-For-Review: move phabricator to new hardware generation - https://phabricator.wikimedia.org/T280597 (10Dzahn) [23:43:43] 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥), 10serviceops-collab, 10Patch-For-Review: move phabricator to new hardware generation - https://phabricator.wikimedia.org/T280597 (10Dzahn)