[00:06:16] 10Release-Engineering-Team, 10Scap: scap backport applied +2 to wrong changeset when run shortly after a rebase and hung waiting for merge - https://phabricator.wikimedia.org/T333241 (10thcipriani) [00:34:41] 10Continuous-Integration-Infrastructure, 10Wikimedia-Fundraising-CiviCRM, 10Patch-For-Review: CiviCRM CI jobs fails when migrating from Stretch to Bullseye - https://phabricator.wikimedia.org/T307178 (10Eileenmcnaughton) @hashar is this closable? [04:43:40] 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review, 10Release, 10Train Deployments: 1.41.0-wmf.7 deployment blockers - https://phabricator.wikimedia.org/T330213 (10hashar) I went to browse https://test.wikipedia.org/wiki/Main_Page it has a top banner showing the proper version: > This is... [05:03:24] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Radar), 10SRE: git: detected dubious ownership in repository at '/srv/mediawiki-staging' - https://phabricator.wikimedia.org/T325128 (10hashar) 05Open→03Resolved a:03hashar For other use case, looks like that will be done as part {T335354} I... [05:12:25] 10Gerrit, 10wikitech.wikimedia.org: Wikitech->Gerrit account block and unblock has stopped working - https://phabricator.wikimedia.org/T306297 (10hashar) [05:12:29] 10Gerrit, 10Security: Cannot assign user name "XXX" to account ####; name already in use. - https://phabricator.wikimedia.org/T216605 (10hashar) [05:14:21] 10Gerrit, 10Release-Engineering-Team: Fix up Gerrit usernames having upper case character(s) - https://phabricator.wikimedia.org/T323097 (10hashar) [05:14:24] 10Gerrit, 10wikitech.wikimedia.org: Wikitech->Gerrit account block and unblock has stopped working - https://phabricator.wikimedia.org/T306297 (10hashar) [05:15:09] 10Gerrit, 10wikitech.wikimedia.org: Wikitech->Gerrit account block and unblock has stopped working - https://phabricator.wikimedia.org/T306297 (10hashar) 05Open→03Resolved a:03hashar @MarcoAurelio yes it is fixed per my earlier comment T306297#7867248 The issue was due to the mixed case, most cases got... [07:19:31] 10Gerrit: gerrit-replica seems out of date breaking extension distributor - https://phabricator.wikimedia.org/T335730 (10Bawolff) [07:19:53] 10Gerrit, 10ExtensionDistributor: gerrit-replica seems out of date breaking extension distributor - https://phabricator.wikimedia.org/T335730 (10Bawolff) [07:20:13] 10Gerrit, 10ExtensionDistributor: gerrit-replica seems out of date breaking extension distributor - https://phabricator.wikimedia.org/T335730 (10Bawolff) [07:29:58] 10GitLab (CI & Job Runners), 10Release-Engineering-Team, 10Patch-For-Review: Consider adding "official" node images to list of allowed images for gitlab runners - https://phabricator.wikimedia.org/T335320 (10Jelto) 05Open→03Resolved p:05Triage→03Medium a:03Jelto Thanks for opening the issue and sub... [07:36:22] 10Release-Engineering-Team, 10Scap: scap backport applied +2 to wrong patchset when run shortly after a rebase and hung waiting for merge - https://phabricator.wikimedia.org/T333241 (10Tgr) [08:04:58] (03PS2) 10Hashar: Add BulkBlock extension [integration/config] - 10https://gerrit.wikimedia.org/r/913541 (owner: 10Vedmaka Wakalaka) [08:05:08] (03CR) 10Hashar: [C: 03+2] Add BulkBlock extension [integration/config] - 10https://gerrit.wikimedia.org/r/913541 (owner: 10Vedmaka Wakalaka) [08:06:27] (03Merged) 10jenkins-bot: Add BulkBlock extension [integration/config] - 10https://gerrit.wikimedia.org/r/913541 (owner: 10Vedmaka Wakalaka) [08:06:29] (03CR) 10Hashar: [C: 03+2] zuul: [mediawiki/extensions/PageViewInfoGA] Add dependency [integration/config] - 10https://gerrit.wikimedia.org/r/913925 (owner: 10Umherirrender) [08:07:43] (03Merged) 10jenkins-bot: zuul: [mediawiki/extensions/PageViewInfoGA] Add dependency [integration/config] - 10https://gerrit.wikimedia.org/r/913925 (owner: 10Umherirrender) [08:07:51] (03PS2) 10Hashar: Zuul: [mediawiki/extensions/StandardDialogs] add OOJSPlus [integration/config] - 10https://gerrit.wikimedia.org/r/913191 (owner: 10Robert Vogel) [08:09:30] (03CR) 10Hashar: [C: 03+2] "I have slightly amended the commit message." [integration/config] - 10https://gerrit.wikimedia.org/r/913191 (owner: 10Robert Vogel) [08:10:14] (03CR) 10Hashar: [C: 03+2] "Deployed" [integration/config] - 10https://gerrit.wikimedia.org/r/913541 (owner: 10Vedmaka Wakalaka) [08:10:16] (03CR) 10Hashar: [C: 03+2] "Deployed" [integration/config] - 10https://gerrit.wikimedia.org/r/913925 (owner: 10Umherirrender) [08:10:43] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/StandardDialogs] add OOJSPlus [integration/config] - 10https://gerrit.wikimedia.org/r/913191 (owner: 10Robert Vogel) [08:11:06] 10Phabricator, 10Release-Engineering-Team, 10serviceops-collab, 10User-brennen: Migrate phabricator.wikimedia.org to Phorge as upstream - https://phabricator.wikimedia.org/T333885 (10valerio.bozzolan) >>! In T333885#8817557, @Aklapper wrote: > Thanks. In my understanding, for the Wikimedia migration we wil... [08:12:51] (03Abandoned) 10Hashar: Add PageViewInfoGA extension dependency [integration/config] - 10https://gerrit.wikimedia.org/r/910848 (owner: 10Lens0021) [08:13:19] (03CR) 10Hashar: [C: 03+2] "Deployed" [integration/config] - 10https://gerrit.wikimedia.org/r/913191 (owner: 10Robert Vogel) [08:14:13] (03CR) 10Hashar: [C: 03+2] Zuul: [mediawiki/extensions/PageImages] Enable Sonar Codehealth [integration/config] - 10https://gerrit.wikimedia.org/r/909330 (https://phabricator.wikimedia.org/T321837) (owner: 10Pwangai) [08:14:15] (03CR) 10Hashar: [C: 03+2] Zuul: [mediawiki/extensions/CategoryTree] Enable Sonar Codehealth [integration/config] - 10https://gerrit.wikimedia.org/r/909331 (https://phabricator.wikimedia.org/T321837) (owner: 10Pwangai) [08:14:17] !log Reloaded Zuul for https://gerrit.wikimedia.org/r/913541 https://gerrit.wikimedia.org/r/913925 https://gerrit.wikimedia.org/r/913191 [08:14:18] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [08:15:23] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/PageImages] Enable Sonar Codehealth [integration/config] - 10https://gerrit.wikimedia.org/r/909330 (https://phabricator.wikimedia.org/T321837) (owner: 10Pwangai) [08:15:26] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/CategoryTree] Enable Sonar Codehealth [integration/config] - 10https://gerrit.wikimedia.org/r/909331 (https://phabricator.wikimedia.org/T321837) (owner: 10Pwangai) [08:15:55] !log Reloaded Zuul to enable Sonar Codehealth on CategoryTree https://gerrit.wikimedia.org/r/909331 and PageImages https://gerrit.wikimedia.org/r/909330 # T321837 [08:15:58] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [08:15:58] T321837: Repositories integrated into Codehealth Pipeline (Production) - https://phabricator.wikimedia.org/T321837 [08:17:07] (03CR) 10Hashar: [C: 03+1] build: Switch phan to special library mode [integration/docroot] - 10https://gerrit.wikimedia.org/r/909349 (https://phabricator.wikimedia.org/T334492) (owner: 10Jforrester) [09:02:53] Good morning! We'll be starting the gitlab migration from codfw back to eqiad in the next few minutes, so there will be some downtime ahead. We expect the process to take approximately two hours. Any issues, please let us know! https://phabricator.wikimedia.org/T335504 [09:28:22] 10GitLab (Integrations), 10Phabricator: Phabricator task gives "Invalid argument supplied for foreach()" - https://phabricator.wikimedia.org/T333347 (10dom_walden) I too see this consistently for this task: T334970 and whenever I go to the home page. [09:29:15] hashar,jnuche: Good morning. We're trying to do the gitlab failover today, but when we took gitlab down, phabricator also went down with the foreach() error in the task above. Are either of you aware of any new hard dependency on gitlab that phabricator has? [09:35:18] 10GitLab, 10Phabricator: Phabricator displays foreach error and is unusable when GitLab is down - https://phabricator.wikimedia.org/T335747 (10Novem_Linguae) [09:35:52] 10GitLab, 10Phabricator: Phabricator displays foreach error and is unusable when GitLab is down - https://phabricator.wikimedia.org/T335747 (10taavi) [09:36:04] 10GitLab (Integrations), 10Phabricator: Phabricator task gives "Invalid argument supplied for foreach()" - https://phabricator.wikimedia.org/T333347 (10taavi) [09:36:23] 10GitLab (Integrations), 10Phabricator: Phabricator does not degrade properly when GitLab is unavailable - https://phabricator.wikimedia.org/T333347 (10taavi) [09:36:29] 10GitLab (Integrations), 10Phabricator: Phabricator does not degrade properly when GitLab is unavailable - https://phabricator.wikimedia.org/T333347 (10LSobanski) Also happened today during a GitLab maintenance (which was subsequently abandoned) so looks like the widget doesn't fail gracefully. [09:43:08] 10Phabricator, 10Release-Engineering-Team, 10serviceops-collab, 10User-brennen: Migrate phabricator.wikimedia.org to Phorge as upstream - https://phabricator.wikimedia.org/T333885 (10valerio.bozzolan) 05Open→03Resolved Maybe this is already under control but I noticed some configurations that - at the... [09:43:27] 10Phabricator, 10Release-Engineering-Team, 10serviceops-collab, 10User-brennen: Migrate phabricator.wikimedia.org to Phorge as upstream - https://phabricator.wikimedia.org/T333885 (10valerio.bozzolan) 05Resolved→03Open (misclick) [10:01:12] eoghan: morning, it seems the issue is coming from the Mr Widget Phabricator extension: https://phabricator.wikimedia.org/T324149 [10:01:26] brennen probably knows more about it [10:02:14] looks like it needs to handle the case when gitlab is not reachable [10:02:44] Yep yep. I'll have a chat later about it, thanks! [10:16:00] 10GitLab (Project Migration), 10Release-Engineering-Team (Priority Backlog 📥), 10API Platform, 10Anti-Harassment, and 19 others: Migrate PipelineLib repos to GitLab - https://phabricator.wikimedia.org/T332953 (10Michaelcochez) We do our development on github. Does it make more sense to restart with a new r... [10:32:11] eoghan: sorry I was having lunch. Phab went down probably because of tasks showing up gitlab merge requests for the current task and it would not have a cache/timeout [10:32:29] there is the same one for Gerrit [10:33:01] fun I filed that T333347 [10:33:01] T333347: Phabricator does not degrade properly when GitLab is unavailable - https://phabricator.wikimedia.org/T333347 [10:35:25] Thanks. Unfortunately we'll either need to fix or disable the extension before we migrate gitlab back, or else we will need to have ~2.5 hours of phabricator downtime while we move gitlab. [10:51:16] 10GitLab (Project Migration), 10Release-Engineering-Team (Priority Backlog 📥), 10API Platform, 10Anti-Harassment, and 19 others: Migrate PipelineLib repos to GitLab - https://phabricator.wikimedia.org/T332953 (10hashar) >>! In T332953#8819607, @Michaelcochez wrote: > We do our development on github. Does i... [10:51:28] eoghan: yeah that sounds reasonable. Brennen would know more for sure (he is on US west coast so not awake yet) [10:51:39] Yep, will chat with him later! [10:52:56] eoghan: and we have the same system for Gerrit, I guess the code there does have a timeout set and/or handle the lack of results gracefully [11:38:31] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure: Migrate all CI jobs from buster to bullseye or later and drop buster testing support - https://phabricator.wikimedia.org/T335765 (10Jdforrester-WMF) [11:41:28] 10Continuous-Integration-Infrastructure, 10Patch-For-Review: Migrate all CI jobs from stretch to buster or later and drop stretch testing support - https://phabricator.wikimedia.org/T278203 (10Jdforrester-WMF) [11:42:29] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure: Migrate Quibble images from buster to bullseye - https://phabricator.wikimedia.org/T335766 (10Jdforrester-WMF) [12:28:29] 10Phabricator, 10Regression: Phabricator is auto claiming tasks when closing as resolved (again) - https://phabricator.wikimedia.org/T134106 (10thiemowmde) > If a person is working on something, that person should claim that Task. Not doing this is a communication problem that should not be encouraged in this... [12:30:17] (03PS1) 10Jforrester: [WIP] Docker: Provide bullseye-based Quibble images [integration/config] - 10https://gerrit.wikimedia.org/r/914316 (https://phabricator.wikimedia.org/T335766) [12:31:32] (03CR) 10Jforrester: "This errors during install of `python3-wheel`; do we definitely need it?" [integration/config] - 10https://gerrit.wikimedia.org/r/914316 (https://phabricator.wikimedia.org/T335766) (owner: 10Jforrester) [13:48:17] (03PS1) 10Hashar: dockerfiles: update Depends in control files [integration/config] - 10https://gerrit.wikimedia.org/r/914330 (https://phabricator.wikimedia.org/T278203) [13:48:53] (03CR) 10Jforrester: [C: 03+1] "Ha, oops." [integration/config] - 10https://gerrit.wikimedia.org/r/914330 (https://phabricator.wikimedia.org/T278203) (owner: 10Hashar) [13:49:57] (03CR) 10Hashar: [C: 03+2] "Ideally there should be a test to ensure `control` refers to images actually in the `Dockerfile.template`... I will update the task that k" [integration/config] - 10https://gerrit.wikimedia.org/r/914330 (https://phabricator.wikimedia.org/T278203) (owner: 10Hashar) [13:50:17] 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review, 10Release, 10Train Deployments: 1.41.0-wmf.7 deployment blockers - https://phabricator.wikimedia.org/T330213 (10Ladsgroup) ##### Risky Patch! 🚂🔥 * **Change**: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/910756/ * **Summary**: We... [13:50:40] 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review, 10Release, 10Train Deployments: 1.41.0-wmf.7 deployment blockers - https://phabricator.wikimedia.org/T330213 (10Ladsgroup) Also if you see anything with externallinks in logs, that's probably me. Ping me [13:51:22] 10Continuous-Integration-Infrastructure, 10Patch-For-Review: Migrate all CI jobs from stretch to buster or later and drop stretch testing support - https://phabricator.wikimedia.org/T278203 (10hashar) [13:51:40] (03Merged) 10jenkins-bot: dockerfiles: update Depends in control files [integration/config] - 10https://gerrit.wikimedia.org/r/914330 (https://phabricator.wikimedia.org/T278203) (owner: 10Hashar) [13:55:42] hashar: I think we should just nuke analytics/wikistats from orbit and stop trying to help people write code for it. :-9 [14:01:41] 10Gerrit, 10ExtensionDistributor: gerrit-replica seems out of date breaking extension distributor - https://phabricator.wikimedia.org/T335730 (10hashar) a:03hashar Browsing GitHub https://github.com/wikimedia/mediawiki-extensions-ConfirmAccount gives me https://gerrit.wikimedia.org/r/c/mediawiki/extensions/C... [14:09:03] 10Gerrit, 10ExtensionDistributor: gerrit-replica seems out of date breaking extension distributor - https://phabricator.wikimedia.org/T335730 (10hashar) I have triggered the replication manually: ` $ ssh -p 29418 gerrit.wikimedia.org replication start mediawiki/extensions/ConfirmAccount --now --wait Replicate... [14:20:31] <_joe_> All CI on jenkins / zuul is currently broken [14:24:20] 10Gerrit, 10ExtensionDistributor: gerrit-replica seems out of date breaking extension distributor - https://phabricator.wikimedia.org/T335730 (10hashar) 05Open→03Resolved After yesterday change got merged, I do get a successful replication message: ` [2023-05-01 08:01:18,705] Replication to gerrit2@gerrit2... [14:32:39] 10Continuous-Integration-Infrastructure, 10Patch-For-Review: Migrate all CI jobs from stretch to buster or later and drop stretch testing support - https://phabricator.wikimedia.org/T278203 (10hashar) [14:32:42] 10Continuous-Integration-Infrastructure, 10Patch-For-Review: Migrate all CI jobs from stretch to buster or later and drop stretch testing support - https://phabricator.wikimedia.org/T278203 (10hashar) For `docker.wikimedia.org/releng/cassandra311` I have filed T335780 [14:32:59] 10Continuous-Integration-Infrastructure, 10Patch-For-Review: Migrate all CI jobs from stretch to buster or later and drop stretch testing support - https://phabricator.wikimedia.org/T278203 (10hashar) [14:56:15] 10GitLab (Integrations), 10Phabricator: Phabricator does not degrade properly when GitLab is unavailable - https://phabricator.wikimedia.org/T333347 (10brennen) a:03brennen I thought I had this handled, but clearly I was mistaken. I'll put a more robust guard around the call to the GitLab API. [15:01:53] 10GitLab (Integrations), 10Phabricator, 10Release-Engineering-Team, 10User-brennen: Phabricator does not degrade properly when GitLab is unavailable - https://phabricator.wikimedia.org/T333347 (10brennen) 05Open→03In progress p:05Triage→03High [15:02:55] 10Phabricator, 10Regression: Phabricator is auto claiming tasks when closing as resolved (again) - https://phabricator.wikimedia.org/T134106 (10valerio.bozzolan) >>! @thiemowmde > Closing a task is pretty much the opposite of "I'm going to work on it". I understand that you are opinionated, and I understand... [15:20:29] 10Release-Engineering-Team, 10API Platform, 10AQS2.0, 10Platform Engineering, and 5 others: Define a procedure/pattern to populate test environments - https://phabricator.wikimedia.org/T334851 (10sbassett) [15:50:08] 10Phabricator, 10Regression: Phabricator is auto claiming tasks when closing as resolved (again) - https://phabricator.wikimedia.org/T134106 (10thiemowmde) I really don't get why these users don't just use the "Closed By" filter, do the {nav Assign / Claim} click like everybody else, or set up a bot that does... [16:14:44] 10Release-Engineering-Team, 10API Platform, 10AQS2.0, 10Platform Engineering, and 5 others: Define a procedure/pattern to populate test environments - https://phabricator.wikimedia.org/T334851 (10Mstyles) @Htriedman perhaps some of the data privacy techniques could be used here to generate test data from t... [16:35:05] 10Release-Engineering-Team, 10API Platform, 10AQS2.0, 10Platform Engineering, and 5 others: Define a procedure/pattern to populate test environments - https://phabricator.wikimedia.org/T334851 (10Htriedman) Hi all! This is a really interesting problem, and I think that there are definitely some data privac... [16:55:19] Project beta-code-update-eqiad build #441997: 04FAILURE in 2 min 19 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/441997/ [16:55:55] !log scap version "4.52.0" installed on 15 hosts [16:55:57] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:57:09] dancy: I assume the code update failure was you because it says scap doesn't exist [16:57:16] yeah [17:00:11] 10Beta-Cluster-Infrastructure: beta-code-update-eqiad failing to checkout submodule - https://phabricator.wikimedia.org/T335602 (10dancy) 05Open→03Resolved a:03dancy scap 4.52.0 has been deployed to beta and production. [17:05:18] Yippee, build fixed! [17:05:19] Project beta-code-update-eqiad build #441998: 09FIXED in 2 min 17 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/441998/ [18:08:33] 10GitLab (Integrations), 10Phabricator, 10Release-Engineering-Team, 10Patch-For-Review, 10User-brennen: Phabricator does not degrade properly when GitLab is unavailable - https://phabricator.wikimedia.org/T333347 (10brennen) Noting here where I messed this up. This extension was originally a copypasta o... [20:29:47] 10Phabricator, 10Regression: Phabricator is auto claiming tasks when closing as resolved (again) - https://phabricator.wikimedia.org/T134106 (10valerio.bozzolan) > The best idea I currently have is to add an action {nav Change Status > Resolve & Claim} and make {nav Resolved} behave sane again. That is exactl... [21:49:32] 10Continuous-Integration-Config, 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: wikimedia/fundraising/civicrm-buildkit repo needs to V+2 itself - https://phabricator.wikimedia.org/T142901 (10Eileenmcnaughton) 05Open→03Declined