[06:58:19] 10GitLab (Infrastructure), 10Release-Engineering-Team, 10collaboration-services, 10Patch-For-Review: Upgrade GitLab to major version 16 - https://phabricator.wikimedia.org/T338460 (10Jelto) [07:00:27] 10GitLab (Infrastructure), 10Release-Engineering-Team, 10collaboration-services, 10Patch-For-Review: Upgrade GitLab to major version 16 - https://phabricator.wikimedia.org/T338460 (10Jelto) Actual version upgrade is tracked in: T343297 [08:56:22] 10Phabricator, 10Patch-For-Review: Update custom allowlist in src/applications/phurl/storage/PhabricatorPhurlURL.php - https://phabricator.wikimedia.org/T343267 (10CodeReviewBot) aklapper opened https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/11 Update custom PhURL domain filter list [08:56:47] 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review: Update custom allowlist in src/applications/phurl/storage/PhabricatorPhurlURL.php - https://phabricator.wikimedia.org/T343267 (10Aklapper) a:03Aklapper [08:57:34] 10GitLab (Pipeline Services Migration🐤), 10Release-Engineering-Team (Priority Backlog 📥), 10Toolforge, 10cloud-services-team, 10Patch-For-Review: Move Toolforge PipelineLib repositories to GitLab - https://phabricator.wikimedia.org/T334399 (10CodeReviewBot) dcaro opened https://gitlab.wikimedia.org/repos... [09:04:06] raising T343291, about to go into meetings else I'd take a proper look [09:04:07] T343291: [betacluster] Cannot login - UserLogin RuntimeException: Failed to run getConfiguration.php - https://phabricator.wikimedia.org/T343291 [09:08:24] 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥), 10User-brennen: Work out how to bring in new Phabricator translations again - https://phabricator.wikimedia.org/T318763 (10Aklapper) p:05Triage→03Medium [09:08:28] 10Phabricator: Make OAuth session persistent in Phabricator - https://phabricator.wikimedia.org/T140448 (10Aklapper) p:05Triage→03Low [09:09:04] 10Phabricator (Upstream), 10Upstream: Send reminder email on open/stalled tasks when Due Date is reached - https://phabricator.wikimedia.org/T302785 (10Aklapper) p:05Triage→03Low [09:09:26] 10Phabricator (Upstream), 10Upstream: Subscriber changes not reported by email if Herald also changes subscribers in the same action - https://phabricator.wikimedia.org/T133073 (10Aklapper) p:05Triage→03Low [09:09:30] 10Phabricator (Upstream), 10Upstream: Allow locking adding Task comments to members of certain projects - https://phabricator.wikimedia.org/T292505 (10Aklapper) p:05Triage→03Low [09:09:45] 10Phabricator (Upstream), 10Mobile, 10Upstream: Unable to "manage user" on mobile - https://phabricator.wikimedia.org/T310543 (10Aklapper) p:05Triage→03Low [09:09:47] 10Phabricator, 10MediaWiki-extensions-Translate, 10translatewiki.net, 10I18n: Improvements for automatic reporting of tasks from translatewiki to Phabricator - https://phabricator.wikimedia.org/T188379 (10Aklapper) p:05Triage→03Low [09:14:27] 10Phabricator, 10Patch-For-Review, 10Technical-Debt: Revert unused downstream Phab code to suppress triggering mail (predates silent batch-editing) - https://phabricator.wikimedia.org/T343264 (10CodeReviewBot) aklapper opened https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/12 Rev... [09:15:39] 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review, 10Technical-Debt: Revert unused downstream Phab code to suppress triggering mail (predates silent batch-editing) - https://phabricator.wikimedia.org/T343264 (10Aklapper) a:03Aklapper [09:23:24] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team: [betacluster] Cannot login - UserLogin RuntimeException: Failed to run getConfiguration.php - https://phabricator.wikimedia.org/T343291 (10TheresNoTime) Little suspicious of {420f22226656594f7af7a5c113ebf970ae6adcc8} ([[ https://gerrit.wikimedia.org/r/... [09:25:42] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team: [betacluster] Cannot login - UserLogin RuntimeException: Failed to run getConfiguration.php - https://phabricator.wikimedia.org/T343291 (10TheresNoTime) ==== Error ==== * mwversion: 1.41.0-alpha * reqId: ZMob5kBUY-0MXePlnXmT9gAAABg * https://beta-log... [10:44:08] GitLab needs a short maintenance restart in one hour [11:15:00] 10Release-Engineering-Team (Seen), 10MW-on-K8s, 10SRE, 10Traffic, and 2 others: Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536 (10Clement_Goubert) [11:58:30] brennen: Do you know of a way to have a Herald rule be editable by more than one non-admin? [11:59:03] It seems rules are either personal or global admin rules [12:28:43] it's one of those, yes [12:29:20] https://phabricator.wikimedia.org/tag/acl*phabricator/ would be permissions to edit global Herald rules without having to be an admin [13:50:47] Oh nice, that could work. [13:50:59] andre: would you mind adding lgaulia to that? [13:56:09] * Krinkle did so. I sometimes forget I'm a phab admin [13:59:47] ehehe, thanks [14:14:31] 10GitLab (CI & Job Runners), 10collaboration-services, 10serviceops: Standardize Debian package builds on GitLab CI - https://phabricator.wikimedia.org/T304491 (10LSobanski) p:05Low→03High [14:51:31] 10GitLab (CI & Job Runners), 10collaboration-services, 10serviceops: Standardize Debian package builds on GitLab CI - https://phabricator.wikimedia.org/T304491 (10taavi) This is what we're currently using for Toolforge packages: https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/tree/main/debian-build... [14:52:19] hm, is there a reason why phab admins (that are not in acl*phabricator) can't edit membership of that group? [15:01:52] taavi: I don't think so. Let me fix that. [15:08:49] 10Gitlab-Application-Security-Pipeline, 10Security Team AppSec, 10Security-Team, 10Security, 10user-sbassett: Deprecate unused / problematic Gitlab AppSec CI includes - https://phabricator.wikimedia.org/T343262 (10sbassett) a:03sbassett [15:09:26] 10Gitlab-Application-Security-Pipeline, 10Security Team AppSec, 10Security-Team, 10SecTeam-Processed, and 2 others: Deprecate unused / problematic Gitlab AppSec CI includes - https://phabricator.wikimedia.org/T343262 (10sbassett) [15:13:04] 10GitLab (CI & Job Runners), 10collaboration-services, 10serviceops: Standardize Debian package builds on GitLab CI - https://phabricator.wikimedia.org/T304491 (10MatthewVernon) I'm looking at this as a KR for this quarter (for some packages we want to deploy), and I'm expecting to use a `dgit`-based approac... [15:53:19] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Escape Goats🐐), 10Patch-For-Review: Report failures to docpub users - https://phabricator.wikimedia.org/T343222 (10CodeReviewBot) jnuche opened https://gitlab.wikimedia.org/repos/releng/jenkins-deploy/-/merge_requests/35 docpub: notify first build fai... [15:53:35] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Escape Goats🐐), 10Patch-For-Review: Report failures to docpub users - https://phabricator.wikimedia.org/T343222 (10CodeReviewBot) jnuche opened https://gitlab.wikimedia.org/repos/releng/docpub/-/merge_requests/8 send triggerer's email to Jenkins job [16:18:30] 10Release-Engineering-Team (Priority Backlog 📥), 10dev-images, 10mwcli, 10User-brennen: Add php-luasandbox to dev-images used by mwcli - https://phabricator.wikimedia.org/T286678 (10CodeReviewBot) brennen closed https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/9 Draft: Add php-luasand... [16:33:10] PROBLEM - Check systemd state on doc2002 is CRITICAL: CRITICAL - degraded: The following units failed: rsync-doc-host-data-sync.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [16:36:14] 10GitLab (Account Approval), 10Release-Engineering-Team: Requesting GitLab account activation for Pepe piton - https://phabricator.wikimedia.org/T343286 (10brennen) 05Open→03Resolved a:03brennen [17:31:00] RECOVERY - Check systemd state on doc2002 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [18:25:24] 10Release-Engineering-Team (Escape Goats🐐), 10Patch-For-Review, 10Release, 10Train Deployments: 1.41.0-wmf.20 deployment blockers - https://phabricator.wikimedia.org/T340248 (10dancy) [18:26:11] 10Release-Engineering-Team (Escape Goats🐐), 10Patch-For-Review, 10Release, 10Train Deployments: 1.41.0-wmf.20 deployment blockers - https://phabricator.wikimedia.org/T340248 (10dancy) Rolling the train back to group0 due to T343375. [19:06:19] 10Release-Engineering-Team (Escape Goats🐐), 10Patch-For-Review, 10Release, 10Train Deployments: 1.41.0-wmf.20 deployment blockers - https://phabricator.wikimedia.org/T340248 (10Lucas_Werkmeister_WMDE) >>! In T340248#9064285, @dancy wrote: > Rolling the train back to group0 due to T343375. That issue sound... [19:07:02] (I probably can’t help out with the train blocker beyond ^ that comment until tomorrow, and might not have much time tomorrow either, sorry) [19:08:44] OK. I'll try reverting https://gerrit.wikimedia.org/r/c/mediawiki/core/+/940355 [19:34:54] it looks like it could be it, yes [19:35:13] Anyone want to give a +2 to https://gerrit.wikimedia.org/r/c/mediawiki/core/+/944853/ ? [19:35:17] (the revert) [19:35:33] yep [19:35:40] thx [19:36:14] +2ed [19:50:37] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team: [betacluster] Cannot login - UserLogin RuntimeException: Failed to run getConfiguration.php - https://phabricator.wikimedia.org/T343291 (10vaughnwalters) This also applies to temp users. When attempting to edit on https://de.wikipedia.beta.wmflabs.org... [20:30:54] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team: [betacluster] Cannot login - UserLogin RuntimeException: Failed to run getConfiguration.php - https://phabricator.wikimedia.org/T343291 (10TheresNoTime) `name=normalized_message,lines=10 Executing: /bin/bash '/srv/mediawiki/php-master/vendor/wikimedia/... [20:46:05] o/ I'm starting to feel like T343291 is more than it seems [20:46:05] T343291: [betacluster] Cannot login - UserLogin RuntimeException: Failed to run getConfiguration.php - https://phabricator.wikimedia.org/T343291 [20:49:48] yeah I think jobqueue: Use a proper config system for dealing with settings (https://gerrit.wikimedia.org/r/c/mediawiki/core/+/937045) is to blame [20:51:55] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team: [betacluster] Cannot login - UserLogin RuntimeException: Failed to run getConfiguration.php - https://phabricator.wikimedia.org/T343291 (10TheresNoTime) Revert of {ab802a66e685c9e588167f127996f037594679bc} submitted at https://gerrit.wikimedia.org/r/c/... [21:10:33] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team: [betacluster] Cannot login - UserLogin RuntimeException: Failed to run getConfiguration.php - https://phabricator.wikimedia.org/T343291 (10TheresNoTime) Temporarily doing `lang=diff diff --git a/includes/jobqueue/JobQueueGroupFactory.php b/includes/job... [21:11:11] 10Release-Engineering-Team (Escape Goats🐐), 10Patch-For-Review, 10Release, 10Train Deployments: 1.41.0-wmf.20 deployment blockers - https://phabricator.wikimedia.org/T340248 (10dancy) [21:13:26] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team: [betacluster] Cannot login - UserLogin RuntimeException: Failed to run getConfiguration.php - https://phabricator.wikimedia.org/T343291 (10TheresNoTime) [21:13:29] 10Release-Engineering-Team (Escape Goats🐐), 10Release, 10Train Deployments: 1.41.0-wmf.21 deployment blockers - https://phabricator.wikimedia.org/T340249 (10TheresNoTime) [21:14:39] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team: [betacluster] Cannot login - UserLogin RuntimeException: Failed to run getConfiguration.php - https://phabricator.wikimedia.org/T343291 (10TheresNoTime) p:05Triage→03Unbreak! I could be wrong, but as it stands without that revert, this'll break thi... [22:19:27] 10Release-Engineering-Team (Seen), 10MW-on-K8s, 10SRE, 10Traffic, and 2 others: Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536 (10Krinkle) [22:28:50] dancy: In case anyone else noticed ^ [22:28:55] lots of jsonTruncated noise lately [22:30:25] DuplicateParse errors. [23:10:57] 10GitLab (Project Migration), 10collaboration-services: Migrate SRE repositories to GitLab - operations/debs - https://phabricator.wikimedia.org/T341991 (10colewhite) [23:12:51] 10GitLab (Project Migration), 10collaboration-services: Migrate SRE repositories to GitLab - operations/software - https://phabricator.wikimedia.org/T341504 (10colewhite)