[00:09:11] 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review, 10Release, 10Train Deployments: 1.40.0-wmf.23 deployment blockers - https://phabricator.wikimedia.org/T325586 (10ssastry) [00:15:03] PROBLEM - Check systemd state on doc1002 is CRITICAL: CRITICAL - degraded: The following units failed: rsync-doc-doc2001.codfw.wmnet.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [00:36:41] 10GitLab (CI & Job Runners), 10Release-Engineering-Team, 10mwbot-rs, 10mwcli: GitLab CI jobs failing with "You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" - https://phabricator.wikimedia.org/T329216 (10bd808) htt... [01:12:49] RECOVERY - Check systemd state on doc1002 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [02:30:52] 10GitLab (CI & Job Runners), 10Release-Engineering-Team, 10mwbot-rs, 10mwcli: GitLab CI jobs failing with "You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" - https://phabricator.wikimedia.org/T329216 (10Legoktm) >... [04:29:03] 10GitLab (Auth & Access), 10Release-Engineering-Team, 10LDAP, 10Upstream: Enforce the minimum character of the LDAP user name to be more than two characters. - https://phabricator.wikimedia.org/T329811 (10I) [04:30:31] 10GitLab (Auth & Access), 10Release-Engineering-Team, 10Upstream: Cannot log into gitlab.wikimedia.org with LDAP username which has only one character - https://phabricator.wikimedia.org/T329728 (10I) Another workaround so far is to limit the LDAP username to at least two characters, The task has been submit... [08:51:20] 10Release-Engineering-Team, 10LDAP: Enforce the minimum character of the LDAP user name to be more than two characters. - https://phabricator.wikimedia.org/T329811 (10Peachey88) [09:32:01] 10Release-Engineering-Team, 10LDAP: Enforce the minimum character of the LDAP user name to be more than two characters. - https://phabricator.wikimedia.org/T329811 (10taavi) I don't think there's a benefit here in having two separate tasks, so I'm going to merge this one into the parent one. [09:32:09] 10Release-Engineering-Team, 10LDAP: Enforce the minimum character of the LDAP user name to be more than two characters. - https://phabricator.wikimedia.org/T329811 (10taavi) [09:32:22] 10GitLab (Auth & Access), 10Release-Engineering-Team, 10Upstream: Cannot log into gitlab.wikimedia.org with LDAP username which has only one character - https://phabricator.wikimedia.org/T329728 (10taavi) [09:35:07] 10GitLab (Auth & Access), 10Release-Engineering-Team, 10Upstream: Cannot log into gitlab.wikimedia.org with LDAP username which has only one character - https://phabricator.wikimedia.org/T329728 (10I) 05Stalled→03Open >>! In T329728#8619051, @brennen wrote: > > As such, I'm not sure there's a lot we can... [09:39:41] o/ the stuck Kartographer CI jobs from yesterday didn't all get cleared it seems — https://integration.wikimedia.org/zuul shows 889179's codehealth job has been pending for 17hrs+ [09:40:00] (tl;dr could someone clear those jobs?) [09:41:37] 10Release-Engineering-Team, 10LDAP: Enforce the minimum character of the LDAP user name to be more than two characters. - https://phabricator.wikimedia.org/T329811 (10Peachey88) @taavi I believe this task is more about forcing the restriction in our actual LDAP directory server, and the other task is about enf... [10:25:46] (03PS2) 10Kosta Harlan: zuul: Add GrowthExperiments to extension-javascript-documentation [integration/config] - 10https://gerrit.wikimedia.org/r/888022 (https://phabricator.wikimedia.org/T329034) [10:31:19] !log restarted Zuul to clear stale events [10:31:20] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:31:49] TheresNoTime: those stuck pipelines are gone now [10:32:24] Thanks! :) [11:27:46] 10Beta-Cluster-Infrastructure: beta cluster down - https://phabricator.wikimedia.org/T329592 (10Jgiannelos) I just re-run the tests and looks like the relevant test is passing now. Thanks! [13:19:56] 10Beta-Cluster-Infrastructure: beta cluster down - https://phabricator.wikimedia.org/T329592 (10taavi) [13:19:58] 10Beta-Cluster-Infrastructure, 10Math, 10Mathoid: Mathoid on beta cluster is down - https://phabricator.wikimedia.org/T329747 (10taavi) 05Open→03Resolved [14:05:01] 10Gerrit, 10Pywikibot: 500 server error when pulling Pywikibot i18n - https://phabricator.wikimedia.org/T329452 (10binbot) One more strange thing: when I unpack the Pywikibot zip version, `c:\Pywikibot\scripts\i18n\.git` is a supdirectory. When I pull it from git, the same name belongs to a file, containing `"... [15:17:08] 10Project-Admins: Create three Phab Projects for Machine Learning: Lift Wing, Pilot Flag, Test Grounds - https://phabricator.wikimedia.org/T264774 (10Aklapper) @calbon: Could you please answer the last comment? Thanks in advance! [16:26:39] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (GitLab V: Event Horizon 🌄): Disallow direct access to the docker.io registry from gitlab runners - https://phabricator.wikimedia.org/T329679 (10bd808) If possible the proxy should be configured to use a named Docker account when pulling from the Docker u... [16:34:33] 10Deployments, 10Release-Engineering-Team: Mediawiki deploy servers should not be mediawiki installation targets - https://phabricator.wikimedia.org/T329857 (10dancy) [16:38:14] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (GitLab V: Event Horizon 🌄): Disallow direct access to the docker.io registry from gitlab runners - https://phabricator.wikimedia.org/T329679 (10dancy) Thanks for the research @bd808 and @Addshore . @bd808 I may hit you up for those credentials soon. [17:06:41] 10Project-Admins: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706 (10Xaosflux) Hi, can I be added so I can create tags; working on managing some of the community tech events. I understand the guidelines from https://www.mediawiki.org/wiki/Phabricator/... [17:24:10] 10Project-Admins: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706 (10bd808) >>! In T706#8622364, @Xaosflux wrote: > Hi, can I be added so I can create tags; working on managing some of the community tech events. I understand the guidelines from https:... [17:47:46] 10Release-Engineering-Team (GitLab V: Event Horizon 🌄): Add Gitlab JWT support to Reggie - https://phabricator.wikimedia.org/T323394 (10demon) 05Open→03Resolved [17:47:48] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (GitLab III: GitLab in LA 🪃): Try Reggie for buildkit caching - https://phabricator.wikimedia.org/T323149 (10demon) [18:00:35] 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review, 10Release, 10Train Deployments: 1.40.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T325585 (10WMDE-leszek) [18:11:10] 10Release-Engineering-Team (Radar), 10MediaWiki-Vagrant, 10Parsoid (Tracking): Decide the fate of Parsoid/JS Puppet in Mediawiki-Vagrant - https://phabricator.wikimedia.org/T259988 (10Aklapper) [18:11:27] 10Release-Engineering-Team (Radar), 10MediaWiki-Vagrant, 10Parsoid, 10WMDE-TechWish-Maintenance, 10WMDE-Templates-FocusArea: Support Parsoid/PHP in MediaWiki-Vagrant - https://phabricator.wikimedia.org/T258940 (10Aklapper) 05Open→03Resolved Closing per last comments [18:13:39] 10Deployments, 10Release-Engineering-Team: MediaWiki deploy servers should not be mediawiki installation targets - https://phabricator.wikimedia.org/T329857 (10Krinkle) [18:13:48] 10Deployments, 10Release-Engineering-Team, 10Performance-Team (Radar): MediaWiki deploy servers should not be mediawiki installation targets - https://phabricator.wikimedia.org/T329857 (10Krinkle) [18:26:01] 10Gerrit: Code changes can no longer be saved due to 403 error - https://phabricator.wikimedia.org/T329726 (10Lectrician1) @Ammarpad Why can't users make edits to other people's changes on the web interface? This is actually so annoying. I don't want to be required to clone an entire repo just to make a one-line... [18:41:35] (03PS1) 10Subramanya Sastry: Update wiki-specific CSS based on bug fixes to Cite CSS script [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/889834 [18:42:01] 10Deployments, 10Release-Engineering-Team, 10Performance-Team (Radar): MediaWiki deploy servers should not be mediawiki installation targets - https://phabricator.wikimedia.org/T329857 (10dancy) [18:44:16] 10Gerrit: Code changes can no longer be saved due to 403 error - https://phabricator.wikimedia.org/T329726 (10Aklapper) >>! In T329726#8622633, @Lectrician1 wrote: > Why can't users make edits to other people's changes on the web interface? Because past vandalism and spam [18:45:56] 10Gerrit: Code changes can no longer be saved due to 403 error - https://phabricator.wikimedia.org/T329726 (10Bawolff) Also, i don't think its web interface specific. I think the same thing would happen if you cloned the repo and weren't in the trusted contributors group. [18:51:53] (03CR) 10Subramanya Sastry: "Changes based on output in https://gerrit.wikimedia.org/r/c/mediawiki/services/parsoid/+/889812" [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/889834 (owner: 10Subramanya Sastry) [20:08:05] 10Phabricator, 10serviceops-collab: create aphlict2001 (Phabricator realtime notifications codfw) - https://phabricator.wikimedia.org/T322369 (10Dzahn) The production role has been applied, which is great! Though it looks like next we need a scap deployment to aphlict2001. Probably would be best to add this... [20:26:22] 10Gerrit: Code changes can no longer be saved due to 403 error - https://phabricator.wikimedia.org/T329726 (10Lectrician1) yeah I figured that out [20:45:26] 10Release-Engineering-Team (GitLab V: Event Horizon 🌄): Refactor buildkitd deployment for better build container isolation - https://phabricator.wikimedia.org/T329213 (10dduvall) p:05Triage→03Medium a:03dduvall [21:09:11] 10Scap: Kubernetes configuration file is group-readable - https://phabricator.wikimedia.org/T329899 (10TheresNoTime) [22:35:03] 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review, 10Release, 10Train Deployments: 1.40.0-wmf.23 deployment blockers - https://phabricator.wikimedia.org/T325586 (10dduvall) 05Open→03Resolved [23:57:24] 10Release-Engineering-Team: Deploy phabricator to aphlict2002.codfw.wmnet - https://phabricator.wikimedia.org/T329908 (10eoghan)