[01:36:12] PROBLEM - Check systemd state on doc1001 is CRITICAL: CRITICAL - degraded: The following units failed: rsync-doc-doc1002.eqiad.wmnet.service,rsync-doc-doc2001.codfw.wmnet.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [01:55:51] ^ manually started. should recover now [01:55:56] /me away [02:21:28] RECOVERY - Check systemd state on doc1001 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [04:19:24] pushed a fix for the wikimedia auth extension to remove syntax usage that's deprecated in php 7.4 [04:19:49] the wikimedia auth extension for phabricator, that is [04:57:03] 10Phabricator: Create a phabricator blog for Language team - https://phabricator.wikimedia.org/T306329 (10Arrbee) Thank you and it would be good to have an ACL for the Language team. How can we have one? [06:34:16] 10Phabricator: Create a phabricator blog for Language team - https://phabricator.wikimedia.org/T306329 (10abi_) Please also provide access for @abi_ [07:24:46] mutante: thanks :) I have no idea why puppet would have been disabled though [07:25:07] I usually `!log` those but might have missed it :D [08:10:12] 10GitLab (Administration, Settings & Policy), 10Release-Engineering-Team (Next), 10User-brennen: Establish a routine GitLab deployment / update window - https://phabricator.wikimedia.org/T287117 (10Jelto) We discussed in last ITC meeting that a dedicated GitLab update and maintenance window is not needed now... [08:10:46] 10GitLab (Infrastructure), 10Data-Persistence-Backup, 10serviceops, 10Patch-For-Review, 10User-brennen: Backups for GitLab - https://phabricator.wikimedia.org/T274463 (10Jelto) >>! In T274463#7835100, @jcrespo wrote: > [...] > BTW, restores directly to a different host are possible, although a bit cumber... [08:14:56] (03CR) 10Jaime Nuche: [C: 03+2] deploy-promote: migrate script to `tools/scap` repository [tools/release] - 10https://gerrit.wikimedia.org/r/769026 (https://phabricator.wikimedia.org/T302488) (owner: 10Jaime Nuche) [08:15:53] (03Merged) 10jenkins-bot: deploy-promote: migrate script to `tools/scap` repository [tools/release] - 10https://gerrit.wikimedia.org/r/769026 (https://phabricator.wikimedia.org/T302488) (owner: 10Jaime Nuche) [08:19:56] 10GitLab (CI & Job Runners), 10serviceops: upgrade gitlab-runners to bullseye - https://phabricator.wikimedia.org/T297659 (10Jelto) >>! In T297659#7862187, @Dzahn wrote: > @Jelto All the (non-protected) prod runners are upgraded. Now I was just wondering about the 2 protected runners. They are paused. Should... [08:20:12] 10GitLab (CI & Job Runners), 10serviceops: upgrade gitlab-runners to bullseye - https://phabricator.wikimedia.org/T297659 (10Jelto) [08:31:57] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Radar), 10Security-Team, 10serviceops, and 2 others: Setup GitLab Runner in trusted environment - https://phabricator.wikimedia.org/T295481 (10Jelto) [09:03:22] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Doing): Add concurrent parameter to profile::gitlab::runner - https://phabricator.wikimedia.org/T293833 (10Jelto) 05Openβ†’03Resolved This has been implemented in https://gerrit.wikimedia.org/r/732093, I'm closing this task. [09:23:03] 10Release-Engineering-Team, 10Scap: Allow Scap to push to Gerrit without operator creds - https://phabricator.wikimedia.org/T306425 (10jnuche) [09:24:29] 10Release-Engineering-Team (Doing), 10Scap, 10User-brennen: scap deploy-promote fails on git push - https://phabricator.wikimedia.org/T304557 (10jnuche) 05Openβ†’03Resolved [09:28:52] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen), 10Cloud-VPS (Debian Stretch Deprecation), 10Patch-For-Review: Move all Wikimedia CI (WMCS integration project) instances from stretch to buster/bullseye - https://phabricator.wikimedia.org/T252071 (10hashar) >>! In T252071#7861196,... [09:30:30] 10Continuous-Integration-Infrastructure, 10Zuul: zuul-merger takes a while to recreate repository branches - https://phabricator.wikimedia.org/T220606 (10hashar) [09:30:32] 10Release-Engineering-Team (🌱 Spring Cleaning β€” April 2022): Delete wmf branches from Gerrit repositories - https://phabricator.wikimedia.org/T303828 (10hashar) [09:32:09] 10Continuous-Integration-Infrastructure, 10Zuul: zuul-merger takes a while to recreate repository branches - https://phabricator.wikimedia.org/T220606 (10hashar) 05Openβ†’03Resolved a:03hashar git configuration has been tweaked for the zuul-merger. The rest of the improvement will be achieved by deleting o... [09:35:26] 10Release-Engineering-Team (🌱 Spring Cleaning β€” April 2022): Delete wmf branches from Gerrit repositories - https://phabricator.wikimedia.org/T303828 (10hashar) a:05hasharβ†’03None I am unassigning myself since I have way too many tasks to juggle with. To resume I think we would need to: A) decide whether we... [09:45:39] (03CR) 10Hashar: "> In production this means that two additional servers will be used to spread the rsync load." [tools/scap] - 10https://gerrit.wikimedia.org/r/779962 (https://phabricator.wikimedia.org/T305466) (owner: 10Ahmon Dancy) [09:48:22] catching up on my gerrit notifications and I see a gem by dancy & jnuche to fix scap low level stuff https://gerrit.wikimedia.org/r/c/mediawiki/tools/scap/+/779559 [09:48:24] well done :] [09:50:57] 10Release-Engineering-Team (Next), 10Scap: scap proxies are CPU and/or network bound - https://phabricator.wikimedia.org/T305466 (10hashar) Coming back from vacations and seeing low level network issue being addressed is definitely 5 stars worthy. That is great! ⭐ ⭐ ⭐ ⭐ ⭐ [11:01:12] hashar: that was dancy, I just reviewed the changes. I do agree it was a really cool fix :) [11:55:56] 10Phabricator: De-link my aodit@wikimedia.org staff email from personal volunteer profile - https://phabricator.wikimedia.org/T305919 (10Aklapper) For reasons I do not know, all authentication methods were removed from https://phabricator.wikimedia.org/p/Astuthiodit_1/ so that account has become inaccessible. :(... [12:06:36] 10Beta-Cluster-Infrastructure, 10Abstract Wikipedia team, 10Patch-For-Review: Create a Beta Cluster version of Wikifunctions.org - https://phabricator.wikimedia.org/T284162 (10ori) What does "re-jig the services to actually expose them under a useful name" mean? [12:07:32] 10Phabricator: De-link my aodit@wikimedia.org staff email from personal volunteer profile - https://phabricator.wikimedia.org/T305919 (10jcrespo) a:03jcrespo @Aklapper thanks for the research- I will return soon from lunch, and having confirmed we have fresh Phabricator db backups, I will try to fix as suggest... [12:25:36] 10Phabricator: De-link my aodit@wikimedia.org staff email from personal volunteer profile - https://phabricator.wikimedia.org/T305919 (10Samwalton9) Astuthi and I went through this this morning and were able to leverage Gmail aliases to register her staff account at @AOdit_WMF and volunteer account at @Astuthi_O... [12:32:31] (03PS1) 10Majavah: Update to only support helm 3 [releng/local-charts] - 10https://gerrit.wikimedia.org/r/784251 [13:02:47] (03CR) 10Hashar: [C: 03+2] zuul: add new mail for Zabe [integration/config] - 10https://gerrit.wikimedia.org/r/783450 (owner: 10Zabe) [13:04:38] (03Merged) 10jenkins-bot: zuul: add new mail for Zabe [integration/config] - 10https://gerrit.wikimedia.org/r/783450 (owner: 10Zabe) [13:05:05] (03CR) 10Hashar: "I have deployed the change" [integration/config] - 10https://gerrit.wikimedia.org/r/783450 (owner: 10Zabe) [13:13:59] jnuche: for puppet dev environment jbond could probably help he has done a fair amount of that front ;) [13:14:40] I am guessing some people from SRE can give you a tour probably [13:26:10] hashar: thanks, I already asked a couple of people and they gave me some suggestions [14:00:39] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team, 10SecTeam-Processed, and 2 others: Jenkins plugins security advisory - 2022-04-12 - https://phabricator.wikimedia.org/T306418 (10sbassett) [14:00:41] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team, 10SecTeam-Processed, and 2 others: Jenkins plugins security advisory - 2022-04-12 - https://phabricator.wikimedia.org/T306418 (10sbassett) p:05Triageβ†’03Low [14:01:18] 10Beta-Cluster-Infrastructure, 10MonoBook, 10Beta-Cluster-reproducible: 160px empty space above the Betacommons logo on Monobook - https://phabricator.wikimedia.org/T306436 (10AlexisJazz) [14:12:28] 10Phabricator, 10Project-Admins, 10Release-Engineering-Team (Radar), 10Security-Team, and 2 others: Move the #acl_security_volunteer policy outside of #acl_security - https://phabricator.wikimedia.org/T305890 (10sbassett) >>! In T305890#7858154, @thcipriani wrote: > This fights the model of phabricator a b... [14:17:12] 10Phabricator: De-link my aodit@wikimedia.org staff email from personal volunteer profile - https://phabricator.wikimedia.org/T305919 (10jcrespo) I ran the following command: ` UPDATE user_email SET address = 'T305919@example.org' WHERE address = 'aodit@wikimedia.org' LIMIT 1; ` The regular email, if @Aklapper... [14:19:27] 10Phabricator: De-link my aodit@wikimedia.org staff email from personal volunteer profile - https://phabricator.wikimedia.org/T305919 (10jcrespo) a:05jcrespoβ†’03Samwalton9 [14:28:28] 10Beta-Cluster-Infrastructure, 10MonoBook, 10Beta-Cluster-reproducible: 160px empty space above the Betacommons logo on Monobook - https://phabricator.wikimedia.org/T306436 (10Deepanshu039) I would like to work on this issue. [14:51:59] 10GitLab (Administration, Settings & Policy), 10Release-Engineering-Team (Next), 10User-brennen: Establish a routine GitLab deployment / update window - https://phabricator.wikimedia.org/T287117 (10brennen) 05Openβ†’03Declined Makes sense. We can revisit this in future if needed. [14:54:53] 10Beta-Cluster-Infrastructure, 10Abstract Wikipedia team, 10Patch-For-Review: Create a Beta Cluster version of Wikifunctions.org - https://phabricator.wikimedia.org/T284162 (10Jdforrester-WMF) >>! In T284162#7863979, @ori wrote: > What does "re-jig the services to actually expose them under a useful name" me... [15:17:50] (03CR) 10Bernard Wang: Add selenium daily job to Vector (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/777884 (https://phabricator.wikimedia.org/T301184) (owner: 10Jdlrobson) [15:32:19] 10Gerrit, 10wikitech.wikimedia.org: Wikitech->Gerrit account block and unblock has stopped working - https://phabricator.wikimedia.org/T306297 (10hashar) The `wmfGerritSetActive()` function is invoked by MediaWiki hooks `BlockIPComplete` and `UnblockUserComplete`. The hooks are being passed the MediaWiki usern... [15:48:52] 10Beta-Cluster-Infrastructure, 10MonoBook, 10Beta-Cluster-reproducible: 160px empty space above the Betacommons logo on Monobook - https://phabricator.wikimedia.org/T306436 (10Aklapper) @Deepanshu039 Hi and welcome. Feel free to, and please check https://www.mediawiki.org/wiki/New_Developers#Some_general_com... [15:51:37] 10Gerrit, 10wikitech.wikimedia.org: Wikitech->Gerrit account block and unblock has stopped working - https://phabricator.wikimedia.org/T306297 (10hashar) From `/a/accounts/Fomafix/external.ids` ` lang=json { "identity": "mailto:fomafix@googlemail.com", "email_address": "fomafix@googlemail.com", "... [16:04:55] 10Beta-Cluster-Infrastructure, 10MediaWiki-Search, 10PageImages, 10Readers-Web-Backlog, and 3 others: PageImages ignores MediaWiki:Bad image list, (uses Pageimages-denylist_test instead) displaying search results that are inappropriate for some readers - https://phabricator.wikimedia.org/T306246 (10Ahecht) [17:03:54] (03PS6) 10Jdlrobson: Add selenium daily job to Vector [integration/config] - 10https://gerrit.wikimedia.org/r/777884 (https://phabricator.wikimedia.org/T301184) [17:06:46] 10Beta-Cluster-Infrastructure, 10SRE, 10Traffic, 10Beta-Cluster-reproducible: Beta cluster down: Error: 502, Next Hop Connection Failed (Feb 2022) - https://phabricator.wikimedia.org/T302699 (10dom_walden) This is happening again. I am also seeing: ` Request from 52.225.87.246 via deployment-cache-text06 d... [17:14:10] hello! we have this patch that needs to go out with wmf.8. I can still cherry pick to that branch, right, since it isn't live yet? https://gerrit.wikimedia.org/r/c/mediawiki/core/+/783911 [17:14:36] Dmaza apparently doesn't have +2 rights for this patch, I guess because it's a release branch. So hoping someone else could +2 for me [17:14:46] otherwise we'll backport it later [17:20:13] 10Beta-Cluster-Infrastructure, 10SRE, 10Traffic, 10Beta-Cluster-reproducible: Beta cluster down: Error: 502, Next Hop Connection Failed (Feb 2022) - https://phabricator.wikimedia.org/T302699 (10Zabe) ` zabe@deployment-mediawiki12:~$ sudo tail /var/log/apache2.log Apr 19 17:13:55 deployment-mediawiki12 apac... [17:20:21] taavi: ^ [17:21:05] RhinosF1: I'm not touching that, if people care about deployment-prep working they should actually contribute to maintaining it [17:29:10] taavi: no, musikanimal's question [17:29:34] that should be directed to the train conductors, not me [17:30:42] from T305214 it appears that would be brennen [17:30:43] T305214: 1.39.0-wmf.8 deployment blockers - https://phabricator.wikimedia.org/T305214 [17:33:29] it's not a big deal either way, we will backport in ~2.5 hours if https://gerrit.wikimedia.org/r/c/mediawiki/core/+/783911 isn't merged. It is *possible* we'll have some corrupt log entries during the interim but as a new feature and only being on group0, chances are somewhat slim [17:34:50] jeena: fyi ^ [17:38:08] musikanimal: Short version: No. Don't merge into a production branch without immediately deploying it, please. Even if you think it's fine. [17:39:59] I am currently attempting to deploy to testwikis but won't deploy to group0 until the blocker is fixed [17:40:18] is the beta cluster down for anyone else? [17:40:50] MatmaRex: not working for me [17:40:50] MatmaRex: Yes. T302699 [17:40:51] T302699: Beta cluster down: Error: 502, Next Hop Connection Failed (Feb 2022) - https://phabricator.wikimedia.org/T302699 [17:41:48] oh, i couldn't find that because it's an old task. thanks [17:41:54] musikanimal: you can go ahead and cherry pick since I haven't deployed anything yet [17:42:12] MatmaRex: dwalden was fiddling just now. Possibly SRE re-did something about Varnish in puppet that broke everything? [17:42:20] we'll need to check out the change on the deploy box since that branch is already checked out [17:42:44] jeena: okay thanks. I just need someone other than me to +2 https://gerrit.wikimedia.org/r/c/mediawiki/core/+/783911/ [17:42:47] but let us know when merged and we can do that [17:44:25] also happy to wait until the backport window if this too complicated :) [17:44:26] actually musikanimal that is not listed as a blocker on the train task so It would be better to wait until the backport window [17:47:50] okay yeah, that was going to be my next question -- whether this is truly worthy of being a train blocker. If an admin uses the new "delete associated talk page" option (which goes out on this train) and there's a template in the deletion reason, the log entry is permanently corrupted with the contents of the template. Definitely don't want to ship this beyond group0 but otherwise it's not that big of a deal [17:54:31] hmm, maybe asking on the task would get an answer [17:55:14] sorry for the delay, I'm dealing with some problems trying to prepare for deployment [17:57:45] 10Release-Engineering-Team (🌱 Spring Cleaning β€” April 2022), 10Patch-For-Review, 10Release, 10Train Deployments: 1.39.0-wmf.8 deployment blockers - https://phabricator.wikimedia.org/T305214 (10MusikAnimal) [17:58:09] okay, going to err on the side of caution... I've added it as a train blocker. cc jeena [17:59:13] πŸ‘ We won't deploy to group0 until both blockers are resolved anyway [17:59:36] okay great. thanks everyone for the help! [18:00:42] just to clarify, T305214 issue has been solved. It is just not part of 1.39.0-wmf.8 [18:00:43] T305214: 1.39.0-wmf.8 deployment blockers - https://phabricator.wikimedia.org/T305214 [18:41:06] 10Phabricator, 10Project-Admins, 10Release-Engineering-Team (Radar), 10Security-Team, and 2 others: Move the #acl_security_volunteer policy outside of #acl_security - https://phabricator.wikimedia.org/T305890 (10DannyS712) >>! In T305890#7864187, @sbassett wrote: >>>! In T305890#7858154, @thcipriani wrote:... [18:55:28] (03CR) 10Thcipriani: "We should get this deployedβ€”tripped over it today in train πŸ˜‚" [tools/scap] - 10https://gerrit.wikimedia.org/r/781059 (owner: 10Ahmon Dancy) [19:43:07] just a reminder that we only need at +2 at https://gerrit.wikimedia.org/r/c/mediawiki/core/+/783911/ then I guess for you to check it out on the deploy box [19:52:36] musikanimal: can you add it to the backport schedule? [19:52:48] sure [19:52:50] thanks! [19:55:31] it's T306431 right? [19:55:32] T306431: Templates get transcluded in (un)delete reason for associated talk page - https://phabricator.wikimedia.org/T306431 [19:56:28] jeena: yes [19:59:26] done [20:32:31] 10Release-Engineering-Team (🌱 Spring Cleaning β€” April 2022), 10Patch-For-Review, 10Release, 10Train Deployments: 1.39.0-wmf.8 deployment blockers - https://phabricator.wikimedia.org/T305214 (10MusikAnimal) [20:57:09] Thanks for taking care of the bug dmaza musikanimal ! [20:57:17] thank you! [21:18:48] 10Continuous-Integration-Infrastructure, 10MinervaNeue, 10Vector, 10Accessibility, and 3 others: Add automated accessibility tests in CI to generate accessibility benchmarks for Skins - https://phabricator.wikimedia.org/T301184 (10Jdlrobson) a:03zeljkofilipin We need help from Zeljko to merge https://ger... [21:23:43] 10Beta-Cluster-Infrastructure, 10Beta-Cluster-reproducible: 160px empty space above the Betacommons logo on Monobook - https://phabricator.wikimedia.org/T306436 (10matmarex) Caused by https://commons.wikimedia.beta.wmflabs.org/wiki/MediaWiki:Gadget-betaCommons.css [21:23:47] 10Beta-Cluster-Infrastructure, 10Beta-Cluster-reproducible: 160px empty space above the Betacommons logo on Monobook - https://phabricator.wikimedia.org/T306436 (10matmarex) [21:27:55] jeena: thank you!! [21:28:41] 10Release-Engineering-Team (🌱 Spring Cleaning β€” April 2022), 10Patch-For-Review, 10Release, 10Train Deployments: 1.39.0-wmf.8 deployment blockers - https://phabricator.wikimedia.org/T305214 (10Zabe) [21:51:57] 10Beta-Cluster-Infrastructure, 10Beta-Cluster-reproducible: 160px empty space above the Betacommons logo on Monobook - https://phabricator.wikimedia.org/T306436 (10AlexisJazz) >>! In T306436#7866172, @matmarex wrote: > Caused by https://commons.wikimedia.beta.wmflabs.org/wiki/MediaWiki:Gadget-betaCommons.css... [22:20:42] 10Beta-Cluster-Infrastructure, 10Beta-Cluster-reproducible: 160px empty space above the Betacommons logo on Monobook - https://phabricator.wikimedia.org/T306436 (10matmarex) >>! In T306436#7866231, @AlexisJazz wrote: > Thanks. I have a hack ready if nobody is interested in fixing it some other way, but I just... [22:26:36] 10Beta-Cluster-Infrastructure, 10Beta-Cluster-reproducible: 160px empty space above Betacommons logo on Monobook (due to local MediaWiki:Gadget-betaCommons.css) - https://phabricator.wikimedia.org/T306436 (10Aklapper) [22:28:35] 10Beta-Cluster-Infrastructure, 10Beta-Cluster-reproducible: 160px empty space above Betacommons logo on Monobook (due to local MediaWiki:Gadget-betaCommons.css) - https://phabricator.wikimedia.org/T306436 (10matmarex) 05Openβ†’03Resolved a:03matmarex Fixed both problems. * https://commons.wikimedia.beta.wm...