[00:07:24] 10Release-Engineering-Team (Next), 10Release, 10Train Deployments: 1.38.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T293953 (10Jdlrobson) [00:07:55] 10Release-Engineering-Team (Next), 10Release, 10Train Deployments: 1.38.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T293953 (10Jdlrobson) T255682 is currently in an undeployable state, but that should be fixed shortly. Just adding it as a blocker as a precaution. [00:52:16] RECOVERY - SSH on contint1001.mgmt is OK: SSH OK - OpenSSH_6.6 (protocol 2.0) https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [03:29:06] legoktm: Worth adding pip-audit support to LibUp? https://twitter.com/di_codes/status/1466109114442989570?s=21 Not sure how much Python we have, but … [05:50:13] 10GitLab, 10Striker, 10Tools: Replace Diffusion integration with Gitlab integration in Striker (toolsadmin) - https://phabricator.wikimedia.org/T296893 (10Xover) @bd808 I still can't find any mention of Toolforge on mw:GitLab or its subpages, so do we need a separate task to set policy for access/groups/dire... [05:52:35] 10GitLab, 10Striker, 10Tools: Replace Diffusion integration with Gitlab integration in Striker (toolsadmin) - https://phabricator.wikimedia.org/T296893 (10Reedy) >>! In T296893#7542832, @Xover wrote: > I also wonder whether this would be an opportunity to migrate from separate developer accounts to general S... [05:55:16] 10Release-Engineering-Team, 10Scap, 10serviceops: Deploy Scap version 4.1.0 - https://phabricator.wikimedia.org/T296867 (10Legoktm) a:03Legoktm [06:58:40] 10Release-Engineering-Team (Doing), 10serviceops: Build MediaWiki images for kubernetes based on php 7.4 - https://phabricator.wikimedia.org/T293996 (10Legoktm) 05Open→03Resolved [08:28:07] !log shutoff deployment-mx02, replaced by deployment-mx03 [08:28:08] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [08:33:24] !log delete deployment-deploy01 [08:33:25] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [08:33:52] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Radar): Migrate deployment-prep away from Debian Stretch to Buster/Bullseye - https://phabricator.wikimedia.org/T278641 (10Majavah) [08:34:34] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Radar), 10Patch-For-Review: Upgrade deployment-prep deploy* hosts to Buster - https://phabricator.wikimedia.org/T278689 (10Majavah) 05Stalled→03Resolved a:03Majavah Resolved. [08:54:54] (03Abandoned) 1020after4: WIP: workboard metrics [releng/ddd] - 10https://gerrit.wikimedia.org/r/708319 (owner: 1020after4) [08:55:21] (03Abandoned) 1020after4: Added packaging (setup.cfg and pyproject.toml) and rearranged files [releng/ddd] - 10https://gerrit.wikimedia.org/r/701620 (owner: 1020after4) [08:57:30] (03CR) 1020after4: [V: 03+2 C: 03+2] Reconfigure Phatality for OpenSearch Dashboards 1.0.0 [releng/phatality] - 10https://gerrit.wikimedia.org/r/742777 (owner: 10Cwhite) [09:31:54] 10Project-Admins: Create project tag for Schematree_recommender - https://phabricator.wikimedia.org/T296599 (10Aklapper) It's really up to you and who is involved in those projects that already have project tags here (and you probably know the folks way better than I do). :) [12:42:19] (03PS14) 10Kosta Harlan: [DNM] CI full run with extensions [integration/quibble] - 10https://gerrit.wikimedia.org/r/742201 [12:55:56] 10Release-Engineering-Team (Deployment Training Requests): Deployment training request for JKieserman - https://phabricator.wikimedia.org/T296024 (10ArielGlenn) This has been done :-) [14:59:59] 10Release-Engineering-Team (Next), 10Release, 10Train Deployments: 1.38.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T293953 (10Jdlrobson) [15:05:19] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Radar): Migrate deployment-prep away from Debian Stretch to Buster/Bullseye - https://phabricator.wikimedia.org/T278641 (10Majavah) [15:42:37] 10GitLab, 10Striker, 10Tools: Replace Diffusion integration with Gitlab integration in Striker (toolsadmin) - https://phabricator.wikimedia.org/T296893 (10bd808) >>! In T296893#7542832, @Xover wrote: > @bd808 I still can't find any mention of Toolforge on mw:GitLab or its subpages Seems reasonable since I j... [15:46:28] 10GitLab: Self-reported GitLab SSH host keys don’t appear to match actual host keys - https://phabricator.wikimedia.org/T296944 (10Lucas_Werkmeister_WMDE) [15:49:12] 10GitLab: Self-reported GitLab SSH host key fingerprints don’t appear to match actual host key fingerprints - https://phabricator.wikimedia.org/T296944 (10Lucas_Werkmeister_WMDE) [15:50:52] 10GitLab: Self-reported GitLab SSH host key fingerprints don’t appear to match actual host key fingerprints - https://phabricator.wikimedia.org/T296944 (10Lucas_Werkmeister_WMDE) [16:05:03] PROBLEM - SSH on contint1001.mgmt is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [16:50:56] (03PS2) 10Ahmon Dancy: values-traindev.yaml: Enable php.devel_mode [tools/train-dev] - 10https://gerrit.wikimedia.org/r/743033 [16:53:56] 10GitLab, 10Striker, 10Tools: Replace Diffusion integration with Gitlab integration in Striker (toolsadmin) - https://phabricator.wikimedia.org/T296893 (10Legoktm) It would be nice if we could automatically synchronize Tool maintainer lists and Git repository owners, whether that's automated via LDAP or manu... [18:05:06] dwisehaupt said he had switch the R/O connection string to point back to the primary DB at 1900 UTC yesterday [18:12:49] oops, wrong channel [18:32:25] 10Beta-Cluster-Infrastructure, 10Discovery-Search: deployment-wdqs01 Puppet failure - https://phabricator.wikimedia.org/T296959 (10Majavah) [18:38:28] 10Release-Engineering-Team (Deployment Training Requests): Deployment training request for JKieserman - https://phabricator.wikimedia.org/T296024 (10thcipriani) 05Open→03Resolved a:05thcipriani→03ArielGlenn ## {icon check-circle-o} Training completed! Hey @JKieserman thanks for attending deployment trai... [18:59:39] 10Release-Engineering-Team, 10Scap, 10serviceops: Deploy Scap version 4.1.0 - https://phabricator.wikimedia.org/T296867 (10Legoktm) Canaries have been upgraded, did a `scap pull` from mwdebug1001, seemed fine. Will do the rest tomorrow. [19:08:34] RECOVERY - SSH on contint1001.mgmt is OK: SSH OK - OpenSSH_6.6 (protocol 2.0) https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [19:42:00] 10Release-Engineering-Team, 10Scap, 10serviceops: Deploy Scap version 4.1.0 - https://phabricator.wikimedia.org/T296867 (10dancy) Thanks. The big test will be the next `scap sync-world`. [20:09:00] thcipriani: this is definitely not the main message that i took from recent discussions, but someone did mention releng/ci folks being resource starved. Is that true today, and if so is it because wmcs people have been telling you 'no'? [20:10:39] 10GitLab (CI & Job Runners), 10Release-Engineering-Team, 10Security Team AppSec, 10Security-Team, and 2 others: Confirm with releng which docker images make the most sense to use - https://phabricator.wikimedia.org/T296805 (10sbassett) a:03sbassett [20:12:22] 10GitLab (CI & Job Runners), 10Release-Engineering-Team, 10Security Team AppSec, 10Security-Team, and 2 others: Confirm with releng which docker images make the most sense to use - https://phabricator.wikimedia.org/T296805 (10sbassett) Hey #release-engineering-team - I know this is probably a terrible week... [20:17:00] andrewbogott: hey, thanks for reaching out, nope: no one has told me know about requesting new wmcs projects or expanding existing projects. One thing I have asked for in the past is access to k8s resources to run CI: we've never figured that out (obviously it's a huge thing to figure out). The fact that our CI instances are long-lived is a challenge that we haven't really figured out how [20:17:01] to reckon with yet: that's the thing I can think of WRT wmcs. Helpful? [20:17:57] thcipriani: it is. I just wanted to make sure you were blocked RIGHT NOW for lack of compute resources. [20:18:12] definitely not, thank you for checking in <3 [20:18:20] k8s-as-a-service is another story :/ Something we'd like to offer but there are many mysteries in that direction [20:18:39] *weren't [20:19:57] totally. And I understand it's not easy, but I'm telling folks: no one has the resources for that but maybe someone should if any team is interested in taking that up :) [20:20:39] "resources" in that sentence meaning "people and time" vs compute [20:20:54] * andrewbogott nods [20:21:25] Michael is reading about k8s-a-a-s /right now/ but there are many steps on the road as you can imagine [20:25:48] 10Continuous-Integration-Config, 10MW-on-K8s, 10MediaWiki-SettingsLoader, 10serviceops-radar, 10Patch-For-Review: Install php-yaml for use by SettingsLoader - https://phabricator.wikimedia.org/T296331 (10Legoktm) The yaml extension should now be enabled on all appserver and api canaries now, which is `mw... [20:26:43] 10Continuous-Integration-Config, 10MW-on-K8s, 10MediaWiki-SettingsLoader, 10serviceops, 10Patch-For-Review: Install php-yaml for use by SettingsLoader - https://phabricator.wikimedia.org/T296331 (10Legoktm) [21:47:29] 10Continuous-Integration-Config, 10LibUp, 10Security-Team, 10Tools: Move php-security-checker.wmcloud.org to Toolforge - https://phabricator.wikimedia.org/T296967 (10Legoktm) [22:28:49] 10Continuous-Integration-Infrastructure, 10DC-Ops, 10netops, 10ops-codfw: DRAC firmware upgrades codfw (was: Flapping codfw management alarm ( contint2001.mgmt/SSH is CRITICAL )) - https://phabricator.wikimedia.org/T283582 (10Dzahn) one case of mw2252.mgmt right now [22:33:33] 10Continuous-Integration-Infrastructure, 10DC-Ops, 10netops, 10ops-codfw: DRAC firmware upgrades codfw (was: Flapping codfw management alarm ( contint2001.mgmt/SSH is CRITICAL )) - https://phabricator.wikimedia.org/T283582 (10Dzahn) @AntiCompositeNumber Not for sure but it seems likely that it could also b... [22:42:38] James_F: that looks pretty great. We have a lot of Python projects but we very rarely pin dependencies... [22:42:57] Ack. [22:49:19] 10Continuous-Integration-Config, 10LibUp, 10Security-Team, 10Tools: Move php-security-checker.wmcloud.org to Toolforge - https://phabricator.wikimedia.org/T296967 (10bd808) https://wikitech.wikimedia.org/wiki/Nova_Resource:Redirects can be used to create a redirect from wmcloud to toolforge in an attempt t... [23:43:35] (03PS1) 10Jeena Huneidi: scap backport: validate changes [tools/scap] - 10https://gerrit.wikimedia.org/r/743272 (https://phabricator.wikimedia.org/T294453)