[01:17:20] 10Release-Engineering-Team (Radar), 10Scap, 10SRE: mwdebug1001 and mwdebug1002 are reliably the last two hosts to finish scap-cdb-rebuild - https://phabricator.wikimedia.org/T203625 (10Legoktm) 05Openβ†’03Resolved I've scapped once or twice recently and didn't notice it being egregiously slow. We can call... [04:28:38] 10Continuous-Integration-Infrastructure, 10OOUI: Demos page for OOUI in php is broken - https://phabricator.wikimedia.org/T297035 (10Legoktm) [05:02:31] 10Phabricator, 10Design: Task symbol for "Open" suggests "alert" and is inconsistent in its placement when task types are also used - https://phabricator.wikimedia.org/T297249 (10AntiCompositeNumber) I'm of the opinion that there doesn't need to be an icon to indicate an open task, as it's the default state. [08:12:51] 10Project-Admins, 10Math, 10Wikibase (3rd party installations): Request for a project board NFDI - https://phabricator.wikimedia.org/T297344 (10Physikerwelt) [08:18:42] 10Project-Admins, 10Math, 10Wikibase (3rd party installations): Request for a project board NFDI - https://phabricator.wikimedia.org/T297344 (10Aklapper) 05Openβ†’03Stalled Hi, please see https://www.mediawiki.org/wiki/Phabricator/Creating_and_renaming_projects for required info [08:41:48] 10Project-Admins, 10Campaign-Tools (Campaign-Tools-Sprint-2): Create project tag for the CampaignEvents extension - https://phabricator.wikimedia.org/T297295 (10Peachey88) 05Openβ†’03Resolved a:03Peachey88 #CampaignEvents has been created, Interested people are welcome to join the project as {icon users} m... [08:51:21] 10Continuous-Integration-Config, 10Release-Engineering-Team (Next), 10MediaWiki-Core-Tests, 10Code-Health, and 6 others: Reduce runtime of MW shared gate Jenkins jobs to 5 min - https://phabricator.wikimedia.org/T225730 (10kostajh) [09:00:51] 10Continuous-Integration-Config, 10Release-Engineering-Team (Next), 10MediaWiki-Core-Tests, 10Code-Health, and 6 others: Reduce runtime of MW shared gate Jenkins jobs to 5 min - https://phabricator.wikimedia.org/T225730 (10kostajh) [09:27:54] (03PS2) 10Hashar: operations-puppet-catalog-compiler: Add FAIL_FAST parameter [integration/config] - 10https://gerrit.wikimedia.org/r/743365 (https://phabricator.wikimedia.org/T296984) (owner: 10David Caro) [09:28:04] 10Release-Engineering-Team (Doing), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.38.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T293953 (10elukey) Hi everybody, there is an alert for Eventgate Analytics External event validation errors for the schema `mediaw... [09:28:40] (03CR) 10Hashar: [C: 03+1] "I have amended to fix the parameter name in the commit message. let me know when we can sync the job update with the ppc new release :)" [integration/config] - 10https://gerrit.wikimedia.org/r/743365 (https://phabricator.wikimedia.org/T296984) (owner: 10David Caro) [10:09:27] (03CR) 10Hashar: [C: 03+2] "Clarified with John, the jobs can be updated before the new pcc release :]" [integration/config] - 10https://gerrit.wikimedia.org/r/743365 (https://phabricator.wikimedia.org/T296984) (owner: 10David Caro) [10:11:22] (03Merged) 10jenkins-bot: operations-puppet-catalog-compiler: Add FAIL_FAST parameter [integration/config] - 10https://gerrit.wikimedia.org/r/743365 (https://phabricator.wikimedia.org/T296984) (owner: 10David Caro) [10:24:23] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Done by Thu 04 Nov 🧟), 10Security-Team, 10User-brennen: Enable runners for projects under gitlab.wikimedia.org security group - https://phabricator.wikimedia.org/T294050 (10Jelto) @sbassett I seems all runners were offline. During work on T295481 I p... [11:13:00] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10LDAP-Access-Requests: Grant Access to ciadmin for dcaro - https://phabricator.wikimedia.org/T297364 (10hashar) Use case is to shepherd the puppet compiler agents and be able to update the puppet compiler whenever needed. +1 [11:17:27] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10LDAP-Access-Requests: Grant Access to ciadmin for dcaro - https://phabricator.wikimedia.org/T297364 (10dcaro) 05Openβ†’03Resolved a:03dcaro Access granted! \o/ [11:19:46] hashar: hi, can I delete xdummy puppet module? https://phabricator.wikimedia.org/T133183#7554509 It seems it used to be on contint [11:20:30] Amir1: are you on an archeological hunt? :) [11:21:04] I'm on "let's clean up this mess" hunt :P [11:21:19] More context: T272559 [11:21:20] T272559: Unused puppet resources audit, 2021 - https://phabricator.wikimedia.org/T272559 [11:21:29] we have so much unused stuff in puppet [11:22:55] (03PS1) 10David Caro: Review access change [software/puppet-compiler] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/745377 [11:22:55] Amir1: but in short yes please drop it ;) [11:23:20] Thanks <3 [11:24:58] hashar: hopefully a quick question that should let me do some of your work for you! https://phabricator.wikimedia.org/T247653#7553885 [13:12:23] 10Continuous-Integration-Infrastructure, 10Patch-For-Review: Add Python 3.9 to Wikimedia CI - https://phabricator.wikimedia.org/T289222 (10hashar) [13:24:27] (03CR) 10Hashar: "We had a related discussion recently with cscott about adding Parsoid as a dependency which has been captured at: https://www.mediawiki.or" [integration/config] - 10https://gerrit.wikimedia.org/r/742939 (https://phabricator.wikimedia.org/T284611) (owner: 10Kosta Harlan) [13:25:56] (03CR) 10Hashar: [C: 03+1] "Nice and well done! Once deployed may you announce the php 8.1 addition to wikitech-l / #engineering-all ? :]" [integration/config] - 10https://gerrit.wikimedia.org/r/743995 (https://phabricator.wikimedia.org/T293509) (owner: 10Jforrester) [13:33:03] 10Project-Admins, 10Math, 10Wikibase (3rd party installations): Request for a project board NFDI - https://phabricator.wikimedia.org/T297344 (10Physikerwelt) 05Stalledβ†’03Open [13:33:30] 10Project-Admins, 10Math, 10Wikibase (3rd party installations): Request for a project board NFDI - https://phabricator.wikimedia.org/T297344 (10Physikerwelt) p:05Triageβ†’03Medium [13:39:06] anyone here could help me with a scap error happening on the beta cluster? `13:33:09 deploy failed: Did not find a file named maps-canaries in search path: ['/srv/deployment/kartotherian/deploy/scap', '/etc/dsh/group'] [13:54:52] mbsantos: https://gerrit.wikimedia.org/r/plugins/gitiles/maps/kartotherian/deploy/+/refs/heads/master/scap/environments/beta/scap.cfg#1 [13:55:20] is wrong, https://wikitech.wikimedia.org/wiki/News/Phasing_out_the_.wmflabs_domain [13:56:03] the deployment server was replaced recently, and the new one thinks it's in eqiad1.wikimedia.cloud and the old did not [14:00:27] nice catch, thanks majavah I'll change that [14:31:36] 10Release-Engineering-Team (Deployment Training Requests): Deployment training request for David Martin - https://phabricator.wikimedia.org/T297321 (10thcipriani) p:05Triageβ†’03Medium a:03thcipriani Hi @DMartin-WMF I've added you to the backport training on Thursday, December 16. See you then! [14:51:45] PROBLEM - SSH on contint1001.mgmt is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [14:59:55] (03CR) 10Jbond: Review access change (031 comment) [software/puppet-compiler] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/745377 (owner: 10David Caro) [15:24:06] 10Phabricator, 10Data-Engineering, 10Data-Engineering-Kanban: Herald rule for Data-Engineering - https://phabricator.wikimedia.org/T295397 (10Ottomata) My PR to have Data-Engineering added has been merged. @Urbanecm can we make a Herald rule to do > if tagged with Analytics-Radar remove tag Data-Engineerin... [15:24:54] 10Phabricator, 10Data-Engineering, 10Data-Engineering-Kanban: Herald rule for Data-Engineering - https://phabricator.wikimedia.org/T295397 (10Ottomata) a:05Ottomataβ†’03Milimetric [15:27:13] 10Phabricator, 10Data-Engineering, 10Data-Engineering-Kanban: Herald rule for Data-Engineering - https://phabricator.wikimedia.org/T295397 (10Urbanecm) @Ottomata @Milimetric Created {H393} for you. Can you verify the rule is correct, and either resolve the task or request changes? [15:34:18] 10Phabricator, 10Data-Engineering, 10Data-Engineering-Kanban: Herald rule for Data-Engineering - https://phabricator.wikimedia.org/T295397 (10Ottomata) > if tagged with Analytics-Radar remove tag Data-Engineering Tested, this works! https://phabricator.wikimedia.org/herald/transcript/4545650/ > if tagged wi... [15:49:59] 10Phabricator, 10Data-Engineering, 10Data-Engineering-Kanban: Herald rule for Data-Engineering - https://phabricator.wikimedia.org/T295397 (10Urbanecm) >>! In T295397#7559997, @Ottomata wrote: >> if tagged with Analytics-Radar remove tag Data-Engineering > Tested, this works! > https://phabricator.wikimedia.... [15:52:41] RECOVERY - SSH on contint1001.mgmt is OK: SSH OK - OpenSSH_6.6 (protocol 2.0) https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [15:53:26] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Done by Thu 04 Nov 🧟), 10Security-Team, 10User-brennen: Enable runners for projects under gitlab.wikimedia.org security group - https://phabricator.wikimedia.org/T294050 (10sbassett) 05Openβ†’03Resolved @Jelto - ah, ok, thanks. Yes, things look go... [15:55:03] 10Release-Engineering-Team (Doing), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.38.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T293953 (10dancy) [15:56:59] 10Phabricator, 10Data-Engineering, 10Data-Engineering-Kanban: Herald rule for Data-Engineering - https://phabricator.wikimedia.org/T295397 (10Ottomata) Ah, it works! https://phabricator.wikimedia.org/T297399 We can resolve this task. [15:57:11] 10Phabricator, 10Data-Engineering, 10Data-Engineering-Kanban: Herald rule for Data-Engineering - https://phabricator.wikimedia.org/T295397 (10Ottomata) 05Openβ†’03Resolved a:05Milimetricβ†’03Ottomata [16:00:51] 10Release-Engineering-Team (Doing), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.38.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T293953 (10dancy) >>! In T293953#7558703, @elukey wrote: > Hi everybody, > > there is an alert for Eventgate Analytics External ev... [16:01:40] 10Release-Engineering-Team (Next), 10Release, 10Train Deployments: 1.38.0-wmf.13 deployment blockers - https://phabricator.wikimedia.org/T293954 (10Jdlrobson) [16:28:47] (03PS3) 10Hashar: jjb: expand -v docker run option and move to end [integration/config] - 10https://gerrit.wikimedia.org/r/740245 [16:32:10] (03CR) 10Hashar: [C: 03+2] jjb: expand -v docker run option and move to end [integration/config] - 10https://gerrit.wikimedia.org/r/740245 (owner: 10Hashar) [16:33:29] (03PS11) 10Hashar: jjb: play with Jinja2 [integration/config] - 10https://gerrit.wikimedia.org/r/739282 [16:34:03] (03Merged) 10jenkins-bot: jjb: expand -v docker run option and move to end [integration/config] - 10https://gerrit.wikimedia.org/r/740245 (owner: 10Hashar) [16:41:05] (03PS12) 10Hashar: jjb: play with Jinja2 [integration/config] - 10https://gerrit.wikimedia.org/r/739282 [16:41:07] (03PS6) 10Hashar: jjb: migrate docker --volume to "volumes" yaml map [integration/config] - 10https://gerrit.wikimedia.org/r/739583 [16:52:54] 10Phabricator, 10Design: Task symbol for "Open" suggests "alert" and is inconsistent in its placement when task types are also used - https://phabricator.wikimedia.org/T297249 (10MBinder_WMF) Barring an alternative icon that universally conveys "open" I feel similarly, though that may be simply what I am used... [16:54:43] (03CR) 10Jforrester: Zuul: Add PHP81 as voting for libraries, PHP extensions etc. (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/743995 (https://phabricator.wikimedia.org/T293509) (owner: 10Jforrester) [17:03:34] 10GitLab (Infrastructure), 10serviceops: Migrate gitlab-test instance to puppet - https://phabricator.wikimedia.org/T297411 (10Jelto) [17:55:33] 10Release-Engineering-Team (Next), 10Release, 10Train Deployments: 1.38.0-wmf.13 deployment blockers - https://phabricator.wikimedia.org/T293954 (10Catrope) ##### Risky Patch! πŸš‚πŸ”₯ * **Change**: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/666434/ "Upgrade Vue to the migration build of Vue 3" * **Summa... [18:10:45] 10Release-Engineering-Team (Doing), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.38.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T293953 (10Catrope) [18:10:56] 10Release-Engineering-Team (Next), 10Release, 10Train Deployments: 1.38.0-wmf.13 deployment blockers - https://phabricator.wikimedia.org/T293954 (10Catrope) [18:11:01] 10Release-Engineering-Team (Doing), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.38.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T293953 (10Catrope) [18:11:09] 10Release-Engineering-Team (Next), 10Release, 10Train Deployments: 1.38.0-wmf.13 deployment blockers - https://phabricator.wikimedia.org/T293954 (10Catrope) [19:31:31] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Next), 10User-brennen: Provision untrusted GitLab job runners to handle user-level projects and merge requests from forks - https://phabricator.wikimedia.org/T297426 (10brennen) [19:33:43] 10Release-Engineering-Team (Yak Shaving πŸƒπŸͺ’), 10Scap, 10Developer Productivity: Fix mwdebug dashboard to show mw and scap messages again - https://phabricator.wikimedia.org/T288576 (10mmodell) After playing with kibana for a few minutes, I can't figure out any search criteria that actually does match scap log... [19:33:49] 10Release-Engineering-Team (Doing), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.38.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T293953 (10dancy) [19:43:39] 10Phabricator, 10Release-Engineering-Team (Yak Shaving πŸƒπŸͺ’), 10User-MModell: On Phabricator workboard, show status of associated Gerrit patches - https://phabricator.wikimedia.org/T215148 (10mmodell) So I'm looking into integrating something like this into phabricator proper. I also want to figure out how to... [19:52:40] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Next), 10User-brennen: Provision untrusted GitLab job runners to handle user-level projects and merge requests from forks - https://phabricator.wikimedia.org/T297426 (10brennen) p:05Triageβ†’03Medium [19:57:25] PROBLEM - SSH on contint1001.mgmt is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [20:49:26] 10Release-Engineering-Team (Deployment Training Requests): Deployment training request for David Martin - https://phabricator.wikimedia.org/T297321 (10DMartin-WMF) Thanks, @thcipriani . I'm looking forward to it! [20:58:29] RECOVERY - SSH on contint1001.mgmt is OK: SSH OK - OpenSSH_6.6 (protocol 2.0) https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [21:49:48] 10Release-Engineering-Team (Doing), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.38.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T293953 (10dancy) [22:07:41] 10Release-Engineering-Team (Radar), 10WVUI, 10Release Pipeline (Blubber): Use npm version 6.x on blubber-generated docker images - https://phabricator.wikimedia.org/T269957 (10Volker_E) [22:20:27] (03PS1) 10Ahmon Dancy: Use multiple processes to generate git_info cache [tools/scap] - 10https://gerrit.wikimedia.org/r/745615 (https://phabricator.wikimedia.org/T297324) [22:22:55] (03PS2) 10Ahmon Dancy: Use multiple processes to generate git_info cache [tools/scap] - 10https://gerrit.wikimedia.org/r/745615 (https://phabricator.wikimedia.org/T297324) [22:22:59] (03CR) 10jerkins-bot: [V: 04-1] Use multiple processes to generate git_info cache [tools/scap] - 10https://gerrit.wikimedia.org/r/745615 (https://phabricator.wikimedia.org/T297324) (owner: 10Ahmon Dancy) [22:23:19] (03CR) 10Ahmon Dancy: [C: 04-1] Use multiple processes to generate git_info cache [tools/scap] - 10https://gerrit.wikimedia.org/r/745615 (https://phabricator.wikimedia.org/T297324) (owner: 10Ahmon Dancy) [22:23:40] (03CR) 10jerkins-bot: [V: 04-1] Use multiple processes to generate git_info cache [tools/scap] - 10https://gerrit.wikimedia.org/r/745615 (https://phabricator.wikimedia.org/T297324) (owner: 10Ahmon Dancy) [22:24:58] (03PS3) 10Ahmon Dancy: Use multiple processes to generate git_info cache [tools/scap] - 10https://gerrit.wikimedia.org/r/745615 (https://phabricator.wikimedia.org/T297324) [22:25:33] (03CR) 10jerkins-bot: [V: 04-1] Use multiple processes to generate git_info cache [tools/scap] - 10https://gerrit.wikimedia.org/r/745615 (https://phabricator.wikimedia.org/T297324) (owner: 10Ahmon Dancy) [22:32:18] (03PS4) 10Ahmon Dancy: Use multiple processes to generate git_info cache [tools/scap] - 10https://gerrit.wikimedia.org/r/745615 (https://phabricator.wikimedia.org/T297324) [22:38:03] 10Release-Engineering-Team (Yak Shaving πŸƒπŸͺ’), 10Scap, 10Patch-For-Review: Speed up cache_git_info - https://phabricator.wikimedia.org/T297324 (10dancy) [22:41:51] 10Release-Engineering-Team (Yak Shaving πŸƒπŸͺ’), 10Scap, 10Patch-For-Review: Speed up cache_git_info - https://phabricator.wikimedia.org/T297324 (10dancy) [22:41:59] 10Release-Engineering-Team (Yak Shaving πŸƒπŸͺ’), 10Scap: Stop trying to avoid rsyncing l10n CDB files - https://phabricator.wikimedia.org/T297326 (10dancy) [23:22:23] 10GitLab (Infrastructure), 10Release-Engineering-Team (Radar), 10serviceops: Self-reported GitLab SSH host key fingerprints don’t appear to match actual host key fingerprints - https://phabricator.wikimedia.org/T296944 (10Dzahn) Brennen got that right, there are 2 separate SSH daemons on each gitlab server.... [23:37:08] 10GitLab (Infrastructure), 10Release-Engineering-Team (Yak Shaving πŸƒπŸͺ’), 10serviceops, 10Upstream: Self-reported GitLab SSH host key fingerprints don’t appear to match actual host key fingerprints - https://phabricator.wikimedia.org/T296944 (10brennen) > I am tempted to call this a minor bug or improvement... [23:39:02] 10GitLab (Infrastructure), 10Release-Engineering-Team (Yak Shaving πŸƒπŸͺ’), 10serviceops, 10Upstream: Self-reported GitLab SSH host key fingerprints don’t appear to match actual host key fingerprints - https://phabricator.wikimedia.org/T296944 (10Dzahn) Cool! Yea, so our case is we have 2 IP addresses on the s... [23:43:10] (03CR) 10Jeena Huneidi: "small nit" [tools/scap] - 10https://gerrit.wikimedia.org/r/745615 (https://phabricator.wikimedia.org/T297324) (owner: 10Ahmon Dancy) [23:47:30] (03PS5) 10Ahmon Dancy: Use multiple processes to generate git_info cache [tools/scap] - 10https://gerrit.wikimedia.org/r/745615 (https://phabricator.wikimedia.org/T297324) [23:48:06] (03CR) 10Ahmon Dancy: Use multiple processes to generate git_info cache (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/745615 (https://phabricator.wikimedia.org/T297324) (owner: 10Ahmon Dancy)