[00:04:16] (03PS1) 10Zabe: Archive PoolCounter extension [integration/config] - 10https://gerrit.wikimedia.org/r/890197 [00:04:40] (03PS2) 10Zabe: Archive PoolCounter extension [integration/config] - 10https://gerrit.wikimedia.org/r/890197 (https://phabricator.wikimedia.org/T324631) [00:07:34] (03CR) 10Krinkle: [C: 03+2] Archive PoolCounter extension [integration/config] - 10https://gerrit.wikimedia.org/r/890197 (https://phabricator.wikimedia.org/T324631) (owner: 10Zabe) [00:08:39] (03Merged) 10jenkins-bot: Archive PoolCounter extension [integration/config] - 10https://gerrit.wikimedia.org/r/890197 (https://phabricator.wikimedia.org/T324631) (owner: 10Zabe) [00:21:19] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/890197 [00:21:20] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [06:32:08] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team: Upgrade Jenkins to latest LTS 2.375.3 - https://phabricator.wikimedia.org/T330045 (10hashar) [08:05:47] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team: Upgrade Jenkins to latest LTS 2.375.3 - https://phabricator.wikimedia.org/T330045 (10hashar) From https://debmonitor.wikimedia.org/packages/jenkins : {F36862012 size=full} @jnuche and I will proceed with the upgrades at some po... [08:34:02] 10Continuous-Integration-Infrastructure, 10MediaWiki-Core-Tests, 10MW-1.35-release: Wikimedia CI is failing for REL1_35 (both core and extension repos) due to missing wikimedia/avro package - https://phabricator.wikimedia.org/T329608 (10hashar) [08:46:12] 10Continuous-Integration-Infrastructure, 10MediaWiki-Core-Tests, 10MW-1.35-release: Wikimedia CI is failing for REL1_35 (both core and extension repos) due to missing wikimedia/avro package - https://phabricator.wikimedia.org/T329608 (10hashar) > wikimedia/avro went away on github (possibly due to T278569, a... [09:01:49] 10Continuous-Integration-Config, 10MediaWiki-Codesniffer, 10MediaWiki-Core-Tests: Consider enforcing usage of MediaWiki(Unit|Integration)TestCase in MW PHPUnit Test - https://phabricator.wikimedia.org/T319536 (10kostajh) >>! In T319536#8300292, @Daimona wrote: > Sounds good to me. In the past I've used Inte... [09:07:03] 10Phabricator, 10Release-Engineering-Team (Radar), 10Patch-For-Review, 10User-brennen: Show "other assignee" avatar on tasks in workboard - https://phabricator.wikimedia.org/T329974 (10Aklapper) [09:51:49] 10Phabricator: Help with Phabricator MFA - https://phabricator.wikimedia.org/T329964 (10AdesojiThisDot) I want to move it because I am leaving the consultancy I currently work at and won't have access to that email anymore [09:52:53] 10Phabricator: Help with Phabricator MFA - https://phabricator.wikimedia.org/T329964 (10Aklapper) @AdesojiThisDot: I'm wondering why you would not want to create a separate, personal account in that case? [10:03:13] Hey :) Do y'all know when https://wikitech.wikimedia.org/wiki/Deployments rolls over to the next week? [10:23:03] 10Continuous-Integration-Config, 10MediaWiki-Codesniffer, 10MediaWiki-Core-Tests: Consider enforcing usage of MediaWikiUnitTestCase and MediaWikiIntegrationTestCase in MW PHPUnit Test - https://phabricator.wikimedia.org/T319536 (10hashar) [10:24:53] 10Phabricator, 10Release-Engineering-Team, 10serviceops-collab: phd on phab2002 does not start or should be masked - https://phabricator.wikimedia.org/T329285 (10hashar) Looks good @Dzahn , and thank you for having taken care of the devtools instance! [10:26:22] 10Release-Engineering-Team, 10Scap: scap.cfg hostname config sections are not applied - https://phabricator.wikimedia.org/T329144 (10hashar) 05Open→03Resolved The specific fix is https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/71/diffs?commit_id=d6e07a0de5897f172802aea57dac82327a16ae58 , it... [10:29:18] 10GitLab (Infrastructure), 10serviceops-collab, 10Datacenter-Switchover: Switchover gitlab-replica (gitlab2002 -> gitlab1003) - https://phabricator.wikimedia.org/T329930 (10Jelto) [10:31:11] 10GitLab (Infrastructure), 10serviceops-collab, 10Datacenter-Switchover: Switchover gitlab-replica (gitlab2002 -> gitlab1003) - https://phabricator.wikimedia.org/T329930 (10Jelto) [10:32:18] 10Scap, 10serviceops: Kubernetes configuration file is group-readable - https://phabricator.wikimedia.org/T329899 (10Clement_Goubert) [10:36:29] 10Gerrit, 10serviceops-collab, 10Patch-For-Review: Issues with Gerrit test instance - https://phabricator.wikimedia.org/T329444 (10hashar) I am not sure `gerrit-prod-1001` is in a workable state currently. It has some mixed changes in it to prepare for some operations (changing the user, moving files around)... [10:36:34] 10Gerrit, 10serviceops-collab, 10Patch-For-Review: Issues with Gerrit test instance - https://phabricator.wikimedia.org/T329444 (10hashar) 05Open→03Declined [12:19:20] 10Phabricator: Help with Phabricator MFA - https://phabricator.wikimedia.org/T329964 (10SimoneThisDot) Hi Andre, Just to let you know, the Adeosji request above has been initiated by Adam Baso request to ensure that we do not leave situations in which "Stale emails" are associated with phabricator account. As... [12:24:18] 10Phabricator: Reset Phabricator 2FA for matthiasmullie - https://phabricator.wikimedia.org/T330073 (10matthiasmullie) [12:43:56] claime: it should automatically [12:44:04] Blame thcipriani if bot is broken [12:46:15] Thanks RhinosF1, I'll see with him when he comes in if it still hasn't rolled over, for now it is stuck displaying last week + this week [12:46:48] claime: it should have at 2am on Saturday I believe [12:49:24] (03CR) 10Zfilipin: [C: 03+2] zuul: [mediawiki/extensions/DismissableSiteNotice] Enable Sonar Codehealth (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/889197 (https://phabricator.wikimedia.org/T321837) (owner: 10Pwangai) [12:54:08] 10GitLab (Infrastructure), 10serviceops-collab, 10Datacenter-Switchover: Switchover gitlab-replica (gitlab2002 -> gitlab1003) - https://phabricator.wikimedia.org/T329930 (10Jelto) [13:50:15] 10GitLab (Infrastructure), 10serviceops-collab, 10Datacenter-Switchover: Switchover gitlab-replica (gitlab2002 -> gitlab1003) - https://phabricator.wikimedia.org/T329930 (10Jelto) [14:40:30] 10Scap, 10serviceops: Kubernetes configuration file is group-readable - https://phabricator.wikimedia.org/T329899 (10JMeybohm) This has been discussed in helm a couple of time already (latest incarnation is https://github.com/helm/helm/issues/11083). We're having those files group-readable on purpose to allow... [14:43:31] 10Scap, 10serviceops: Kubernetes configuration file is group-readable - https://phabricator.wikimedia.org/T329899 (10Clement_Goubert) This is not a big deal, but I understand the confusion. As mentioned in the irc discussion, a possible course of action is to make these files `0600` and have `scap` use `sudo`... [14:44:07] 10Scap, 10MW-on-K8s, 10serviceops: Kubernetes configuration file is group-readable - https://phabricator.wikimedia.org/T329899 (10Clement_Goubert) [15:11:06] o/ [15:11:38] got a user having issues with the deployment pipeline and I seem unable to figure out what's up. [15:11:42] context is: https://gerrit.wikimedia.org/r/c/mediawiki/services/mathoid/+/866662 [15:11:57] and the console output is at https://integration.wikimedia.org/ci/job/mathoid-pipeline-rehearse/46/console [15:12:44] and it apparently failes to run helm install [15:14:21] ah, found it [15:14:23] 8m30s Warning Failed pod/mathoid-8u4mbqgt-c8944958c-2tngk Error: failed to start container "mathoid-8u4mbqgt": Error response from daemon: OCI runtime create failed: container_linux.go:367: starting container process caused: exec: "nodejs": executable file not found in $PATH: unknown [15:17:26] ok, I think I know how to fix it, however this wasn't visible to the dev/deployer, we need to figure how to fix that [15:19:47] 10GitLab (Infrastructure), 10serviceops-collab: ensure Gitlab logs end up in logstash - https://phabricator.wikimedia.org/T322261 (10eoghan) 05Open→03Resolved The logs are in logstash, just hard to find. https://logstash.wikimedia.org/goto/82d8dd92e36776ce581aeef953ff6891 Thanks to @colewhite for helping... [15:31:19] akosiaris: do these users have access to deploy servers, or only to the deployment pipeline? [15:55:23] only to pipeline, but I think the easy way to fix this is to unconditionally call kubectl get events right after helm install [15:55:42] that way we won't need to do weird things like detecting failures in helm hooks [15:56:12] or if not unconditionally, at least in the teardown handler I think I am seeing in that console output [15:58:41] it should be doable with a postsync hook using a conditional on .Event.Error [16:14:56] 10GitLab (Infrastructure), 10serviceops-collab, 10Datacenter-Switchover, 10Patch-For-Review: Switchover gitlab-replica (gitlab2002 -> gitlab1003) - https://phabricator.wikimedia.org/T329930 (10Jelto) a:03eoghan [16:20:46] 10GitLab (Infrastructure), 10serviceops-collab, 10Datacenter-Switchover: Switchover gitlab (gitlab1004 -> gitlab2002) - https://phabricator.wikimedia.org/T329931 (10Jelto) [16:23:35] 10GitLab (Infrastructure), 10serviceops-collab, 10Datacenter-Switchover: Switchover gitlab (gitlab1004 -> gitlab2002) - https://phabricator.wikimedia.org/T329931 (10LSobanski) p:05Triage→03High [23:35:07] 10Phabricator: Help with Phabricator MFA - https://phabricator.wikimedia.org/T329964 (10Aklapper) Thanks for the additional context! If a WMF staff member gets offboarded, usually their `@wikimedia.org` email address, wiki account, Phabricator account, etc get disabled, so other folks don't "expect replies" from...