[04:35:00] <wikibugs>	 10Gerrit, 10wikitech.wikimedia.org: Can't login into Gerrit with a Wikimedia Developer account with non-unique email address - https://phabricator.wikimedia.org/T270233 (10Tgr) >>! In T270233#9494689, @hashar wrote: > Declining since the root cause was two accounts having the same email addresses while Gerrit...
[09:40:39] <wikibugs>	 10Gerrit, 10wikitech.wikimedia.org: Can't login into Gerrit with a Wikimedia Developer account with non-unique email address - https://phabricator.wikimedia.org/T270233 (10hashar) 05Declined→03Open Sure sorry, I have declined the task in a rush while triaging tasks concerning Gerrit. Though surely Gerrit s...
[10:12:34] <jnuche>	 !log updating CI Jenkins version to 2.426.3 to address vulnerabilities
[10:12:36] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[10:16:49] <jnuche>	 !log CI Jenkins update to 2.426.3 complete
[10:16:50] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[10:48:17] <eoghan>	 hashar: I'm going to deploy this in the next while, are you ok with that? https://gerrit.wikimedia.org/r/c/operations/puppet/+/987135
[10:48:27] <eoghan>	 hashar: It's the regular rsync job for the lfs data in gerrit
[11:43:57] <hashar>	 eoghan: sorry I was busy writing some javascript/tests etc
[11:44:29] <hashar>	 I guess as long as the rsync doesn't end up deleting all objects somehow, I guess it is fine :-]
[11:49:46] <wikibugs>	 10Phabricator (Upstream), 10Upstream, 10User-greg: Closed task with "Unknown status" - https://phabricator.wikimedia.org/T125803 (10awight) Here's a recent occurrence: T345176#9468860 notably "via bulk update"
[12:11:30] <eoghan>	 hashar: That's what we always hope for not to happen... Should be fine anyway, the change is there and we see the job is rsyncing from the current primary gerrit1003 and to gerrit2002. I suppose there's a risk that the data in gerrit2002 has diverged and has something that is not on gerrit1003
[12:12:52] <eoghan>	 But if that's the case, it's broken already 🤷
[12:14:21] <wmf-insecte>	 Yippee, build fixed!
[12:14:21] <wmf-insecte>	 Project mediawiki-core-doxygen-docker build #49876: 09FIXED in 10 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/49876/
[13:26:44] <hashar>	 eoghan: only gerrit1003 can receive new objects
[13:27:09] <hashar>	 I supposed the LFS data on gerrit2002 is either a left over from a previous switch over
[13:27:20] <hashar>	 or some rsync that was done when the machine got reimaged
[13:27:26] <hashar>	 so it is fine :)
[13:27:41] <hashar>	 ideally I'd like to move those out to Swift/Ceph or whatever distributed file system
[13:27:48] <hashar>	 but that is a different story
[14:01:50] <wikibugs>	 10Phabricator (Upstream), 10Release-Engineering-Team, 10Infrastructure-Foundations, 10collaboration-services, and 2 others: Ensure that phabricator.wikimedia.org adheres to Google's sender guidelines - https://phabricator.wikimedia.org/T355691 (10taavi) Is it known/ok that Phab is sending mail from `phabri...
[14:24:09] <eoghan>	 hashar: That would be nice, I'd like to do the same with the same artefacts etc in gitlab. I think we have it on a roadmap for later this year.
[14:30:48] <wikibugs>	 10Phabricator (Upstream), 10Release-Engineering-Team, 10Infrastructure-Foundations, 10collaboration-services, and 2 others: Ensure that phabricator.wikimedia.org adheres to Google's sender guidelines - https://phabricator.wikimedia.org/T355691 (10eoghan) There shouldn't be anything directly sending from ph...
[14:39:54] <eoghan>	 hashar: Ok, I'm going to deploy this now. 
[15:36:33] <wikibugs>	 10Phabricator (Upstream), 10Patch-For-Review, 10Upstream: Activity pane on front page no longer shows New Tasks by default after Phorge migration - https://phabricator.wikimedia.org/T344835 (10valerio.bozzolan) This change is now in branch `stable` and under tag `2023.49`:  https://we.phorge.it/D25447  Maybe...
[16:31:13] <mutante>	 gitlab is about to go down for a couple minutes soon. server has to move to a different rack. I silenced it with downtime cookbook.
[16:58:07] <wikibugs>	 10Project-Admins: Create project tag for MediaWiki-extensions-CommunityConfiguration - https://phabricator.wikimedia.org/T356197 (10Sgs)
[16:58:21] <mutante>	 gitlab is back 
[16:58:31] <mutante>	 actual downtime was only 7 minutes
[16:59:07] <mutante>	 and most of it was the router learning new routes and pushing switch config.. not even the part that he physically moved the server metal around, which was amazing
[17:18:51] <brennen>	 nice
[17:40:14] <brennen>	 !log phab1004: testing public_task_dump.py tweaks, take 3
[17:40:15] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[17:58:19] <wikibugs>	 10Gitlab-Application-Security-Pipeline, 10Security-Team, 10SecTeam-Processed, 10Security, 10user-sbassett: [EPIC] Application Security Pipeline Components for Gitlab - Phase 2 Work - https://phabricator.wikimedia.org/T342177 (10sbassett)
[17:58:36] <wikibugs>	 10Gitlab-Application-Security-Pipeline, 10Security-Team, 10SecTeam-Processed, 10Security: Add basic unit tests and CI support for semgrep-merge-tool - https://phabricator.wikimedia.org/T353747 (10sbassett) 05Open→03In progress p:05Triage→03Medium a:03sguebo_WMF
[18:18:31] <mutante>	 wanna review a change on "foreachwikiindblist"?  https://gerrit.wikimedia.org/r/c/operations/puppet/+/992263
[18:18:52] <mutante>	 as that could mess with deployments I wasnt so sure who to ask
[18:19:12] <mutante>	 (when I was added as reviewer)
[18:46:47] <dancy>	 mutante: I added some comments.
[18:52:44] <mutante>	 dancy: :) thanks!
[19:38:11] <wikibugs>	 10Phabricator (Upstream), 10Release-Engineering-Team, 10Infrastructure-Foundations, 10collaboration-services, and 2 others: Ensure that phabricator.wikimedia.org adheres to Google's sender guidelines - https://phabricator.wikimedia.org/T355691 (10jhathaway) >>! In T355691#9498461, @taavi wrote: > Is it kno...
[19:42:18] <wikibugs>	 10Phabricator (Upstream), 10Release-Engineering-Team, 10Infrastructure-Foundations, 10collaboration-services, and 2 others: Ensure that phabricator.wikimedia.org adheres to Google's sender guidelines - https://phabricator.wikimedia.org/T355691 (10taavi) Interesting, it seems like Phabricator only signs mai...
[20:10:18] <wikibugs>	 10Project-Admins: Create project tag for MediaWiki-extensions-CommunityConfiguration - https://phabricator.wikimedia.org/T356197 (10Jdforrester-WMF) Do you want this to be a component or a milestone underneath https://phabricator.wikimedia.org/project/profile/6608/, it's not clear to me?
[20:31:54] <wikibugs>	 10Release-Engineering-Team, 10collaboration-services: Ensure that gitlab.wikimedia.org adheres to Google's sender guidelines - https://phabricator.wikimedia.org/T355776 (10Dzahn) - [x] notification emails are sent from `gitlab@gitlab.wikimedia.org` so this is a separate domain., DNS verified ipv4/ipv6 forward...
[20:32:11] <wikibugs>	 10Release-Engineering-Team, 10collaboration-services: Ensure that gitlab.wikimedia.org adheres to Google's sender guidelines - https://phabricator.wikimedia.org/T355776 (10Dzahn)
[20:33:35] <jinxer-wm>	 (Queue (Jenkins jobs + Zuul functions) alert) firing: <no value>   - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert
[20:36:57] <wikibugs>	 10Release-Engineering-Team, 10collaboration-services: Ensure that gitlab.wikimedia.org adheres to Google's sender guidelines - https://phabricator.wikimedia.org/T355776 (10Dzahn) mail route:  Received: from git by gitlab2002.wikimedia.org with local (Exim Received: from gitlab2002.wikimedia.org .. by mx1001.wi...
[20:40:57] <wikibugs>	 10Release-Engineering-Team, 10collaboration-services: Ensure that gitlab.wikimedia.org adheres to Google's sender guidelines - https://phabricator.wikimedia.org/T355776 (10Dzahn)
[20:43:35] <jinxer-wm>	 (Queue (Jenkins jobs + Zuul functions) alert) resolved: <no value>   - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert
[20:52:58] <wikibugs>	 10Phabricator (Upstream), 10Release-Engineering-Team, 10Infrastructure-Foundations, 10collaboration-services, and 2 others: Ensure that phabricator.wikimedia.org adheres to Google's sender guidelines - https://phabricator.wikimedia.org/T355691 (10jhathaway) ah, that seems like a bug? However, dkim signatur...
[20:53:27] <wikibugs>	 (03PS1) 10Jforrester: build: Upgrade phpunit to 9.6.16 [integration/docroot] - 10https://gerrit.wikimedia.org/r/994313 (https://phabricator.wikimedia.org/T342110)
[20:55:20] <wikibugs>	 (03CR) 10Reedy: [C: 03+2] build: Upgrade phpunit to 9.6.16 [integration/docroot] - 10https://gerrit.wikimedia.org/r/994313 (https://phabricator.wikimedia.org/T342110) (owner: 10Jforrester)
[20:56:00] <wikibugs>	 (03Merged) 10jenkins-bot: build: Upgrade phpunit to 9.6.16 [integration/docroot] - 10https://gerrit.wikimedia.org/r/994313 (https://phabricator.wikimedia.org/T342110) (owner: 10Jforrester)
[20:58:26] <Daimona>	 James_F: thanks for the mass-upgrade! Currently catching up and fixing the few that are failing.
[20:58:39] <James_F>	 Daimona: Of course!
[20:58:53] <James_F>	 They're all passing locally, but of course CI is dark and full of shadows.
[20:59:49] <James_F>	 Also, getting Wikibase's upstreams to upgrade is going to take years, probably. It generally does. :-(
[21:00:57] <wikibugs>	 10Release-Engineering-Team, 10collaboration-services, 10Patch-For-Review: Ensure that gitlab.wikimedia.org adheres to Google's sender guidelines - https://phabricator.wikimedia.org/T355776 (10Dzahn) per comment on the related ticket for phabricator (T355691#9500163) "dkim signature should be okay, since our...
[21:01:10] <Daimona>	 I just realized that withConsecutive, unlike other deprecated methods, does not emit a deprecation warning
[21:01:23] <James_F>	 Oh, fun.
[21:01:29] <James_F>	 So all that migration wasn't quite so urgent.
[21:01:38] <James_F>	 Daimona: Thank you for your service.
[21:01:46] <wikibugs>	 10Release-Engineering-Team, 10collaboration-services, 10Patch-For-Review: Ensure that gitlab.wikimedia.org adheres to Google's sender guidelines - https://phabricator.wikimedia.org/T355776 (10Dzahn) To actually check the "spam rate" box we first have to add google-site-verification TXT record to DNS for gitl...
[21:01:47] <Daimona>	 So CI would be green anyway. But I'll clean those up after the upgrade.
[21:02:10] <Daimona>	 (That is, CI is green unless phan is configured to analyze tests, which I think it does in most repos)
[21:03:31] <wikibugs>	 10Release-Engineering-Team, 10collaboration-services, 10Patch-For-Review: Ensure that gitlab.wikimedia.org adheres to Google's sender guidelines - https://phabricator.wikimedia.org/T355776 (10Dzahn)
[21:15:14] <wikibugs>	 (03PS1) 10QChris: Allow “Gerrit Managers” to import history [extensions/CommunityConfiguration] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/994329
[21:15:16] <wikibugs>	 (03CR) 10QChris: [V: 03+2 C: 03+2] Allow “Gerrit Managers” to import history [extensions/CommunityConfiguration] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/994329 (owner: 10QChris)
[21:15:18] <wikibugs>	 (03PS1) 10QChris: Import done. Revoke import grants [extensions/CommunityConfiguration] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/994330
[21:15:20] <wikibugs>	 (03CR) 10QChris: [V: 03+2 C: 03+2] Import done. Revoke import grants [extensions/CommunityConfiguration] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/994330 (owner: 10QChris)
[21:19:16] <wikibugs>	 10Release-Engineering-Team, 10collaboration-services, 10Patch-For-Review: Ensure that gitlab.wikimedia.org adheres to Google's sender guidelines - https://phabricator.wikimedia.org/T355776 (10Dzahn) mail volume is all over the place per day: {P55902}
[22:06:47] <wikibugs>	 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Onboarding 🚀), 10Patch-For-Review: Automatically clean up unused buildkitd PersistentVolumeClaims periodically - https://phabricator.wikimedia.org/T351478 (10CodeReviewBot) sandeeps merged https://gitlab.wikimedia.org/repos/releng/k8s-pvc-cleaner/-/mer...
[22:24:02] <wikibugs>	 10Gerrit, 10wikitech.wikimedia.org: Can't login into Gerrit with a Wikimedia Developer account with non-unique email address - https://phabricator.wikimedia.org/T270233 (10Tgr) >>! In T270233#9497614, @hashar wrote: > My guess is for us to enforce the email uniqueness on the Wikitech side  I agree that would m...
[23:31:32] <wikibugs>	 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Onboarding 🚀), 10Patch-For-Review: Automatically clean up unused buildkitd PersistentVolumeClaims periodically - https://phabricator.wikimedia.org/T351478 (10CodeReviewBot) dancy opened https://gitlab.wikimedia.org/repos/releng/k8s-pvc-cleaner/-/merge_...