[02:41:53] <icinga-wm>	 PROBLEM - PHD should be supervising processes on phab1001 is CRITICAL: PROCS CRITICAL: 2 processes with UID = 497 (phd) https://wikitech.wikimedia.org/wiki/Phabricator
[02:44:29] <icinga-wm>	 RECOVERY - PHD should be supervising processes on phab1001 is OK: PROCS OK: 4 processes with UID = 497 (phd) https://wikitech.wikimedia.org/wiki/Phabricator
[07:07:07] <wikibugs>	 10Continuous-Integration-Infrastructure, 10ci-test-error (WMF-deployed Build Failure): Error: ENOSPC: no space left on device - https://phabricator.wikimedia.org/T312005 (10noarave) This originated from this patch https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/810320/5 I don't think it recurr...
[07:31:26] <wikibugs>	 10Continuous-Integration-Infrastructure, 10ci-test-error (WMF-deployed Build Failure): Error: ENOSPC: no space left on device - https://phabricator.wikimedia.org/T312005 (10noarave) 05Open→03Invalid
[07:55:57] <wikibugs>	 10Release-Engineering-Team: Create mathlatexml table? - https://phabricator.wikimedia.org/T309686 (10Physikerwelt) 05Open→03Resolved a:03Physikerwelt https://en.wikipedia.org/wiki/Special:Preferences#mw-prefsection-rendering still works. So I assume everything is done.
[08:15:21] <wikibugs>	 (03PS1) 10Robert Vogel: Add CI dependencies to `Extension:MenuEditor` [integration/config] - 10https://gerrit.wikimedia.org/r/816700
[08:17:58] <wikibugs>	 (03CR) 10CI reject: [V: 04-1] Add CI dependencies to `Extension:MenuEditor` [integration/config] - 10https://gerrit.wikimedia.org/r/816700 (owner: 10Robert Vogel)
[08:46:11] <wikibugs>	 10GitLab (Infrastructure), 10Release-Engineering-Team, 10serviceops, 10serviceops-collab, 10User-brennen: GitLab major release: 15.x - https://phabricator.wikimedia.org/T309062 (10Jelto) > Based on last time, we should give this a month or so to bake in; filing now for planning purposes. As best I unders...
[08:54:11] <wikibugs>	 (03PS1) 10Jaime Nuche: install-world: check if extra masters before attempting to sync [tools/scap] - 10https://gerrit.wikimedia.org/r/816707
[09:21:59] <wikibugs>	 (03CR) 10Jaime Nuche: [C: 03+2] install-world: check if extra masters before attempting to sync [tools/scap] - 10https://gerrit.wikimedia.org/r/816707 (owner: 10Jaime Nuche)
[09:22:57] <wikibugs>	 10Project-Admins: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706 (10EChetty) Hi there, I would love to be able to create project labels, create and modify sprint boards and manage projects for my teams. I am the data platform product manager that loo...
[09:26:18] <wikibugs>	 (03Merged) 10jenkins-bot: install-world: check if extra masters before attempting to sync [tools/scap] - 10https://gerrit.wikimedia.org/r/816707 (owner: 10Jaime Nuche)
[09:28:23] <wikibugs>	 10Project-Admins: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706 (10Ladsgroup) Done!  //Usual disclaimer: Please follow the [guidelines](https://www.mediawiki.org/wiki/Phabricator/Creating_and_renaming_projects#Creating_new_projects) (on project icon...
[09:29:33] <wikibugs>	 10Project-Admins: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706 (10EChetty) Thank you @Ladsgroup !
[12:02:21] <wikibugs>	 (03PS1) 10Jaime Nuche: serial lock: move default location to `/var/lock` [tools/scap] - 10https://gerrit.wikimedia.org/r/816752
[12:12:25] <wikibugs>	 (03CR) 10Jaime Nuche: [C: 03+2] serial lock: move default location to `/var/lock` [tools/scap] - 10https://gerrit.wikimedia.org/r/816752 (owner: 10Jaime Nuche)
[12:17:55] <wikibugs>	 (03Merged) 10jenkins-bot: serial lock: move default location to `/var/lock` [tools/scap] - 10https://gerrit.wikimedia.org/r/816752 (owner: 10Jaime Nuche)
[12:29:46] <wmf-insecte>	 Project beta-update-databases-eqiad build #60269: 04FAILURE in 9 min 46 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/60269/
[13:20:01] <wmf-insecte>	 Project beta-update-databases-eqiad build #60270: 04STILL FAILING in 0.98 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/60270/
[13:31:50] <RhinosF1>	 Reedy: vendor hasn't been done ^
[14:01:13] <James_F>	 RhinosF1: Indeed, someone 'helpfully' force-merged the MW patch without merging the vendor one.
[14:03:09] <RhinosF1>	 James_F: on what planet do we do that?
[14:03:27] <RhinosF1>	 Outside of something being on fire
[14:03:28] <James_F>	 Indeed.
[14:04:07] <RhinosF1>	 James_F: I think someone might need a lesson in how we do things
[14:07:39] <RhinosF1>	 James_F: you are joking right
[14:07:48] <James_F>	 No.
[14:08:07] <RhinosF1>	 Since when is use the big red emergency button before try it again in the book of debugging computers James_F
[14:08:43] <James_F>	 It's easy to get confused when Google go out of their way to point users to the big flashy "merge this" button in the gerrit interface.
[14:20:01] <wmf-insecte>	 Project beta-update-databases-eqiad build #60271: 04STILL FAILING in 1 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/60271/
[14:57:51] <wikibugs>	 (03PS1) 10Jaime Nuche: Revert "serial lock: move default location to `/var/lock`" [tools/scap] - 10https://gerrit.wikimedia.org/r/816797
[14:59:53] <wikibugs>	 (03CR) 10Jforrester: Make composer-php80 run on gate-and-submit for MW core (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/816062 (https://phabricator.wikimedia.org/T300463) (owner: 10Brian Wolff)
[15:07:47] <wikibugs>	 (03CR) 10Jaime Nuche: [C: 03+2] Revert "serial lock: move default location to `/var/lock`" [tools/scap] - 10https://gerrit.wikimedia.org/r/816797 (owner: 10Jaime Nuche)
[15:12:37] <wikibugs>	 (03Merged) 10jenkins-bot: Revert "serial lock: move default location to `/var/lock`" [tools/scap] - 10https://gerrit.wikimedia.org/r/816797 (owner: 10Jaime Nuche)
[15:30:03] <wmf-insecte>	 Yippee, build fixed!
[15:30:04] <wmf-insecte>	 Project beta-update-databases-eqiad build #60272: 09FIXED in 10 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/60272/
[15:32:51] <hexmode>	 joining so I can ask questions instead of screw things up on gerrit/zuul :P
[15:40:43] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Discovery-Search: Verify (and/or fix) Elasticsearch beta cluster problems - https://phabricator.wikimedia.org/T313521 (10Gehel) 05Open→03Invalid Not clear what is broken. Nothing obvious was found.
[16:00:42] <gehel>	 Hello releng! Would T312198 be part of your scope?
[16:00:44] <stashbot>	 T312198: Developer productivity: Shared ElasticSearch instance - https://phabricator.wikimedia.org/T312198
[16:00:44] <wikibugs>	 10Release-Engineering-Team, 10Discovery, 10Discovery-Search, 10Elasticsearch, 10Developer Productivity: Developer productivity: Shared ElasticSearch instance - https://phabricator.wikimedia.org/T312198 (10Gehel)
[16:42:38] <wikibugs>	 10GitLab (Infrastructure), 10Release-Engineering-Team, 10serviceops, 10serviceops-collab, 10User-brennen: GitLab major release: 15.x - https://phabricator.wikimedia.org/T309062 (10brennen) > The release of GitLab 15 was two month ago. GitLab 15.2 was released last week too. Do you have any plans/preferen...
[16:43:54] <wikibugs>	 10GitLab (Infrastructure), 10Release-Engineering-Team, 10serviceops, 10serviceops-collab, 10User-brennen: GitLab major release: 15.x - https://phabricator.wikimedia.org/T309062 (10brennen) Based on past experience, it does seem likely that we can expect a critical security release that hasn't been backpo...
[17:05:11] <wikibugs>	 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Radar), 10Security-Team, 10serviceops, and 2 others: Setup GitLab Runner in trusted environment - https://phabricator.wikimedia.org/T295481 (10sbassett) @Jelto - From a security perspective, as long as `$GITLAB_TOKEN`'s value is never disclosed in an...
[17:18:17] <wikibugs>	 10GitLab (Project Migration), 10Release-Engineering-Team: Create new GitLab project group: wm-juniors-il - https://phabricator.wikimedia.org/T313750 (10GBecher1)
[17:25:59] <wikibugs>	 (03CR) 10Brian Wolff: Make composer-php80 run on gate-and-submit for MW core (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/816062 (https://phabricator.wikimedia.org/T300463) (owner: 10Brian Wolff)
[17:28:42] <RhinosF1>	 thcipriani: https://phabricator.wikimedia.org/T313706
[17:32:59] <zabe>	 RhinosF1, https://phabricator.wikimedia.org/E1532
[17:55:48] <RhinosF1>	 zabe: well they might be stuck then
[17:59:06] <RhinosF1>	 greg-g: are you maybe the next level?
[18:08:47] <greg-g>	 tyler should be good enough :)
[18:11:00] <mutante>	 I think this all started because he isn't here
[18:12:11] <taavi>	 i'm not sure where the urgency comes here, if there's a need to run a script manually before tyler's back we surely have people who already have access and can run the script for them
[18:13:16] <RhinosF1>	 greg-g: Tyler is out of office and I'm not sure he'll see it in time.
[18:13:21] <RhinosF1>	 taavi: well there's that.
[18:14:45] <RhinosF1>	 greg-g: you can happily tell them to wait until he's back and if it's too late it's too late
[18:23:26] <greg-g>	 tyler's new boss is not in this irc channel, afaik
[18:31:33] <bawolff>	 Reedy: We're backporting php8.1 fixes to REL1_38 and REL1_37 right? (But not REL1_35?)?
[18:38:57] <RhinosF1>	 greg-g: no idea who it even is
[19:03:51] <bawolff>	 "Fatal error: Class "PHPUnit\Framework\TestFailure" not found in /workspace/src/includes/CommentFormatter/StringCommentIterator.php on line 0" is definitely an error i haven't seen before...
[20:56:47] <Krinkle>	 brennen: how sursprising is it to see wmf.19 code running today?
[20:57:08] <brennen>	 er...
[20:57:10] <Krinkle>	 apparently 5% of today's traffic is runnig wmf.19 according to https://performance.wikimedia.org/arclamp/svgs/daily/2022-07-25.excimer-wall.index.svgz?x=10.0&y=1429
[20:57:18] <Krinkle>	 (the rest wmf.21)
[20:57:37] <brennen>	 pretty surprising: https://versions.toolforge.org/
[20:57:50] <Krinkle>	 SAL says we switched group2 on 21st
[20:58:06] <brennen>	 indeed i see some errors for .19
[20:58:35] <brennen>	 i wonder if... something was depooled and didn't get synced on thursday?
[20:58:44] <Krinkle>	 ok, please file task and escalate as you see fit. I'm OOO. Yeah, maybe ask SRE as well.
[20:58:52] <brennen>	 ack, will do
[21:06:18] <wikibugs>	 10Release-Engineering-Team: Some traffic seems to be reaching 1.39.0-wmf.19 code - https://phabricator.wikimedia.org/T313770 (10brennen)
[21:06:55] <wikibugs>	 10Release-Engineering-Team: Some traffic seems to be reaching 1.39.0-wmf.19 code - https://phabricator.wikimedia.org/T313770 (10brennen)
[21:07:18] <wikibugs>	 10Release-Engineering-Team, 10User-brennen: Some traffic seems to be reaching 1.39.0-wmf.19 code - https://phabricator.wikimedia.org/T313770 (10brennen)
[21:07:57] <wikibugs>	 10Release-Engineering-Team, 10User-brennen: Some traffic seems to be reaching 1.39.0-wmf.19 code - https://phabricator.wikimedia.org/T313770 (10Zabe)
[21:11:00] <wikibugs>	 10Release-Engineering-Team, 10User-brennen: Some traffic seems to be reaching 1.39.0-wmf.19 code - https://phabricator.wikimedia.org/T313770 (10Peachey88)
[21:25:58] <dduvall>	 !log disabling puppet on untrusted gitlab-runners to test deployment of https://gerrit.wikimedia.org/r/c/operations/puppet/+/815769
[21:26:00] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[21:55:53] <wikibugs>	 10Release-Engineering-Team, 10User-brennen: Some traffic seems to be reaching 1.39.0-wmf.19 code - https://phabricator.wikimedia.org/T313770 (10brennen) p:05Triage→03Unbreak! Well, that didn't seem to do it.  Still seeing errors for wmf.19.  I'm going to mark this as a blocker for 1.39.0-wmf.22 (T308075).
[21:56:15] <wikibugs>	 10Release-Engineering-Team, 10User-brennen: Some traffic seems to be reaching 1.39.0-wmf.19 code - https://phabricator.wikimedia.org/T313770 (10brennen)
[21:56:17] <wikibugs>	 10Release-Engineering-Team (Priority Backlog 📥), 10Release, 10Train Deployments: 1.39.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T308075 (10brennen)
[22:16:26] <dduvall>	 !log re-enabled puppet on untrusted runners following testing of https://gerrit.wikimedia.org/r/c/operations/puppet/+/815769
[22:16:27] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[22:22:41] <wikibugs>	 10Phabricator, 10Release-Engineering-Team (The Decommission Mission 💀), 10serviceops, 10serviceops-collab: Setup rsync for phab data on disk - https://phabricator.wikimedia.org/T313360 (10dduvall) >>! In T313360#8098173, @Dzahn wrote: > from syncing data last time back in 2019 >  > https://gerrit.wikimedia...
[22:32:53] <wikibugs>	 10Release-Engineering-Team, 10User-brennen: Some traffic seems to be reaching 1.39.0-wmf.19 code - https://phabricator.wikimedia.org/T313770 (10Dzahn) confirmed /srv/mediawiki/php-1.39.0-wmf.21 exists on every single appserver  it has been pointed out on IRC this affects only the canary servers  canary servers...
[22:33:38] <wikibugs>	 10Release-Engineering-Team, 10User-brennen: Some traffic seems to be reaching 1.39.0-wmf.19 code - https://phabricator.wikimedia.org/T313770 (10brennen) Noting also that `wikiversions.json` is up to date on the canaries.
[22:34:13] <wikibugs>	 10Release-Engineering-Team, 10User-brennen: Some traffic seems to be reaching 1.39.0-wmf.19 code - https://phabricator.wikimedia.org/T313770 (10brennen) Something related to {T311386}?
[22:42:17] <wikibugs>	 10Phabricator, 10Release-Engineering-Team (The Decommission Mission 💀), 10serviceops, 10serviceops-collab: Setup rsync for phab data on disk - https://phabricator.wikimedia.org/T313360 (10Dzahn) @dduvall You guys could double check if you think we need anything _in addition _ to /srv/repos. Because that is...
[23:11:37] <wikibugs>	 (03PS1) 10Cwhite: Support OpenSearch-Dashboards 2.1.0 [releng/phatality] - 10https://gerrit.wikimedia.org/r/816867 (https://phabricator.wikimedia.org/T304440)
[23:27:17] <wikibugs>	 10Release-Engineering-Team, 10User-brennen: Some traffic seems to be reaching 1.39.0-wmf.19 code - https://phabricator.wikimedia.org/T313770 (10brennen) Further from notes from IRC (thanks Zabe and Dzahn for investigation):    - Errors are all PHP 7.2   - Canaries just not getting restarted?     - Something in...