[18:52:25] Krinkle: BTW, not sure if you saw my comment on T401255 – is this strictly needed? We've apparently not done it before for dev-images. [18:52:26] T401255: Switch PHP 8.3 image of MediaWiki-Docker from Sury to WMF - https://phabricator.wikimedia.org/T401255 [19:07:39] James_F: not strictly needed no. We've definitely been on the WMF package before for a php version iirc, but not as part of the upgrade and not the last 2-3 versions. I proposed it's a thing to value, same reason we do it for CI. If it weren't for the mismatching distro, I think the timing as early step (while we work on other things), would've worked too! [19:08:10] Maybe it was done for 7.4? It's not been done for 8.1, and we've been of course on 8.3 for a while now. [19:08:34] I think the WMF build of 8.1 maybe didn't exist when we'd already switched dev-images to 8.3, now I think of it. [19:08:42] I'd like to keep it in the checklist but next time we'll need to coordinate so that we don't end up upgrading too soon or too late based on what SRE will provide. [19:09:10] Perhaps. Are SRE bought in to the push to move faster on these? [19:14:20] That's one way to solve it indeed. [19:16:22] Another is to not lead ahead with dev image distros until SRE has the packages ready. It seems reasonable for prod and CI and DX to not diverge for more than say 3 months. Whenever SRE do it for prod MW, we can still lead and have DX/CI first. [19:16:51] I imagine SRE would be fine to move up the php packaging for the upcoming distro by a few weeks but not say by 12 months. [19:17:54] I'm not saying anything went wrong. I think it was a reasonable assumption they'd be ready or doing the Debian upgrade around now. [19:18:17] But next time we can agree to do it in the same quarter, probably not the same quarter as a php upgrade. [19:19:32] Speaking of, do we when Debian bookworm is planned for prod MW? [19:30:41] Q3, I believe. [19:31:14] But I disagree that MW development testing should wait for SRE to be ready to package/deploy. That seems like putting a lot of pressure on scarce SRE resource. [19:31:43] And testing in CI but not locally isn't ideal for finding edge cases which aren't in test coverage, library issues, etc. [19:43:40] I think it's beneficial for debugging edge cases the other way around, that the three envs run the same distro at least most of the time. It also means we can't otherwise install any packages that are wmf only. Eg latest (or patched) wikidiff, excimer, luasandbox etc. Right now we're not testing with those locally because we're on a different distro [19:44:27] I'm not calling for more resources just to delay updating dev images until shortly before prod plans to do the same. [19:45:18] Right now we're almost a year ahead of prod. We didn't know that but hence why I think it's worth aligning that more in the future. We can still lead by a month or two, that seems useful. [20:56:47] There are certainly issues that are found in the exact versions and combinations used in prod, but I'm not generally convinced that if we hold dev and qa environments back to match prod more closely we will find more defects.