[09:31:16] 06serviceops, 10MW-on-K8s: Allow running one-off scripts manually - https://phabricator.wikimedia.org/T341553#10574262 (10Urbanecm_WMF) Hello! Can we please please find something better to view running jobs? I raised this before via T341553#10362280, and the `jq` solution is not really appropriate. I happen to... [10:28:45] 06serviceops, 10MW-on-K8s, 13Patch-For-Review: Allow members of restricted to run maintenance scripts - https://phabricator.wikimedia.org/T378429#10574515 (10JMeybohm) Sorry, my bad. There is an option to choose a different name for the kubeconfig files generated which unfortunately does not support re-using... [10:58:39] 06serviceops, 06SRE Observability: chartmuseum prometheus metrics cardinality spam - https://phabricator.wikimedia.org/T386808#10574780 (10JMeybohm) This is probably https://github.com/helm/chartmuseum/issues/448 which has commits attached now...so maybe it's fixed but not marked as such. How big of a problem... [11:12:14] 06serviceops, 07Datacenter-Switchover: SRE comms for March 2025 Datacentre switchover - https://phabricator.wikimedia.org/T385157#10574801 (10hnowlan) [11:24:13] 06serviceops, 06Content-Transform-Team, 10Maps (Kartotherian): Review maps outage happened on Feb 17th 2025 - https://phabricator.wikimedia.org/T386648#10574813 (10elukey) 05Open→03Resolved a:03elukey https://wikitech.wikimedia.org/wiki/Incidents/2025-02-17_maps published :) [12:23:34] 06serviceops, 10CX-cxserver, 06Language and Product Localization, 10RESTBase Sunsetting, and 2 others: Decommission cxserver endpoints from RESTBase - https://phabricator.wikimedia.org/T372753#10574941 (10MSantos) [12:41:30] 06serviceops, 06Content-Transform-Team, 10Maps (Kartotherian): Review maps outage happened on Feb 17th 2025 - https://phabricator.wikimedia.org/T386648#10574970 (10elukey) https://wikitech.wikimedia.org/wiki/Incidents/2025-02-19_maps published :) [13:09:56] 06serviceops, 06SRE Observability: chartmuseum prometheus metrics cardinality spam - https://phabricator.wikimedia.org/T386808#10575034 (10fgiunchedi) Potentially a big problem if it goes on and we keep accumulating label values in `url`. The other question in my mind is: why are these requests getting to char... [13:47:10] 06serviceops, 06SRE Observability: chartmuseum prometheus metrics cardinality spam - https://phabricator.wikimedia.org/T386808#10575120 (10JMeybohm) I think we can investigate if an update does fix this (chartmuseum upgrades are usually not that tricky). Dropping `url` would be an an option if we can do so fo... [14:02:01] 06serviceops, 13Patch-For-Review, 07PHP 8.1 support: Update PCRE in PHP 8.1 images to PCRE 10.39 or newer - https://phabricator.wikimedia.org/T386006#10575151 (10MatthewVernon) The relevant SME/service owner was out last week, and is off sick today; hopefully we can move on the wmf-debci changes tomorrow... [14:11:42] 06serviceops, 07PHP 8.1 support: Rebuild and deploy all shellboxes to pickup the latest PCRE version (>10.39) - https://phabricator.wikimedia.org/T387124 (10jijiki) 03NEW [14:12:14] 06serviceops, 07PHP 8.1 support: Rebuild and deploy all shellboxes to pickup the latest PCRE version (>10.39) - https://phabricator.wikimedia.org/T387124#10575181 (10jijiki) p:05Triage→03Medium a:03jijiki [14:55:55] 06serviceops, 13Patch-For-Review, 07PHP 8.1 support: Update PCRE in PHP 8.1 images to PCRE 10.39 or newer - https://phabricator.wikimedia.org/T386006#10575288 (10MatthewVernon) [but you can extract the artifacts from the successful build from gitlab by hand should you wish in the mean time] [15:03:28] 06serviceops, 10MW-on-K8s: mwscript-k8s purgeList does not reliably purge cached URLs - https://phabricator.wikimedia.org/T387127 (10Lucas_Werkmeister_WMDE) 03NEW [15:05:09] 06serviceops, 10MW-on-K8s: mwscript-k8s purgeList does not reliably purge cached URLs - https://phabricator.wikimedia.org/T387127#10575337 (10Lucas_Werkmeister_WMDE) Speculation: something in our network blocks the purge request that MediaWiki sends to the cache servers? And/or the cache servers aren’t configu... [15:05:59] 06serviceops, 10MediaWiki-Core-HTTP-Cache, 10MW-on-K8s: mwscript-k8s purgeList does not reliably purge cached URLs - https://phabricator.wikimedia.org/T387127#10575345 (10Lucas_Werkmeister_WMDE) [15:24:53] 06serviceops, 13Patch-For-Review, 07PHP 8.1 support: Update PCRE in PHP 8.1 images to PCRE 10.39 or newer - https://phabricator.wikimedia.org/T386006#10575407 (10Scott_French) Many thanks for for the update, Matthew. I might opt for the manual import option today using the artifacts from [[ https://gitlab.wi... [15:37:14] 06serviceops, 06Infrastructure-Foundations: Image publishing via docker-pkg on build2001 repeatedly failing - https://phabricator.wikimedia.org/T385531#10575463 (10elukey) 05Open→03Resolved a:03elukey Closing since the problem was related to a new ML image that was rolled back. [16:37:41] 06serviceops, 13Patch-For-Review, 07PHP 8.1 support: Update PCRE in PHP 8.1 images to PCRE 10.39 or newer - https://phabricator.wikimedia.org/T386006#10575818 (10Scott_French) @MatthewVernon - One question I ran into during the manual include just now: it seems these builds include udeb packages, however by... [17:06:36] 06serviceops, 13Patch-For-Review, 07PHP 8.1 support: Update PCRE in PHP 8.1 images to PCRE 10.39 or newer - https://phabricator.wikimedia.org/T386006#10575910 (10MatthewVernon) I don't know off-hand (@Jelto might?), but if it's a problem, we ought to be able to specify `noudeb` in the `DEB_BUILD_PROFILES` en... [18:30:56] 06serviceops, 06Content-Transform-Team, 06MediaWiki-Engineering, 07OKR-Work: Migrate parsoidtest functionality to kubernetes - https://phabricator.wikimedia.org/T386246#10576330 (10Jdlrobson-WMF) [19:58:33] 06serviceops, 13Patch-For-Review, 07PHP 8.1 support: Update PCRE in PHP 8.1 images to PCRE 10.39 or newer - https://phabricator.wikimedia.org/T386006#10576769 (10hashar) Hi, I am wondering how you plan to upgrade PCRE since it will need to be upgraded in the CI images and I do not know the consequences of ru... [20:32:17] 06serviceops, 13Patch-For-Review, 07PHP 8.1 support: Update PCRE in PHP 8.1 images to PCRE 10.39 or newer - https://phabricator.wikimedia.org/T386006#10576865 (10Scott_French) Thanks, @hashar! I believe this is actually the first time we've done this, so we're kind of figuring this out as we go. Which is to... [21:12:29] 06serviceops, 13Patch-For-Review, 07PHP 8.1 support: Update PCRE in PHP 8.1 images to PCRE 10.39 or newer - https://phabricator.wikimedia.org/T386006#10577006 (10hashar) > At this point, there is no plan to also backport to buster and update PCRE2 in the PHP 7.4 images. Just like many other "base" package, P... [21:13:53] swfrench-wmf: thank you so much for dealing with PHP PCRE / Debian packaging etc :] [21:14:20] hashar: thanks for reminding me about the CI images! :) [21:14:24] I was a bit worried earlier about the PCRE upgrade breaking havoc on CI / dev code etc [21:14:42] but your comment about sury made me look at the 8.2 / 8.3 images and they have PCRE 10.40 :) [21:15:00] which rules out the potential havoc [21:15:40] if you / SRe decide to put the new libpcre package in an extra apt component, it should be quite easy to apply that config to the CI docker images [21:15:58] else, I would do it like sury: ship libpcre under the same component as PHP [21:16:40] I think it is simpler. But I imagine there might be a reason to have libpcre under a dedicated component [21:17:31] thanks for catching that, yeah - I had no idea that sury.org packages were already using 10.40, which makes me a bit less anxious about the step to bookworm's 10.42 [21:18:20] I think he did that cause some newer versions of PHP required a newer version of PCRE than the one provided on Ubuntu/Debian/whatever [21:18:32] or had bugs/issues and it made sense to also maintain PCRE [21:18:51] re: where to source PCRE2: so, we already have the newer PCRE2 packages in `component/pcre2` (which is also where I'm getting them from when building PHP) [21:19:11] ah [21:19:17] the fun trick is [21:19:21] however, I'm considering switching to an in-component (i.e., similar to sury.org) setup [21:19:38] if one has to upgrade the lib there, they will then also magically upgrade it on MediaWiki prod :) [21:20:42] I am inclined to have all packages needed by prod PHP to be in a dedicated component [21:20:47] that sounds easier to reason about [21:20:57] as a downstream end user [21:21:20] it is a lot easier to reason about, yeah - but also for one specific gotcha w.r.t. dependency package versions, which I'll detail in the task later today [21:22:45] basically, if a base image already has PCRE2 installed (e.g., the 10.36 one that ships with bullseye) *and* I don't also update the debian/control file to require the newer package, merely apt-get installing the php packages will not also upgrade the pcre2 packages [21:25:04] in any case, I'm going to revert component/php81 back to packages using the normal bullseye 10.36 for the time being while I figure out a couple of configuration changes [21:25:15] cool :) [21:25:28] so yeah, as I mentioned in the task, please hold off on making any changes that pull in the new component for now :) [21:25:31] thank again [21:25:50] James and me are now subscribed to the task so we can sync there :) [21:26:18] the Docker image should be easy (bump changelog, run docker pkg, done) [21:26:31] I am off to bed! [21:26:45] and again thank you for the packaging work \o/ [21:26:54] have a good night! no problem :) [22:06:04] 06serviceops, 10function-orchestrator, 10Abstract Wikipedia team (25Q3 (Jan–Mar)), 05Wikifunctions Improve performance: increase CPU and Node heap limit? - https://phabricator.wikimedia.org/T385859#10577229 (10ecarg) [22:12:05] 06serviceops, 10MediaWiki-Core-HTTP-Cache, 10MW-on-K8s: mwscript-k8s purgeList does not reliably purge cached URLs - https://phabricator.wikimedia.org/T387127#10577260 (10RLazarus) a:03RLazarus purgeList from mwscript-k8s has worked previously, so my instinct is this is an intermittent problem rather than... [22:40:59] 06serviceops, 13Patch-For-Review, 07PHP 8.1 support: Update PCRE in PHP 8.1 images to PCRE 10.39 or newer - https://phabricator.wikimedia.org/T386006#10577358 (10Scott_French) Thanks for the follow-up, @hashar! That's great that the CI images using sury.org packages for 8.2 and 8.3 are already exercising PC...