[01:38:42] 06serviceops, 13Patch-For-Review: MediaWiki on PHP 8.1 production traffic ramp-up - https://phabricator.wikimedia.org/T383845#10612223 (10Scott_French) I spent some time today looking into how deployments of the larger 8.1-based releases appear more susceptible to timing out in the presence of significant imag... [01:59:01] 06serviceops, 13Patch-For-Review: Migrate production Shellbox variants to PHP 8.1 - https://phabricator.wikimedia.org/T377038#10612232 (10Scott_French) [02:43:13] 06serviceops: Migrate mw-script to PHP 8.1 - https://phabricator.wikimedia.org/T387917#10612287 (10Scott_French) [08:42:51] 06serviceops, 06Data-Engineering, 06Data-Engineering-Radar, 06Data-Platform-SRE, 10Event-Platform: Make eventstreams-internal available to WMF staff without an ssh tunnel - https://phabricator.wikimedia.org/T348763#10612471 (10Gehel) p:05Triage→03Medium [08:49:37] 06serviceops, 06collaboration-services, 06Data-Platform-SRE, 10Prod-Kubernetes, and 2 others: Fix installed key in dependend helmfile releases - https://phabricator.wikimedia.org/T387837#10612485 (10JMeybohm) 05Open→03Resolved a:03JMeybohm The fix has been merged upstream and I've rebuild helmfil... [08:59:07] 06serviceops, 06collaboration-services, 06Data-Platform-SRE, 10Prod-Kubernetes, and 2 others: top-level config key environments must be defined before releases in helmfile.yaml - https://phabricator.wikimedia.org/T387836#10612498 (10JMeybohm) p:05Triage→03High [08:59:13] 06serviceops, 06collaboration-services, 06Data-Platform-SRE, 10Prod-Kubernetes, 07Kubernetes: Migrate release template inheritance in helmfiles from YAML anchors to the inherit field - https://phabricator.wikimedia.org/T387760#10612499 (10JMeybohm) p:05Triage→03Low [09:09:12] 06serviceops, 10CirrusSearch, 10Data-Platform-SRE (2025.03.01 - 2025.03.21), 10Discovery-Search (2025.03.01 - 2025.03.21): Repartition [eqiad|codfw].cirrussearch.update_pipeline.update.v1 topics in kafka-main@[eqiad|codfw] - https://phabricator.wikimedia.org/T387863#10612528 (10Gehel) [09:53:07] 06serviceops, 10Page Content Service, 10Content-Transform-Team (Work In Progress): Pregeneration rules don't pregenerate caches for the same cases restbase did - https://phabricator.wikimedia.org/T388214 (10Jgiannelos) 03NEW [09:55:08] 06serviceops, 10Page Content Service, 10Content-Transform-Team (Work In Progress): Pregeneration rules don't pregenerate caches for the same cases restbase did - https://phabricator.wikimedia.org/T388214#10612689 (10Jgiannelos) [10:00:30] 06serviceops, 10Page Content Service, 10Content-Transform-Team (Work In Progress), 13Patch-For-Review: Pregeneration rules don't pregenerate caches for the same cases restbase did - https://phabricator.wikimedia.org/T388214#10612709 (10Jgiannelos) p:05Triage→03High [10:50:23] 06serviceops, 10Shellbox, 10SyntaxHighlight, 13Patch-For-Review, 07Wikimedia-production-error: Shellbox bubbles GuzzleHttp\Exception\ConnectException when it should probably wrap it in a ShellboxError? - https://phabricator.wikimedia.org/T374117#10612867 (10Lucas_Werkmeister_WMDE) >>! In T374117#10124263... [11:13:28] 06serviceops, 13Patch-For-Review: MediaWiki on PHP 8.1 production traffic ramp-up - https://phabricator.wikimedia.org/T383845#10612896 (10akosiaris) +1e the patch, research + approach LGTM [12:07:04] 06serviceops, 10Page Content Service, 10Content-Transform-Team (Work In Progress), 13Patch-For-Review: Pregeneration rules don't pregenerate caches for the same cases restbase did - https://phabricator.wikimedia.org/T388214#10613035 (10Jgiannelos) As a workaround there is some orchestration in place that r... [12:21:11] 06serviceops: Migrate mw-script to PHP 8.1 - https://phabricator.wikimedia.org/T387917#10613060 (10Clement_Goubert) This would be heplful for `mw-cron` and the staging deployments of `mw-debug`. [14:23:41] 06serviceops, 06Infrastructure-Foundations, 10Maps (Kartotherian), 13Patch-For-Review: Scale up Kartotherian on Wikikube and move live traffic to it - https://phabricator.wikimedia.org/T386926#10613298 (10elukey) I added more capacity to the k8s Wikikube clusters (40 pods right now), and tuned a little mor... [15:18:45] 06serviceops, 06Infrastructure-Foundations, 10Maps (Kartotherian), 13Patch-For-Review: Scale up Kartotherian on Wikikube and move live traffic to it - https://phabricator.wikimedia.org/T386926#10613574 (10akosiaris) > I suspect that some pods reach the maximum and then they go OOM, and k8s restarts the pod... [15:35:37] 06serviceops, 06Release-Engineering-Team: train presync failed - https://phabricator.wikimedia.org/T387823#10613657 (10hashar) The root cause is the Kubernetes deployment reached a ten minutes timeout and aborted. Those time out have been noticed by @Scott_French who is working on the Php 8.1 deployment (... [16:00:12] 06serviceops, 10MW-on-K8s: Remove outdated wikitech periodic jobs - https://phabricator.wikimedia.org/T388249 (10Clement_Goubert) 03NEW [16:39:42] 06serviceops, 10MediaWiki-Configuration, 06MediaWiki-Engineering, 10MW-on-K8s, and 2 others: Uncaught ConfigException: Failed to load configuration from etcd - https://phabricator.wikimedia.org/T346971#10613978 (10Reedy) Still appearing in reasonable numbers ==== Error ==== * mwversion: 1.44.0-wmf.19 *... [16:44:26] 06serviceops, 13Patch-For-Review: Migrate mw-script to PHP 8.1 - https://phabricator.wikimedia.org/T387917#10614008 (10Scott_French) Thanks, @Clement_Goubert - it's good to see that there's at least one other foreseeable concrete use case for this. I've posted a MR for what one approach to this functionality... [17:11:38] 06serviceops: Converge / update image version across shellbox service instances (cleanup) - https://phabricator.wikimedia.org/T388260 (10Scott_French) 03NEW [17:27:06] 06serviceops: Converge / update image version across shellbox service instances (cleanup) - https://phabricator.wikimedia.org/T388260#10614154 (10Lucas_Werkmeister_WMDE) I think it’s okay to go ahead with this 👍 [19:10:39] 06serviceops, 13Patch-For-Review: MediaWiki on PHP 8.1 production traffic ramp-up - https://phabricator.wikimedia.org/T383845#10614417 (10Scott_French) I've now posted two (coupled) patch series that represent the end-game for migrating mw-web and mw-api-ext to 8.1 and putting our capacity distribution back to... [23:05:31] 06serviceops, 13Patch-For-Review, 07PHP 8.1 support: Update PCRE in PHP 8.1 images to PCRE 10.39 or newer - https://phabricator.wikimedia.org/T386006#10614978 (10Scott_French) Following up, I had a bit of time this afternoon to prepare //most// (but not yet all) of the patches necessary to consolidate into `... [23:07:31] 06serviceops: Migrate mw-cron to PHP 8.1 - https://phabricator.wikimedia.org/T387916#10614979 (10Reedy) [23:07:44] 06serviceops, 10MediaWiki-Platform-Team (Radar), 13Patch-For-Review, 07Wikimedia-Performance-recommendation: Add support for JIT in PHP 8.1 images - https://phabricator.wikimedia.org/T384294#10614980 (10Reedy) [23:07:52] 06serviceops, 10Arc-Lamp: Gather PHP 8.1 profiling data - https://phabricator.wikimedia.org/T385199#10614981 (10Reedy) [23:08:30] 06serviceops, 10MediaWiki-Platform-Team (Radar): Performance assessment of PHP 8.1 - https://phabricator.wikimedia.org/T383805#10614982 (10Reedy)