[07:45:09] 06serviceops, 10envoy, 10Observability-Metrics, 13Patch-For-Review, 10SRE Observability (FY2024/2025-Q4): Revisit default envoy histogram buckets - https://phabricator.wikimedia.org/T391333#10825121 (10elukey) Almost all patches merged, deployments will follow naturally during the next months (we cannot... [08:50:09] 06serviceops, 10Shellbox, 10wikitech.wikimedia.org: Shellbox is broken on wikitech-static due to disk fullness - https://phabricator.wikimedia.org/T338520#10825311 (10fnegri) This happened again: ` root@wikitech-static:~# df -h Filesystem Size Used Avail Use% Mounted on udev 979M 0... [08:51:35] 06serviceops, 10Shellbox, 10wikitech.wikimedia.org: Shellbox is broken on wikitech-static due to disk fullness - https://phabricator.wikimedia.org/T338520#10825315 (10fnegri) That worked: ` root@wikitech-static:~# df -h Filesystem Size Used Avail Use% Mounted on udev 979M 0 979M... [09:00:49] hi, anyone around that can help with a train issue? MW image build process keeps getting stuck and I can't roll back the train [09:06:42] image build process is working again [09:47:06] 06serviceops, 06MediaWiki-Platform-Team, 07Epic: Migrate Wikimedia production from PHP 8.1 to PHP 8.3 - https://phabricator.wikimedia.org/T360995#10825425 (10MSantos) @Jdforrester-WMF and @Krinkle, if I may ask, would it be possible to proceed directly to 8.4, or should we first upgrade to 8.3? [09:58:14] 06serviceops, 06DBA, 10Editing-team (Tracking), 10MW-1.44-notes (1.44.0-wmf.28; 2025-05-06), and 3 others: Fatal exception of type "Wikimedia\Rdbms\DBUnexpectedError: Database servers in extension1 are overloaded. In order to protect application servers, t... - https://phabricator.wikimedia.org/T393513#10825458 [10:09:46] 06serviceops, 10Observability-Metrics, 10SRE Observability (FY2024/2025-Q4): Repeated library panels in Grafana showing only after refresh, not on first load - https://phabricator.wikimedia.org/T384831#10825538 (10jijiki) 05Open→03Resolved [11:06:37] btw, we have had 100K memcached error from mediawiki in that past 15 minutes https://logstash.wikimedia.org/goto/e07211b9c8b85b1e7712f1209cac2805 [11:07:02] and since morning we found generally memcached starts throwing errors like this more often in the past month [11:09:59] I think effie rolled an mcrouter update earlier. Would assume the errors are a side effect [11:12:07] Amir1: that was the rollout [11:13:07] ah okay [11:13:33] there is a lot of nois in the ops channel, I wrote it twice :p [13:27:30] 06serviceops, 10MW-on-K8s: Add a way to suspend CronJobs - https://phabricator.wikimedia.org/T394409 (10Clement_Goubert) 03NEW [13:45:09] 06serviceops, 06MediaWiki-Platform-Team, 07Epic: Migrate Wikimedia production from PHP 8.1 to PHP 8.3 - https://phabricator.wikimedia.org/T360995#10826298 (10Jdforrester-WMF) >>! In T360995#10825425, @MSantos wrote: > @Jdforrester-WMF and @Krinkle, if I may ask, would it be possible to proceed directly to 8.... [13:53:23] hi folks. we are considering lowering the TTLs for dyna.wm.org and upload.wm.org to 240 from the current 300. [13:53:31] more info in https://phabricator.wikimedia.org/T394312 [13:54:29] there are bunch of disc-DYNAs though that have a TTL of 300 currently [13:54:47] this change would then mean that the TTL for upload/dyna is _lower_ than that of the disc records [13:55:34] do you anticipate any issues with that? none that we imagine in Traffic but asking specifically in the context of dc-switchover processes and such, which I assume are either independent of that or have checks in places to poll for the correct records [13:56:44] also note that there was a point in time (before Feb 2024) when the dyna TTL was _more_ than these records (600) [13:57:11] 06serviceops, 10MW-on-K8s, 13Patch-For-Review: Add a way to suspend CronJobs - https://phabricator.wikimedia.org/T394409#10826370 (10Clement_Goubert) Copied from CR because it will need to be taken into account in further patches: >>! In https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/114662... [13:58:15] sukhe: hmm let me check the cookbook [13:58:22] thanks claime <3 [13:59:02] 25 │ logger.info('Yes, that is 5 minutes. Blame Joe.') [13:59:03] 26 │ time.sleep(295) [13:59:06] yeah we don't grab it [13:59:25] We set it back to 300 [13:59:28] that's hardcoded [13:59:44] cool so it's an explicit delay that does not depend on dyna's TTL [14:00:10] yeah, but we also reset it to 300 unconditionally [14:00:23] (at the end of the process) [14:01:53] I don't think the TTL for upload/dyna being higher or lower than the disc records is an issue in itself though [14:02:48] fwiw, I can't see anyway this would cause issues [14:02:51] yeah, makes sense. we just thought we should check explicitly! thanks! [14:02:56] they are quite separate entities [14:03:11] akosiaris: thanks, we figured as much, but this is the first time the TTL will actually be lower than the discovery ones, and hence [14:03:16] 06serviceops, 10Page Content Service, 10Content-Transform-Team (Work In Progress): Rollout more wikis: week 4 - https://phabricator.wikimedia.org/T393591#10826468 (10Jgiannelos) 05Open→03Resolved [14:03:28] Yeah it's good to check, no worries [14:33:40] 06serviceops, 06Content-Transform-Team: Bump memory of testreduce1002 - https://phabricator.wikimedia.org/T393904#10826716 (10cscott) 05Open→03Resolved a:03cscott [14:54:47] 06serviceops, 10MW-on-K8s: Investigate startingDeadlineSeconds setting for kubernetes CronJobs - https://phabricator.wikimedia.org/T394423 (10Clement_Goubert) 03NEW [15:04:49] 06serviceops, 06MediaWiki-Platform-Team, 07Epic: Migrate Wikimedia production from PHP 8.1 to PHP 8.3 - https://phabricator.wikimedia.org/T360995#10826879 (10MSantos) >>! In T360995#10826298, @Jdforrester-WMF wrote: >>>! In T360995#10825425, @MSantos wrote: >> @Jdforrester-WMF and @Krinkle, if I may ask, wou... [16:24:13] 06serviceops, 10Page Content Service: mobileapps consistently 503s when a summary of an image is requested - https://phabricator.wikimedia.org/T394433 (10hnowlan) 03NEW [17:25:53] 06serviceops, 10function-orchestrator, 10Abstract Wikipedia team (25Q4 (Apr–Jun)), 07OKR-Work: Enable memcached in the orchestrator - https://phabricator.wikimedia.org/T391986#10827581 (10cmassaro)