[08:59:49] hello traffic folks! [09:00:34] If you have time I'd need somebody to work with me on https://gerrit.wikimedia.org/r/c/operations/puppet/+/1123622, the last time it caused an outage (the endless redirect loop for www.wikipedia.org/) [09:01:19] this time the faulty bit is removed, but when we deploy it I'd need somebody from traffic to watch the status of the cp caches with me to verify that we don't end up another time in some trouble :D [09:01:24] cc: claime --^ [09:01:31] (bonjour :) [09:19:03] 06Traffic: purged event lag keeps piling up in codfw topics after switchover - https://phabricator.wikimedia.org/T389707#10666713 (10Fabfur) p:05High→03Medium Purged restarted everywhere, leaving this open for investigation and lowering priority [09:19:48] elukey: I'm watching it now [09:38:40] +1 [09:51:52] thanks! Going to wait for somebody from serviceops [11:15:21] elukey: yo [11:17:09] you can scap with no image build and pause-after-testserver deploy it, then when it reaches testservers, use XWD to check it works correctly? [11:17:12] Also running httpbb [11:20:03] claime: yo, yes yes I just wanted to coordinate with you first on the steps [11:20:12] I'd be happier to do it with you around [11:20:22] maybe during a mw infra windows [11:20:23] sure [11:20:24] wdyt? [11:20:43] I woke up too late for this morning's :') [11:21:25] I think we can do it now, there's nothing until 1300UTC [11:22:23] nah we can book another time window [11:23:15] https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20250326T1000 ? [11:23:18] would it work? [11:24:00] sure [12:31:28] 06Traffic, 10MW-on-K8s, 06serviceops, 06SRE, 10Release-Engineering-Team (Seen): Deploy mediawiki kubernetes services - https://phabricator.wikimedia.org/T321786#10667658 (10Clement_Goubert) 05Open→03In progress [15:05:38] 10netops, 06Infrastructure-Foundations, 10ops-drmrs: cr1-drmrs to asw1-b12-drmrs link down - https://phabricator.wikimedia.org/T389071#10668768 (10RobH) Replacement optics and fiber will arrive on March 26th by end of day. Not great delivery times considering I chose International Priority DHL. [16:42:50] 10Wikimedia-Apache-configuration, 06Discovery-Search, 06serviceops, 06SRE, and 3 others: www.wikipedia.org: prefilling the search box with the "search" URL parameter does not work - https://phabricator.wikimedia.org/T318285#10669644 (10Gehel) [16:43:20] 10Wikimedia-Apache-configuration, 06Discovery-Search, 06serviceops, 06SRE, and 3 others: www.wikipedia.org: prefilling the search box with the "search" URL parameter does not work - https://phabricator.wikimedia.org/T318285#10669647 (10Gehel) We're not expecting the Search Platform team to do any work on t... [16:45:30] 10Wikimedia-Apache-configuration, 06Discovery-Search, 06serviceops, 06SRE, and 4 others: www.wikipedia.org: prefilling the search box with the "search" URL parameter does not work - https://phabricator.wikimedia.org/T318285#10669663 (10simon04) Let's try to finish this on the #wikimedia-hackathon-2025 then... [16:51:37] elukey: I've taken the liberty to add it to wednesday's mid-day infra window [16:52:28] claime: thanks! [16:52:41] Let me add a note in my calendar [16:53:12] 10Wikimedia-Apache-configuration, 06Discovery-Search, 06serviceops, 06SRE, and 4 others: www.wikipedia.org: prefilling the search box with the "search" URL parameter does not work - https://phabricator.wikimedia.org/T318285#10669816 (10Clement_Goubert) >>! In T318285#10669663, @simon04 wrote: > Let's try t... [16:54:32] 10Wikimedia-Apache-configuration, 06Discovery-Search, 06serviceops, 06SRE, and 3 others: www.wikipedia.org: prefilling the search box with the "search" URL parameter does not work - https://phabricator.wikimedia.org/T318285#10669822 (10simon04) Even better, thank you! Sorry for the noise! [16:57:11] claime: ah snap I have a meeting at that time, just noticed it sigh [16:57:23] it last one hour [16:58:18] ah well, we can move the window after the Citoid/Zotero window? [16:58:27] 1200UTC [16:59:34] it should be also fine to overlap with the citoid/zotero, we rarely see any deployments for it [16:59:46] and it shouldn't impact those services [16:59:49] so we can anticipate [17:01:03] Sure [17:02:29] 10Wikimedia-Apache-configuration, 06Discovery-Search, 06serviceops, 06SRE, and 3 others: www.wikipedia.org: prefilling the search box with the "search" URL parameter does not work - https://phabricator.wikimedia.org/T318285#10669902 (10Clement_Goubert) We may do it at the end of that window or right after... [17:02:33] I'll leave it here so it's at least scheduled [17:02:38] then we can do it when we can [17:12:07] <3 [19:46:16] 06Traffic, 10Citoid, 06Editing-team, 10RESTBase Sunsetting, and 2 others: Switchover plan from restbase to api gateway for Citoid - https://phabricator.wikimedia.org/T361576#10671140 (10Mvolz) Should we resolve this? Or is there more work to do? [19:56:55] FIRING: [2x] SLOMetricAbsent: - https://alerts.wikimedia.org/?q=alertname%3DSLOMetricAbsent [19:56:58] FIRING: SLOMetricAbsent: - https://alerts.wikimedia.org/?q=alertname%3DSLOMetricAbsent [20:01:55] RESOLVED: [2x] SLOMetricAbsent: - https://alerts.wikimedia.org/?q=alertname%3DSLOMetricAbsent [20:01:58] RESOLVED: SLOMetricAbsent: - https://alerts.wikimedia.org/?q=alertname%3DSLOMetricAbsent [22:29:38] 06Traffic, 13Patch-For-Review: Upgrade Varnish from 6.0 to 7.1 - https://phabricator.wikimedia.org/T378737#10671766 (10BCornwall)