[00:41:44] 10Acme-chief, 10Beta-Cluster-Infrastructure, 07Beta-Cluster-reproducible: Warning about /etc/acmecerts/unified contents during puppet run on deployment-cache-text08 & deployment-cache-upload08 - https://phabricator.wikimedia.org/T399419#11183331 (10BCornwall) Hm, it sounded like the directory should have bee... [01:02:39] 06Traffic, 06MediaWiki-Platform-Team, 06Reader Experience Team, 10MobileFrontend (Core PHP): Toggling desktop view doesn't toggle user back into mobile mode - https://phabricator.wikimedia.org/T403866#11183364 (10Krinkle) The change has been deployed, @Etonkovidova. Similar to the previous change, due to... [07:48:21] 06Traffic, 06serviceops, 10WE4.2 Bot detection (WE4.2 hCaptcha account creation trial): Investigate options for per-wiki, percentage-based rollout of hCaptcha - https://phabricator.wikimedia.org/T404184#11183709 (10kostajh) a:03kostajh There's some discussion [[ https://wikimedia.slack.com/archives/C01DFMX... [10:06:40] 06Traffic, 06SRE, 13Patch-For-Review, 07User-notice: Block traffic from user-agents not honoring our policy - https://phabricator.wikimedia.org/T400119#11184198 (10Joe) >>! In T400119#11115049, @TheDJ wrote: > Yeah getting the swagger spec via `curl https://api.wikimedia.org/core/v1/wikipedia/en/search/pag... [12:13:43] fabfur: I'm gonna restart ATS on cp2041, because I have the sinking feeling that my multi-dc.lua change wasn't picked up by the reload [12:13:48] fabfur: objections? [12:16:10] $ sudo traffic_ctl config status [12:16:11] Apache Traffic Server - traffic_server - 9.2.11 - (build # 061813 on Jun 18 2025 at 13:10:58) [12:16:11] Started at Tue Sep 2 13:03:22 2025 [12:16:11] Last reconfiguration at Tue Sep 16 10:39:30 2025 [12:16:11] Configuration is current [12:16:27] I think it picked up the correct configuration but OK for the restart [12:16:56] given lua scripting engine could behave differently, if you want to be extra sure depool it briefly while restarting [12:17:25] ack [12:17:28] cc hnowlan [12:17:35] sgtm [12:17:45] fabfur: What I'm figuring is since I removed a pparam, it may not have worked correctly [12:18:02] 👍 [12:18:07] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad: rows C/D Upgrade Tracking - https://phabricator.wikimedia.org/T404609#11184704 (10cmooney) >>! In T404609#11181649, @RobH wrote: > @cmooney: What do you think is the best way to go about migrating these connections on upcoming C... [12:20:24] fabfur: restart via systemctl restart trafficserver.service ? [12:21:37] yep [12:22:03] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad: rows C/D Upgrade Tracking - https://phabricator.wikimedia.org/T404609#11184711 (10cmooney) @RobH @Jclark-ctr there is also another way we could try to approach this so may as well mention it now before we start planning. Rack-b... [12:22:03] yeah that was it [12:22:12] sorry wrong test [12:22:41] nope still borked, so it's something not working in the rewrite chain [12:26:45] I have no idea what's broken, because the port does get remapped, but not the host [12:26:57] (or it gets remapped again) [12:40:45] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad: rows C/D Upgrade Tracking - https://phabricator.wikimedia.org/T404609#11184793 (10Jclark-ctr) @cmooney I’m flexible to try either way. Maybe a mix could work? We could start with roles that aren’t single points of failure and ar... [13:09:00] FIRING: PurgedHighBacklogQueue: Large backlog queue for purged on cp2041:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://grafana.wikimedia.org/d/RvscY1CZk/purged?var-datasource=codfw%20prometheus/ops&var-instance=cp2041 - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighBacklogQueue [13:09:11] That's probably me [13:09:24] fabfur: I depooled the server but it looks like it's still getting traffic that isn't me... [13:16:45] ok I have litteraly no idea what's going on, I get debug logging for requetsts that are not mine, and don't get logging for my own requests. I give up. [13:17:01] I'll try to set up some time with someone from traffic when y'all have time to assist [13:17:14] I'll run puppet to put the machine back in state [13:24:00] RESOLVED: [2x] PurgedHighBacklogQueue: Large backlog queue for purged on cp2041:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://grafana.wikimedia.org/d/RvscY1CZk/purged?var-datasource=codfw%20prometheus/ops&var-instance=cp2041 - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighBacklogQueue [13:42:55] sry late lunch, wdyt is going on? [13:51:40] 06Traffic, 06Data-Engineering: Reduce noise from duplicate sequence-gap alerts on HaProxy-webrequests - https://phabricator.wikimedia.org/T401383#11185133 (10Fabfur) We deployed a change in HAProxy logging (see T403176) to avoid sending non-utf8 encoded headers to DLQ, this *could* also affect this issue as we... [13:58:39] 06Traffic, 06DC-Ops, 10ops-codfw, 06SRE, 13Patch-For-Review: Q4:rack/setup/install cp20[43-58] codfw - https://phabricator.wikimedia.org/T392851#11185177 (10Jhancock.wm) @elukey 2049 was powered off. once i powered it on the nic came up. I'll not set the root for 2053-8 [14:06:11] 06Traffic, 06Fundraising-Backlog, 06Fundraising-Tech-Roadmap, 10MediaWiki-extensions-CentralNotice, 06SRE: Set expiry time for GeoIP cookies - https://phabricator.wikimedia.org/T122097#11185242 (10Pcoombe) For fundraising banners we use the country from `mw.centralNotice.data.country` (which allows us to... [14:23:37] 06Traffic, 06Fundraising-Backlog, 06Fundraising-Tech-Roadmap, 10MediaWiki-extensions-CentralNotice, 06SRE: Set expiry time for GeoIP cookies - https://phabricator.wikimedia.org/T122097#11185307 (10AKanji-WMF) @XenoRyet and I discussed getting this into our next Sprint as a stretch. [16:12:23] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad: rows C/D Upgrade Tracking - https://phabricator.wikimedia.org/T404609#11185881 (10RobH) I overthought this, we should just move them with an SFP-T to the new port and worry about reimage and migration to full 10G later. [16:13:31] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad: rows C/D Upgrade Tracking - https://phabricator.wikimedia.org/T404609#11185897 (10RobH) [16:22:32] 06Traffic, 06DC-Ops, 10ops-codfw, 06SRE, 13Patch-For-Review: Q4:rack/setup/install cp20[43-58] codfw - https://phabricator.wikimedia.org/T392851#11185938 (10elukey) @Jhancock.wm perfect I can confirm that the provision cookbook ran fine (the test-cookbook version I mean). At this point we could use it to... [16:24:03] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad: rows C/D Upgrade Tracking - https://phabricator.wikimedia.org/T404609#11185965 (10RobH) [16:42:13] hello traffic friends - FYI, there are some `CRITICAL: Service pybal.service has not been restarted after /etc/pybal/pybal.conf was changed (gt 1h).` alerts for lvs1019, lvs2013, and lvs2014. [16:42:13] checking the mtime on pybal.conf vs. diffs in puppet logs, it looks like this is from when [0] was applied, but I also see there's an as-yet unmerged revert for that. [16:42:13] in any case, I wanted to surface here since it's unclear to me what the desired state is :) [16:42:13] [0] https://gerrit.wikimedia.org/r/c/operations/puppet/+/1188309 [16:48:10] 06Traffic, 06DC-Ops, 10ops-codfw, 06SRE, 13Patch-For-Review: Q4:rack/setup/install cp20[43-58] codfw - https://phabricator.wikimedia.org/T392851#11186015 (10Jhancock.wm) yeah that's probably a good idea to do that. I finally got around to getting 53-58 iped. should be done by end of day so they're ready... [17:29:28] 06Traffic, 06DC-Ops, 10ops-codfw, 06SRE, 13Patch-For-Review: Q4:rack/setup/install cp20[43-58] codfw - https://phabricator.wikimedia.org/T392851#11186239 (10Jhancock.wm) update. everything but 2056 is ready. that one has a physical issue. the console and idrac connections are on a removeable card on thes... [21:19:58] 06Traffic, 10Phabricator, 10Release-Engineering-Team (Radar): Phabricator videos fail in Firefox ("Range" request gets 503 from Varnish) - https://phabricator.wikimedia.org/T397661#11187396 (10Aklapper)