[06:18:35] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade core routers to Junos 23.4R2 - https://phabricator.wikimedia.org/T364092#10842593 (10ayounsi) [07:45:51] 06Traffic, 10Prod-Kubernetes, 06serviceops, 07Kubernetes, 13Patch-For-Review: Handling inbound IPIP traffic on low traffic LVS k8s based realservers - https://phabricator.wikimedia.org/T352956#10842714 (10akosiaris) >>! In T352956#10839001, @akosiaris wrote: > #### Long-term > > We probably want to mini... [08:21:52] 06Traffic, 10Prod-Kubernetes, 06serviceops, 07Kubernetes, 13Patch-For-Review: Handling inbound IPIP traffic on low traffic LVS k8s based realservers - https://phabricator.wikimedia.org/T352956#10842876 (10Vgutierrez) >>! In T352956#10839348, @akosiaris wrote: > `lang=bash > nobody@wmfdebug:/$ ip link > 1... [08:27:39] o/ just a heads-up, adding another rule to the ATS gateway script. Nothing particularly novel https://gerrit.wikimedia.org/r/c/operations/puppet/+/1148285 [08:29:50] * vgutierrez looking.. [08:32:12] hnowlan: nothing to be worried about apparently :D [08:34:34] most likely :D thanks! [09:08:54] 06Traffic, 10Prod-Kubernetes, 06serviceops, 07Kubernetes, 13Patch-For-Review: Handling inbound IPIP traffic on low traffic LVS k8s based realservers - https://phabricator.wikimedia.org/T352956#10843107 (10akosiaris) staging-eqiad with an MTU of 1460 as well. `lang=bash 2: eth0@if841: 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade core routers to Junos 23.4R2 - https://phabricator.wikimedia.org/T364092#10843115 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=048b70e3-25f1-4871-b6c8-5ea7b074de1e) set by ayounsi@cumin1002 for 2:00:00 on 2 host(s) and their servic... [09:38:51] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw: setup MPC10E-10C and SCBE3 - https://phabricator.wikimedia.org/T393552#10843244 (10cmooney) 05Open→03Resolved License is now applied and inventory items updated for cr1-codfw and cr2-codfw. [09:40:59] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade core routers to Junos 23.4R2 - https://phabricator.wikimedia.org/T364092#10843254 (10ayounsi) [09:41:16] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade core routers to Junos 23.4R2 - https://phabricator.wikimedia.org/T364092#10843256 (10ayounsi) 05Open→03Resolved All done! Thank you all. [10:10:40] Hey! I am trying to debug an issue with connections hanging on PCS. Currently testing on staging. Can somebody try this request from inside a running mobileapps staging container? [10:11:55] `curl "http://localhost:6500/w/api.php?action=query&format=json&titles=%C4%B0znik_%28District%29%2C_Bursa` [10:12:17] or better `curl -v` [10:12:51] cc hnowlan ^ [10:14:35] sure [10:14:59] I think i am missing the host header [10:15:03] This is better: curl "http://localhost:6500/w/api.php?action=query&format=json&titles=%C4%B0znik_%28District%29%2C_Bursa" -H "host: sw.wikiquote.org" [10:15:49] one sec, curl isn't installed on the pod [10:19:39] nemo-yiannis: I get a 302 [10:19:49] can you paste me the whole verbose output ? [14:13:42] 06Traffic, 10Data-Engineering (Q4 2025 April 1st - June 30th), 13Patch-For-Review: Clean-up varnishkafka webrequest leftovers in Hadoop-world - https://phabricator.wikimedia.org/T394011#10844393 (10JAllemandou) [14:13:49] 06Traffic, 06Experimentation Lab, 13Patch-For-Review: SDS 2.4.4 Edge Uniques Production Cookie Deployment - https://phabricator.wikimedia.org/T391411#10844394 (10Vgutierrez) [14:21:03] 06Traffic: Create VTC tests for HAProxy - https://phabricator.wikimedia.org/T393770#10844420 (10Fabfur) 05Open→03In progress p:05Triage→03Medium [14:45:17] hello, can I help with the deployment of https://gerrit.wikimedia.org/r/c/operations/puppet/+/1135387 (puppet hiera change for trafficserver)? [14:52:01] federico3: if all tests are ok, I think you can merge and let puppet handle this automatically, bringing updated conf on each host [15:07:26] yeah. federico3, let us know if you prefer that we roll this instead. [15:09:01] I'm ok with all options, whichever is easier for you. I can merge it right now if wanted [15:09:32] sure, go ahead [15:09:58] usually we recommend disabling Puppet on A:cp for ATS changes, but this one is fairly "regular" [15:10:13] usually owners merge the patches assuming they have the privileges :) [15:10:43] ok, I'm running the merge [15:10:46] thanks [18:27:15] 06Traffic, 10Prod-Kubernetes, 06serviceops, 07Kubernetes, 13Patch-For-Review: Handling inbound IPIP traffic on low traffic LVS k8s based realservers - https://phabricator.wikimedia.org/T352956#10845430 (10cmooney) What might be worth testing is if PMTUD works. i.e. send a UDP packet to a POD IP of 1500... [18:45:40] FIRING: VarnishHighThreadCount: Varnish's thread count on cp1114:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://grafana.wikimedia.org/d/wiU3SdEWk/cache-host-drilldown?viewPanel=99&var-site=eqiad&var-instance=cp1114 - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [19:18:15] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw: BAD PEM3 on cr2-codfw - https://phabricator.wikimedia.org/T394868#10845555 (10Papaul) ` Case 2025-0520-703157 has been updated by Mathias Zuniga UPDATE HAS BEEN ADDED: Hello Team, Please could you bring me the following com... [19:30:40] FIRING: [2x] VarnishHighThreadCount: Varnish's thread count on cp1114:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://grafana.wikimedia.org/d/wiU3SdEWk/cache-host-drilldown?viewPanel=99&var-site=eqiad&var-instance=cp1114 - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [19:50:40] RESOLVED: VarnishHighThreadCount: Varnish's thread count on cp1114:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://grafana.wikimedia.org/d/wiU3SdEWk/cache-host-drilldown?viewPanel=99&var-site=eqiad&var-instance=cp1114 - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [22:06:50] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw: BAD PEM3 on cr2-codfw - https://phabricator.wikimedia.org/T394868#10846032 (10Papaul) ` Case 2025-0520-703157 has been updated by Mathias Zuniga UPDATE HAS BEEN ADDED: Hi Papaul, Thank you for your update, I have opened a t... [22:09:22] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw: BAD PEM3 on cr2-codfw - https://phabricator.wikimedia.org/T394868#10846033 (10Papaul) p:05Triage→03High a:03Jhancock.wm