[05:48:53] 06Traffic, 06Privacy Engineering, 06Trust-and-Safety, 07Chinese-Sites, 07Privacy: zhwikipedia, zhwikinews API request for every article, links from sitenotice to external, unaffiliated sites - https://phabricator.wikimedia.org/T375253#10837809 (10Hakimi97) > * I'm pretty sure you're not allowed to link t... [06:55:49] 06Traffic, 10MediaWiki-extensions-Disambiguator: Return HTTP status code 300 for disambiguation pages - https://phabricator.wikimedia.org/T332454#10837865 (10SD0001) [07:27:06] 06Traffic, 06Data-Engineering-Radar, 10Observability-Logging, 13Patch-For-Review: Shutdown varnishkafka webrequest instances - https://phabricator.wikimedia.org/T393772#10837899 (10Fabfur) [07:39:41] 06Traffic, 06Data-Engineering-Radar, 10Observability-Logging, 13Patch-For-Review: Shutdown varnishkafka webrequest instances - https://phabricator.wikimedia.org/T393772#10837925 (10Fabfur) [07:40:13] 06Traffic, 06Data-Engineering-Radar, 10Observability-Logging, 13Patch-For-Review: Shutdown varnishkafka webrequest instances - https://phabricator.wikimedia.org/T393772#10837931 (10Fabfur) 05Open→03In progress varnishkafka webrequest has been shut down on all cache hosts [08:36:10] 10netops, 06Infrastructure-Foundations, 10Observability-Alerting, 13Patch-For-Review: Migrate network icinga alerts to gNMI/prometheus - https://phabricator.wikimedia.org/T388641#10838021 (10ayounsi) It's actually multiple of them: * `gnmi_bfd_peer_session_state{}` missing in codfw, while it used to work u... [08:48:49] 10netops, 06Infrastructure-Foundations, 10Observability-Alerting, 13Patch-For-Review: Migrate network icinga alerts to gNMI/prometheus - https://phabricator.wikimedia.org/T388641#10838053 (10cmooney) >>! In T388641#10838021, @ayounsi wrote: > * `gnmi_interfaces_interface_state_counters_in_fcs_errors{}` mi... [09:23:20] 10netops, 06Infrastructure-Foundations, 10Observability-Alerting, 13Patch-For-Review: Migrate network icinga alerts to gNMI/prometheus - https://phabricator.wikimedia.org/T388641#10838158 (10ayounsi) Restarting gNMIc in esams fixed the issue for `gnmi_interfaces_interface_state_counters_out_errors{}`. It s... [09:43:39] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade core routers to Junos 23.4R2 - https://phabricator.wikimedia.org/T364092#10838224 (10ayounsi) [10:35:48] 06Traffic, 06Project-Admins: Create a #HaproxyKafka tag - https://phabricator.wikimedia.org/T394758 (10Fabfur) 03NEW [10:37:49] 06Traffic, 10HaproxyKafka, 06Project-Admins: Create a #HaproxyKafka tag - https://phabricator.wikimedia.org/T394758#10838376 (10Peachey88) 05Open→03Resolved a:03Peachey88 Project created. Please feel free to encourage users to watch the project etc :). [11:06:12] 06Traffic, 06Data-Engineering, 06Data-Engineering-Radar, 10HaproxyKafka, and 2 others: New software: haproxykafka - https://phabricator.wikimedia.org/T370668#10838476 (10Fabfur) [11:06:22] 06Traffic, 10HaproxyKafka: haproxykafka minor features - https://phabricator.wikimedia.org/T374128#10838477 (10Fabfur) [11:06:44] 06Traffic, 10HaproxyKafka, 13Patch-For-Review: Enable SSL client authentication on haproxykafka - https://phabricator.wikimedia.org/T379776#10838479 (10Fabfur) [11:06:57] 06Traffic, 10HaproxyKafka, 10Sustainability (Incident Followup): Avoid logging errors per produced message - https://phabricator.wikimedia.org/T380583#10838480 (10Fabfur) [11:07:04] sorry for the noise, I'm retagging some tasks [11:08:03] 06Traffic, 10HaproxyKafka, 10Data-Engineering (Q4 2025 April 1st - June 30th), 10DPE HAProxy Migration: Add HAproxy termination field to webrequest - https://phabricator.wikimedia.org/T387454#10838486 (10Fabfur) [11:08:17] 06Traffic, 10HaproxyKafka, 10DPE HAProxy Migration: [HAProxy migration] Some 200 requests in VK are logged as 400 in HAProxy - https://phabricator.wikimedia.org/T387451#10838497 (10Fabfur) [11:08:26] 06Traffic, 10HaproxyKafka, 10DPE HAProxy Migration: Make webrequest_frontend being ingested using the in-data `dt` field - https://phabricator.wikimedia.org/T388397#10838498 (10Fabfur) [11:09:40] 06Traffic, 10HaproxyKafka: Split MessageBuffer configuration for different processing channels - https://phabricator.wikimedia.org/T386801#10838501 (10Fabfur) [11:09:46] 06Traffic, 10HaproxyKafka, 10DPE HAProxy Migration: Update HAProxyKafka kafka-timestamp type - https://phabricator.wikimedia.org/T389521#10838502 (10Fabfur) [11:09:58] 06Traffic, 06Data-Engineering-Radar, 10HaproxyKafka, 10Data-Platform-SRE (2025.05.02 - 2025.05.23), and 2 others: Replicate current low-message alerting from VarnishKafka - https://phabricator.wikimedia.org/T391810#10838503 (10Fabfur) [11:10:11] 06Traffic, 10HaproxyKafka, 13Patch-For-Review: haproxykafka service isn't restarted when upgraded - https://phabricator.wikimedia.org/T393016#10838504 (10Fabfur) [11:10:22] 06Traffic, 06Data-Engineering-Radar, 10HaproxyKafka, 10Observability-Logging, 13Patch-For-Review: Shutdown varnishkafka webrequest instances - https://phabricator.wikimedia.org/T393772#10838508 (10Fabfur) [11:11:34] 06Traffic, 10Data-Platform, 10Data-Platform-SRE (2025.05.02 - 2025.05.23), 10Experimentation Lab (Experiment Platform Sprint 6), 13Patch-For-Review: Include all CDN SANs on eventgate-analytics-external.discovery.wmnet:4692 TLS certificate - https://phabricator.wikimedia.org/T394437#10838513 (10BTullis... [11:39:49] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade core routers to Junos 23.4R2 - https://phabricator.wikimedia.org/T364092#10838631 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=8c92db5f-18b6-481b-8642-01c1d92b5cb0) set by cmooney@cumin1003 for 2:00:00 on 10 host(s) and their servi... [12:21:52] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade core routers to Junos 23.4R2 - https://phabricator.wikimedia.org/T364092#10838856 (10ayounsi) [12:28:04] 06Traffic, 06Experimentation Lab, 13Patch-For-Review: SDS 2.4.4 Edge Uniques Production Cookie Deployment - https://phabricator.wikimedia.org/T391411#10838870 (10Vgutierrez) [12:52:37] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade core routers to Junos 23.4R2 - https://phabricator.wikimedia.org/T364092#10838979 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=f40f3f46-731d-46ef-9db5-647d735907d6) set by cmooney@cumin1003 for 3:00:00 on 1 host(s) and their servic... [12:56:41] 06Traffic, 10Prod-Kubernetes, 06serviceops, 07Kubernetes, 13Patch-For-Review: Handling inbound IPIP traffic on low traffic LVS k8s based realservers - https://phabricator.wikimedia.org/T352956#10839001 (10akosiaris) Unfortunately the 2 patches above didn't work. For ml-staging-codfw, just because it's st... [13:01:49] 10netops, 06Infrastructure-Foundations: Downgrade pfw1-codfw to Junos 23.4R2-S3 - https://phabricator.wikimedia.org/T393996#10839059 (10Jgreen) [13:12:55] 06Traffic, 10Liberica: control plane should fetch default gateway MAC address dynamically - https://phabricator.wikimedia.org/T393903#10839128 (10Vgutierrez) 05Open→03Resolved [13:38:13] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw: setup MPC10E-10C and SCBE3 - https://phabricator.wikimedia.org/T393552#10839281 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=5afc68ed-eba5-4a71-b833-f809ae58201b) set by cmooney@cumin1003 for 4:00:00 on 11... [13:59:13] 06Traffic, 10Prod-Kubernetes, 06serviceops, 07Kubernetes, 13Patch-For-Review: Handling inbound IPIP traffic on low traffic LVS k8s based realservers - https://phabricator.wikimedia.org/T352956#10839348 (10akosiaris) `lang=bash nobody@wmfdebug:/$ ip link 1: lo: mtu 65536 qdisc noque... [15:18:08] 06Traffic: Move host normalization to haproxy - https://phabricator.wikimedia.org/T392880#10839706 (10Fabfur) [15:18:21] 06Traffic, 06DC-Ops, 10ops-eqiad, 06SRE: Q3:test NIC for lvs1017 or lvs1018 - https://phabricator.wikimedia.org/T387145#10839707 (10BCornwall) @VRiley-WMF! Is there something still needed on our side or are we good to go? [15:20:03] 06Traffic, 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team: beta cluster: profile::cache::varnish::frontend needs to reload varnish-frontend.service when /etc/varnish/blocked-nets.inc.vcl changes - https://phabricator.wikimedia.org/T358887#10839729 (10ssingh) Hi @bd808: we discussed this in the Tra... [15:36:54] 06Traffic, 06Experimentation Lab, 13Patch-For-Review: SDS 2.4.4 Edge Uniques Production Cookie Deployment - https://phabricator.wikimedia.org/T391411#10839871 (10Vgutierrez) [15:40:06] 06Traffic, 13Patch-For-Review: Move varnish pseudo-headers to vmod_var variables - https://phabricator.wikimedia.org/T373550#10839899 (10BCornwall) [15:51:10] 06Traffic, 06Experimentation Lab, 13Patch-For-Review: SDS 2.4.4 Edge Uniques Production Cookie Deployment - https://phabricator.wikimedia.org/T391411#10839948 (10BBlack) [17:11:07] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw: setup MPC10E-10C and SCBE3 - https://phabricator.wikimedia.org/T393552#10840439 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=e24daea6-0330-4b79-bf33-b9e0f9709a10) set by cmooney@cumin1003 for 2:00:00 on 11... [18:13:52] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade core routers to Junos 23.4R2 - https://phabricator.wikimedia.org/T364092#10840704 (10cmooney) [18:14:37] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade core routers to Junos 23.4R2 - https://phabricator.wikimedia.org/T364092#10840709 (10cmooney) [18:25:02] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw: setup MPC10E-10C and SCBE3 - https://phabricator.wikimedia.org/T393552#10840770 (10cmooney) 05Open→03Resolved a:03cmooney Ok this is now complete. A few niggles along the way that were sorted out with multiple re-seat... [18:55:01] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw: setup MPC10E-10C and SCBE3 - https://phabricator.wikimedia.org/T393552#10840954 (10cmooney) 05Resolved→03Open Actually there are a few bits like the license and the inventory items in Netbox to be completed which I'll take o... [19:47:45] 06Traffic, 06collaboration-services, 10Release-Engineering-Team (Radar): Separate Gerrit https and ssh/git hostnames - https://phabricator.wikimedia.org/T394271#10841273 (10Dzahn) >>! In T394271#10822942, @bd808 wrote: > Naming is always tricky, but I wonder if putting `ssh` in the hostname would be helpful... [21:38:28] 10Domains, 06Traffic: [toolforge] transfer/adopt toolsbeta.org domain to the foundation - https://phabricator.wikimedia.org/T362253#10841860 (10Dzahn) @bcornwall fyi, this is an example for domains that are owned by WMF, managed in MarkMonitor but not pointed to the standard WMF name servers. See T362253#9... [21:43:23] 06Traffic, 10DNS, 06serviceops, 06SRE: Create redirect from tj.*.org to tg.*.org - https://phabricator.wikimedia.org/T393803#10841877 (10Dzahn) [21:48:34] 06Traffic, 10DNS, 06serviceops, 06SRE: Create redirect from tj.*.org to tg.*.org - https://phabricator.wikimedia.org/T393803#10841893 (10Dzahn) Added serviceops because this would have to be added in Apache/appserver redirects (as opposed to redirecting entire domains in ncredir service, which would be tra... [22:24:57] 06Traffic, 06Experimentation Lab: libvmod_wmfuniq: add stats counter for cookie values of incorrect length - https://phabricator.wikimedia.org/T394862 (10BBlack) 03NEW [22:25:27] 06Traffic, 06Experimentation Lab: libvmod_wmfuniq: add stats counter for cookie values of incorrect length - https://phabricator.wikimedia.org/T394862#10841987 (10BBlack) p:05Triage→03Medium a:05KOfori→03BBlack [23:27:13] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw: codfw: BAD PEM3 on cr2-codfw - https://phabricator.wikimedia.org/T394868 (10Papaul) 03NEW