[00:26:44] 06Traffic, 06Data-Engineering, 06Data-Platform-SRE, 10Wikidata, 10Wikidata Analytics: Airflow processes to import dump logs and generate monthly metrics - https://phabricator.wikimedia.org/T403159#11170582 (10Ottomata) I mean, we want the Kafka logging mirror maker piece anyway for T304373, but I think w... [02:30:02] 06Traffic, 06MediaWiki-Platform-Team, 06Reader Experience Team, 10MobileFrontend (Core PHP): Toggling desktop view doesn't toggle user back into mobile mode - https://phabricator.wikimedia.org/T403866#11170665 (10Krinkle) Summary: * The "Request desktop site" button in Safari Mobile works both with the ol... [06:54:06] 06Traffic, 06Data-Engineering, 06SRE: Add pageview information to turnilo's webrequest_sampled_live (is_pageview is always "-") - https://phabricator.wikimedia.org/T402612#11170835 (10JAllemandou) [08:05:46] 06Traffic, 10HaproxyKafka, 10Data-Platform-SRE (2025.09.05 - 2025.09.26), 13Patch-For-Review: Replicate current low-message alerting from VarnishKafka - https://phabricator.wikimedia.org/T391810#11171078 (10Fabfur) 05Open→03Resolved [08:14:58] 06Traffic, 06Data-Engineering: Reduce noise from duplicate sequence-gap alerts on HaProxy-webrequests - https://phabricator.wikimedia.org/T401383#11171106 (10Fabfur) [08:14:59] 06Traffic, 10HaproxyKafka, 10DPE HAProxy Migration: [HAProxy migration] Some 200 requests in VK are logged as 400 in HAProxy - https://phabricator.wikimedia.org/T387451#11171109 (10Fabfur) →14Duplicate dup:03T401383 [09:34:10] 10netops, 06Infrastructure-Foundations, 06SRE: mr1 port utilization alerts shouldn't mention hash page in their IRC logs - https://phabricator.wikimedia.org/T281055#11171385 (10ayounsi) 05Open→03Resolved Those alerts got moved to AM for the core routers and switches. They are not alerting for managem... [09:36:51] FIRING: FermMSS: Unexpected MSS value on 10.2.2.38:8082 @ druid1012 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=4&var-site=eqiad&var-cluster=insetup - https://alerts.wikimedia.org/?q=alertname%3DFermMSS [09:50:51] FIRING: FermMSS: Unexpected MSS value on 10.2.2.38:8082 @ druid1012 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=4&var-site=eqiad&var-cluster=druid_public - https://alerts.wikimedia.org/?q=alertname%3DFermMSS [09:51:40] FIRING: [4x] VarnishHighThreadCount: Varnish's thread count on cp3066:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [09:51:51] RESOLVED: FermMSS: Unexpected MSS value on 10.2.2.38:8082 @ druid1012 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=4&var-site=eqiad&var-cluster=insetup - https://alerts.wikimedia.org/?q=alertname%3DFermMSS [09:56:40] FIRING: [8x] VarnishHighThreadCount: Varnish's thread count on cp3066:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [09:59:21] FIRING: [2x] FermMSS: Unexpected MSS value on 10.2.2.38:8082 @ druid1012 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=4&var-site=eqiad&var-cluster=insetup - https://alerts.wikimedia.org/?q=alertname%3DFermMSS [10:01:40] FIRING: [11x] VarnishHighThreadCount: Varnish's thread count on cp3066:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [10:06:40] FIRING: [12x] VarnishHighThreadCount: Varnish's thread count on cp3066:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [10:11:40] FIRING: [12x] VarnishHighThreadCount: Varnish's thread count on cp3066:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [10:16:40] FIRING: [13x] VarnishHighThreadCount: Varnish's thread count on cp3066:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [10:21:40] FIRING: [12x] VarnishHighThreadCount: Varnish's thread count on cp3066:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [10:24:21] RESOLVED: FermMSS: Unexpected MSS value on 10.2.2.38:8082 @ druid1013 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=4&var-site=eqiad&var-cluster=insetup - https://alerts.wikimedia.org/?q=alertname%3DFermMSS [10:25:51] FIRING: [2x] FermMSS: Unexpected MSS value on 10.2.2.38:8082 @ druid1012 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=4&var-site=eqiad&var-cluster=druid_public - https://alerts.wikimedia.org/?q=alertname%3DFermMSS [10:26:40] FIRING: [11x] VarnishHighThreadCount: Varnish's thread count on cp3066:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [10:31:40] FIRING: [8x] VarnishHighThreadCount: Varnish's thread count on cp3066:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [10:36:40] RESOLVED: [6x] VarnishHighThreadCount: Varnish's thread count on cp3066:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [11:23:53] Hello, looking for a review and help deploying adding 2 hosts to druid-public eqiad 1182700: druid: Add druid druid101[2-3] to druid_public_broker VIP | https://gerrit.wikimedia.org/r/c/operations/puppet/+/1182700 [11:24:52] The services are up and running on the hosts pending a roll restart once done, thanks :) [12:13:39] 06Traffic, 10HaproxyKafka, 10Data-Platform-SRE (2025.09.05 - 2025.09.26), 07Essential-Work, 13Patch-For-Review: Replicate current low-message alerting from VarnishKafka - https://phabricator.wikimedia.org/T391810#11172142 (10Gehel) [13:46:43] FIRING: HaproxyKafkaNoMessages: Unexpected rate of produced HaproxyKafka messages by cp7001 - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaNoMessages - https://grafana.wikimedia.org/d/d3e4e37c-c1d9-47af-9aad-a08dae2b3fd5/haproxykafka?orgId=1&var-site=magru&var-instance=cp7001 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaNoMessages [14:14:50] 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Netbox: Updates for Nokia switch support - https://phabricator.wikimedia.org/T404146#11172893 (10ayounsi) Above patch worked successfully: https://netbox.wikimedia.org/extras/changelog/?request_id=7940ab40-742b-47fb-98c6-fba8e4e2989b Howev... [14:21:30] hello traffic friends - any concerns if I move ahead with merging and deploying (incrementally, with cumin) [0] at some point soon? [14:21:30] vgutierrez: I see you're doing some work on cp7001. I can wait for that to wrap up, of course. [14:21:30] [0] https://gerrit.wikimedia.org/r/c/operations/puppet/+/1184915 [14:21:52] puppet is currently disabled on A:cp [14:22:22] that is good to know :) I will wait until an all-clear from you [14:22:25] give 2 minutes [14:22:55] take your time! I have other work to keep me busy [14:25:51] FIRING: [2x] FermMSS: Unexpected MSS value on 10.2.2.38:8082 @ druid1012 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=4&var-site=eqiad&var-cluster=druid_public - https://alerts.wikimedia.org/?q=alertname%3DFermMSS [14:26:43] RESOLVED: [2x] HaproxyKafkaNoMessages: Unexpected rate of produced HaproxyKafka messages by cp7001 - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaNoMessages - https://grafana.wikimedia.org/d/d3e4e37c-c1d9-47af-9aad-a08dae2b3fd5/haproxykafka?orgId=1&var-site=magru&var-instance=cp7001 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaNoMessages [14:30:39] swfrench-wmf: all yours [14:30:50] thanks! [14:39:16] 06Traffic, 06MediaWiki-Platform-Team: Mobile domain removal: Random page doesn't work with mobile/desktop switching - https://phabricator.wikimedia.org/T404023#11173128 (10Krinkle) This is not a regression, and not something we have much control over. 1. Using Chrome Mobile (via the same real device from Brow... [14:39:26] 06Traffic, 06MediaWiki-Platform-Team: Mobile domain removal: Random page doesn't work with mobile/desktop switching - https://phabricator.wikimedia.org/T404023#11173130 (10Krinkle) [14:39:35] 06Traffic, 10MediaWiki-Platform-Team (Radar), 13Patch-For-Review, 07User-notice: [Rollout Phase 3] Enable unified mobile routing on remaining wikis - https://phabricator.wikimedia.org/T403510#11173131 (10Krinkle) [14:51:25] 06Traffic, 06SRE: Setting up Wikimedia Trust and Safety Help Center with Zendesk product: Seeking Guidance on host mapping - https://phabricator.wikimedia.org/T400952#11173200 (10ssingh) @JAbrams and I discussed this on a call today. @BCornwall will help from lead this from Traffic. The next steps are fairly... [15:32:38] 06Traffic: Add an Allow header on 405 responses - https://phabricator.wikimedia.org/T403767#11173397 (10A_smart_kitten) Noting that I've randomly spot-checked a few subdomains that still don't seem to include an `Allow:` header (or include a potentially-invalid one) with 405 responses (I'm guessing because t... [16:38:12] 06Traffic: Add an Allow header on 405 responses - https://phabricator.wikimedia.org/T403767#11173759 (10Vgutierrez) you got a nice mix of uses cases there @A_smart_kitten. stuff like gerrit and gitlab aren't behind the CDN so those haven't been impacted by this task. For the others, you're right, haproxy i... [17:46:42] 06Traffic, 10MediaWiki-Platform-Team (Radar), 13Patch-For-Review, 07User-notice: [Rollout Phase 3] Enable unified mobile routing on remaining wikis - https://phabricator.wikimedia.org/T403510#11173911 (10Krinkle) [18:26:07] FIRING: [2x] FermMSS: Unexpected MSS value on 10.2.2.38:8082 @ druid1012 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=4&var-site=eqiad&var-cluster=druid_public - https://alerts.wikimedia.org/?q=alertname%3DFermMSS [19:30:43] 06Traffic, 06Data-Engineering, 06SRE: Add pageview information to turnilo's webrequest_sampled_live (is_pageview is always "-") - https://phabricator.wikimedia.org/T402612#11174218 (10CDanis) Thank you @JAllemandou ! This doesn't look too hard to implement in bloblang -- thankfully the definition of [[ htt... [20:35:45] 06Traffic, 10MediaWiki-Platform-Team (Radar), 07patch-welcome: Desktop view link on private wiki login required pages is broken - https://phabricator.wikimedia.org/T352798#11174405 (10Krinkle) > @Jdlrobson-WMF edited projects, added {nav Traffic}; removed {nav MobileFrontend}. > Does not seem related to Mobi... [22:26:07] FIRING: [2x] FermMSS: Unexpected MSS value on 10.2.2.38:8082 @ druid1012 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=4&var-site=eqiad&var-cluster=druid_public - https://alerts.wikimedia.org/?q=alertname%3DFermMSS