[06:34:12] 10netops, 06Infrastructure-Foundations, 10Observability-Alerting, 13Patch-For-Review: Migrate network icinga alerts to gNMI/prometheus - https://phabricator.wikimedia.org/T388641#10697203 (10ayounsi) BFD is deployed, here is the full list of devices not able to expose those metrics (devices that don't have... [09:03:41] 06Traffic, 10Maps, 06SRE: Allow Wikimedia Maps usage on - https://phabricator.wikimedia.org/T390669 (10Odeline_Marteau1) 03NEW [09:04:48] 06Traffic, 10Maps, 06SRE: Allow Wikimedia Maps usage on - https://phabricator.wikimedia.org/T390669#10697535 (10Odeline_Marteau1) [09:17:06] we'd like to do a test (and then full) rollout of https://gerrit.wikimedia.org/r/c/operations/puppet/+/1131748 in a little bit - does that conflict with anything? [09:18:08] 06Traffic, 10Maps, 06SRE: Allow Wikimedia Maps usage on - https://phabricator.wikimedia.org/T390669#10697577 (10Aklapper) [09:19:16] 06Traffic, 10Maps, 06SRE: Allow Wikimedia Maps usage on  - https://phabricator.wikimedia.org/T390669#10697579 (10Aklapper) 05Open→03Declined > Link to site: QGIS That is not a link to a site. > Wikimedia Affiliate supporting project: Odeline-Marteau1 That is not a Wikimedia Affiliate. See... [09:19:46] 06Traffic, 10Maps, 06SRE: Allow Wikimedia Maps usage on  - https://phabricator.wikimedia.org/T390669#10697582 (10Aklapper) a:05Odeline_Marteau1→03None [09:22:36] hnowlan: nope AFAIK [09:23:39] cool, thanks! I will toggle puppet on A:cp in that case [10:58:21] 06Traffic, 10Citoid, 06Editing-team, 10RESTBase Sunsetting, and 3 others: Switch from restbase to api gateway for Citoid - https://phabricator.wikimedia.org/T361576#10697957 (10Mvolz) [10:58:50] 06Traffic, 10Citoid, 06Editing-team, 10RESTBase Sunsetting, and 3 others: Switch from restbase to api gateway for Citoid - https://phabricator.wikimedia.org/T361576#10697960 (10Mvolz) [11:01:04] just finalising the rollout of that change now - some bumps along the way but I think we're looking good [11:51:39] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade core routers to Junos 23.4R2 - https://phabricator.wikimedia.org/T364092#10698172 (10cmooney) [13:18:30] 10netops, 06Infrastructure-Foundations: Enable gNMI on SRX devices and fasw - https://phabricator.wikimedia.org/T390052#10698935 (10ayounsi) Some updates on that front ! **Fundraising switches (fasw)** All good. **Management switches (msw)** After configuration, seems like only msw2-codfw have gNMI listeni... [13:55:02] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, 06SRE: Link down between cr3-ulsfo and cr4-ulsfo - https://phabricator.wikimedia.org/T390731 (10cmooney) 03NEW p:05Triage→03High [13:56:51] FIRING: FermMSS: Unexpected MSS value on 10.2.1.44:443 @ registry2005 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=4&var-site=codfw&var-cluster=misc - https://alerts.wikimedia.org/?q=alertname%3DFermMSS [13:57:48] ^^ elukey is working on that host :) [13:58:25] yeah. I love the "@ host" in the alert itself [14:01:51] RESOLVED: FermMSS: Unexpected MSS value on 10.2.1.44:443 @ registry2005 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=4&var-site=codfw&var-cluster=misc - https://alerts.wikimedia.org/?q=alertname%3DFermMSS [14:53:53] 10netops, 06Infrastructure-Foundations, 06SRE, 10Data-Platform-SRE (2025.03.22 - 2025.04.11): Add QoS markings to profile Hadoop/HDFS analytics traffic - https://phabricator.wikimedia.org/T381389#10699524 (10xcollazo) Ok attempting the below query again now: >>! In T390623#10699223, @xcollazo wrote: >... [15:15:09] 10netops, 06Infrastructure-Foundations, 06SRE, 10Data-Platform-SRE (2025.03.22 - 2025.04.11): Add QoS markings to profile Hadoop/HDFS analytics traffic - https://phabricator.wikimedia.org/T381389#10699685 (10xcollazo) I've succesfully run the following query: >>! In T390623#10699616, @xcollazo wrote: >... [15:16:07] 10netops, 06Infrastructure-Foundations, 06SRE, 10Data-Platform-SRE (2025.03.22 - 2025.04.11): Add QoS markings to profile Hadoop/HDFS analytics traffic - https://phabricator.wikimedia.org/T381389#10699693 (10xcollazo) >We only have these stats for some of the presto hosts, which are those in rows E and... [15:43:54] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, 06SRE: Link down between cr3-ulsfo and cr4-ulsfo - https://phabricator.wikimedia.org/T390731#10699869 (10RobH) Case 01043199 > Support, > > We recently rolled some OS upgrades to our routers and during that, one of the optics on our cross... [15:44:02] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, 06SRE: Link down between cr3-ulsfo and cr4-ulsfo - https://phabricator.wikimedia.org/T390731#10699870 (10RobH) a:05cmooney→03RobH [16:05:14] 10netops, 06Infrastructure-Foundations, 06SRE, 10Data-Platform-SRE (2025.03.22 - 2025.04.11): Add QoS markings to profile Hadoop/HDFS analytics traffic - https://phabricator.wikimedia.org/T381389#10699995 (10cmooney) > This effectively moved 308GB from HDFS Datanodes, thru the routers, to Presto server... [16:22:22] 10netops, 06Infrastructure-Foundations, 06SRE, 10Data-Platform-SRE (2025.03.22 - 2025.04.11): Add QoS markings to profile Hadoop/HDFS analytics traffic - https://phabricator.wikimedia.org/T381389#10700068 (10cmooney) FWIW the largest potential bottleneck in Ashburn are on the 10G interfaces (names star... [17:27:57] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, 06SRE: Link down between cr3-ulsfo and cr4-ulsfo - https://phabricator.wikimedia.org/T390731#10700358 (10cmooney) p:05High→03Low Looks like remote hands replaced the module. ` cmooney@cr4-ulsfo> show log messages | match qsfp Apr 1 17:1... [17:35:08] 10netops, 06Infrastructure-Foundations, 06SRE, 10Data-Platform-SRE (2025.03.22 - 2025.04.11): Add QoS markings to profile Hadoop/HDFS analytics traffic - https://phabricator.wikimedia.org/T381389#10700403 (10xcollazo) Thanks for the pointers @cmooney. --------- Here are my heavy query results: First... [17:42:31] 10netops, 06Infrastructure-Foundations, 06SRE, 10Data-Platform-SRE (2025.03.22 - 2025.04.11): Add QoS markings to profile Hadoop/HDFS analytics traffic - https://phabricator.wikimedia.org/T381389#10700438 (10cmooney) > No one is yelling on IRC so I think I am happy with this. I am done from my side. O... [17:53:08] 10netops, 06Infrastructure-Foundations, 06SRE, 10Data-Platform-SRE (2025.03.22 - 2025.04.11): Add QoS markings to profile Hadoop/HDFS analytics traffic - https://phabricator.wikimedia.org/T381389#10700483 (10cmooney) Also to get a sense of total throughput this graph is good: https://grafana.wikimedia... [17:58:03] 10netops, 06Infrastructure-Foundations, 06SRE, 10Data-Platform-SRE (2025.03.22 - 2025.04.11): Add QoS markings to profile Hadoop/HDFS analytics traffic - https://phabricator.wikimedia.org/T381389#10700491 (10xcollazo) >>! In T381389#10700438, @cmooney wrote: >> No one is yelling on IRC so I think I am... [18:13:40] 06Traffic, 07Browser-Support-Apple-Safari, 07Browser-Support-Firefox, 07Browser-Support-Google-Chrome, 07User-notice: Discovery: Deprecation of TLS 1.2 - https://phabricator.wikimedia.org/T367821#10700550 (10gh87) Shall this task be stalled then? Many computers still use Windows 10, which still lacks TLS...