[06:16:48] hello folks [06:17:10] I was reviewing the SLO dashboards, the ha-proxy eqsin one seems to have exhausted its SLO: [06:17:13] https://grafana.wikimedia.org/d/slo-HAProxy/haproxy-slo-s?orgId=1&var-datasource=thanos&var-site=eqsin&var-cluster=All [06:17:27] *SLO Budget sorry [06:17:46] (just reporting it, in case you want to adjust etc..) [06:18:08] anyway, the dashboards also have the fixed time window for a quarter shifted earlier by a month [06:18:16] if you hate them lemme know [06:18:35] (the idea is that we'll change the time window every quarter with a single commit for all the dashboards) [07:18:14] (VarnishHighThreadCount) firing: (8) Varnish's thread count is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [07:23:14] (VarnishHighThreadCount) firing: (8) Varnish's thread count is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [07:28:14] (VarnishHighThreadCount) firing: (16) Varnish's thread count is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [07:28:29] (VarnishHighThreadCount) firing: (15) Varnish's thread count is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [07:31:03] re: ores traffic drain, I filed https://gerrit.wikimedia.org/r/c/operations/puppet/+/959762 - We are planning to execute this on Monday, lemme know if you are ok! :) [07:33:14] (VarnishHighThreadCount) firing: (16) Varnish's thread count is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [07:48:14] (VarnishHighThreadCount) resolved: (8) Varnish's thread count is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [08:44:19] vgutierrez: <3 [08:55:46] :) looking good, TLS gods are pleased [10:51:24] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:rack/setup/install cp1[098-113] - https://phabricator.wikimedia.org/T342159 (10VRiley-WMF) Hey @ssingh If possible, please let us know the racking configuration for these devices. Currently, all the NVMe SSD have been installed into these servers and awaiting... [12:47:58] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:rack/setup/install cp1[098-113] - https://phabricator.wikimedia.org/T342159 (10BBlack) [12:48:43] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:rack/setup/install cp1[098-113] - https://phabricator.wikimedia.org/T342159 (10BBlack) @VRiley-WMF - Sukhbir's out right now, but I've updated the racking plan on his behalf! [13:00:27] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:rack/setup/install cp1[098-113] - https://phabricator.wikimedia.org/T342159 (10BBlack) Reading a little deeper on this, I think we still have a hostnames issue. If those other 8 hosts are indeed being brought from ulsfo+eqsin. Those 8 hosts, I presume, would be... [13:03:31] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:rack/setup/install cp1[098-113] - https://phabricator.wikimedia.org/T342159 (10BBlack) Adding to the confusion: historically, we once used the hostname `cp1099` back in 2015 for a one-off host: T96873 - therefore that name already exists in both phab and git hist... [13:31:15] 10netops, 10Infrastructure-Foundations, 10SRE, 10cloud-services-team: Configure cloudsw1-b1-codfw and migrate cloud hosts in codfw B1 to it - https://phabricator.wikimedia.org/T327919 (10cmooney) 05In progress→03Resolved [15:42:00] 10Traffic, 10SRE: Allow Varnish to be called on UDS from services other than HAProxy - https://phabricator.wikimedia.org/T347059 (10Fabfur) 05Open→03Declined [15:48:45] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Bring codfw row A-B EVPN switches live and make them gateway for existing Vlans - https://phabricator.wikimedia.org/T347191 (10cmooney) p:05Triage→03Medium [15:54:11] 10Traffic: Varnish should allow PURGE requests only from socket (purged) - https://phabricator.wikimedia.org/T347192 (10Fabfur) [15:56:56] 10Traffic: Varnish should allow PURGE requests only from socket (purged) - https://phabricator.wikimedia.org/T347192 (10Fabfur) [16:58:36] 10Traffic, 10SRE: Varnish should allow PURGE requests only from socket (purged) - https://phabricator.wikimedia.org/T347192 (10Fabfur) [16:58:39] 10Traffic, 10SRE: Varnish should allow PURGE requests only from socket (purged) - https://phabricator.wikimedia.org/T347192 (10Fabfur)