[09:11:19] 10Traffic: Improve runbooks for OCSP-related alerts - https://phabricator.wikimedia.org/T292397 (10elukey) [09:13:15] 10Traffic: Improve runbooks for OCSP-related alerts - https://phabricator.wikimedia.org/T292397 (10Vgutierrez) p:05Triage→03Medium [10:12:17] elukey: thanks, I think we're looking at a ATS memory leak: https://grafana.wikimedia.org/d/wiU3SdEWk/cache-host-drilldown?viewPanel=81&orgId=1&from=1633029225978&to=1633275252884&var-site=eqsin%20prometheus%2Fops&var-instance=cp5006 [10:16:45] wow [10:17:41] meeting pad for tomorrow is up, feel free to add your topics etc [11:59:57] 10netops, 10Infrastructure-Foundations, 10SRE: Netbox info missing on some WMCS elements - https://phabricator.wikimedia.org/T292097 (10ayounsi) Documenting all the cables make sens, feel free to add the one between the cloudstore hosts (or ask DCops) About the IPs, we decided to not track any of the 192.16... [12:01:59] 10Traffic, 10SRE, 10Patch-For-Review: Deploy durum: check service for Wikidough - https://phabricator.wikimedia.org/T289536 (10ayounsi) Note that a few of the durum IPs have both the "DNS name" field set, and "Keep manual DNS" as comment, which I think are mutually exclusive (but not enforced). https://netbo... [13:01:35] 10netops, 10Infrastructure-Foundations, 10SRE: Netbox info missing on some WMCS elements - https://phabricator.wikimedia.org/T292097 (10cmooney) 05Open→03Resolved I've recreated the IP, and put the DNS name in the description with "Keep manual DNS" prefix. It doesn't make much difference, as the Netbox... [13:15:07] 10Traffic, 10SRE, 10Patch-For-Review: Deploy durum: check service for Wikidough - https://phabricator.wikimedia.org/T289536 (10Volans) >>! In T289536#7398142, @ayounsi wrote: > Note that a few of the durum IPs have both the "DNS name" field set, and "Keep manual DNS" as comment, which I think are mutually ex... [15:06:37] 10Traffic, 10DC-Ops, 10SRE, 10ops-ulsfo: Q1:(Need By: TBD) rack/setup/install cp403[3-6].ulsfo.wmnet - https://phabricator.wikimedia.org/T290694 (10RobH) [15:59:01] Traffic folks, reminder to update the SRE doc :) [15:59:11] *meeting doc, but yeah [18:50:57] (VarnishTrafficDrop) firing: 60% GET drop in text@eqsin during the past 30 minutes - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org [18:55:57] (VarnishTrafficDrop) resolved: 66% GET drop in text@eqsin during the past 30 minutes - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org [19:12:57] (VarnishTrafficDrop) firing: 69% GET drop in text@esams during the past 30 minutes - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org [19:17:56] (VarnishTrafficDrop) resolved: 69% GET drop in text@esams during the past 30 minutes - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org