[07:47:30] 10Traffic, 10SRE, 10Patch-For-Review, 10SRE Observability (FY2021/2022-Q1): Use Grizzly for Varnish SLO Grafana dashboard - https://phabricator.wikimedia.org/T289036 (10ema) @herron: I've merged the patch, forced a puppet run on grafana1002.eqiad.wmnet, and followed the instructions at https://wikitech.wik... [08:01:12] 10Traffic, 10DNS, 10SRE: More DNS entries for WikiLearn servers - https://phabricator.wikimedia.org/T290025 (10fgiunchedi) p:05Triage→03Medium [08:43:56] (VarnishTrafficDrop) firing: 68% GET drop in text@ during the past 30 minutes - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org [08:53:56] (VarnishTrafficDrop) resolved: 67% GET drop in text@ during the past 30 minutes - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org [09:33:36] 10Traffic, 10DNS, 10SRE: More DNS entries for WikiLearn servers - https://phabricator.wikimedia.org/T290025 (10Vgutierrez) I guess you also need a proper CAA record to authorize AWS CA to issue certs for learn.wiki [09:46:29] ema, mmandere could I get a sanity check here please? https://gerrit.wikimedia.org/r/c/operations/dns/+/715706 [09:50:03] 10Traffic, 10DNS, 10SRE, 10Patch-For-Review: More DNS entries for WikiLearn servers - https://phabricator.wikimedia.org/T290025 (10Vgutierrez) From https://docs.aws.amazon.com/acm/latest/userguide/setup-caa.html it looks like any of amazon.com, amazontrust.com, awstrust.com or amazonaws.com would do it as... [10:10:14] vgutierrez: looking [10:10:19] thx <3 [10:16:51] 10Traffic, 10DNS, 10SRE, 10Patch-For-Review: More DNS entries for WikiLearn servers - https://phabricator.wikimedia.org/T290025 (10Vgutierrez) 05Open→03Resolved a:03Vgutierrez ` vgutierrez@carrot:~$ host -t CAA learn.wiki learn.wiki has CAA record 0 issue "letsencrypt.org" learn.wiki has CAA record 0... [11:06:46] 10netops, 10Infrastructure-Foundations, 10SRE: 2021-08-26 Primary inbound port utilisation over 80% page for mr1-esams.wikimedia.org - https://phabricator.wikimedia.org/T289820 (10ayounsi) I removed the management routers from the wrong alert, that's why we got paged again. It's now fixed so it won't page wh... [12:09:35] traffic pad had been created by someone already, thanks :) I've updated it with more topics, please add any others as desired [14:15:04] heh I didn't realize the 6 week OKR review is managers only [14:16:54] sudo join_meeting [14:16:58] O:) [14:20:30] lol [14:22:37] 10netops, 10Analytics, 10DBA, 10Infrastructure-Foundations, and 3 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10Ottomata) I just tried to run puppet on an-coord1001 but got: ` Notice: Skipping run of Puppet configuration client; administratively disabled (... [14:23:34] 10netops, 10Analytics, 10DBA, 10Infrastructure-Foundations, and 3 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10RhinosF1) Puppet is under maintenance [14:23:48] 10netops, 10Analytics, 10DBA, 10Infrastructure-Foundations, and 3 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10Ottomata) Oh, sorry @jbond is doing some maintenance and referenced the wrong phab ticket. Ignore ^ [14:56:29] 10Traffic, 10SRE, 10SRE Observability (FY2021/2022-Q1): Use Grizzly for Varnish SLO Grafana dashboard - https://phabricator.wikimedia.org/T289036 (10herron) Thanks @ema! This is helpful feedback >>! In T289036#7320951, @ema wrote: > The diff step, `grr diff dashboardname`, is unclear to me. What is dashboa... [16:30:57] (VarnishTrafficDrop) firing: 69% GET drop in text@ during the past 30 minutes - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org [16:35:57] (VarnishTrafficDrop) resolved: 68% GET drop in text@ during the past 30 minutes - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org