[01:57:56] (EdgeTrafficDrop) firing: (2) 64% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://alerts.wikimedia.org [02:02:56] (EdgeTrafficDrop) resolved: (2) 66% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://alerts.wikimedia.org [03:37:56] (EdgeTrafficDrop) firing: 68% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=codfw&var-cache_type=text - https://alerts.wikimedia.org [03:42:56] (EdgeTrafficDrop) resolved: 68% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=codfw&var-cache_type=text - https://alerts.wikimedia.org [03:49:26] (EdgeTrafficDrop) firing: 65% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=codfw&var-cache_type=text - https://alerts.wikimedia.org [03:54:26] (EdgeTrafficDrop) resolved: 69% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=codfw&var-cache_type=text - https://alerts.wikimedia.org [06:45:56] (EdgeTrafficDrop) firing: 69% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=codfw&var-cache_type=text - https://alerts.wikimedia.org [06:55:56] (EdgeTrafficDrop) resolved: 69% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=codfw&var-cache_type=text - https://alerts.wikimedia.org [06:56:11] (EdgeTrafficDrop) firing: 69% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=codfw&var-cache_type=text - https://alerts.wikimedia.org [07:01:26] (EdgeTrafficDrop) resolved: 69% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=codfw&var-cache_type=text - https://alerts.wikimedia.org [11:13:56] (EdgeTrafficDrop) firing: 47% request drop in text@ulsfo during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=ulsfo&var-cache_type=text - https://alerts.wikimedia.org [11:28:56] (EdgeTrafficDrop) resolved: 66% request drop in text@ulsfo during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=ulsfo&var-cache_type=text - https://alerts.wikimedia.org [13:26:42] 10Traffic, 10SRE, 10decommission-hardware, 10ops-ulsfo, 10Patch-For-Review: decommission cp4031 - https://phabricator.wikimedia.org/T301269 (10MMandere) a:05RobH→03MMandere [13:42:56] (VarnishPrometheusExporterDown) firing: Varnish Exporter on instance cp4031:9331 is unreachable - https://alerts.wikimedia.org [13:43:31] 10Traffic, 10SRE, 10decommission-hardware, 10ops-ulsfo, 10Patch-For-Review: decommission cp4031 - https://phabricator.wikimedia.org/T301269 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by mmandere@cumin1001 for hosts: `cp4031.ulsfo.wmnet` - cp4031.ulsfo.wmnet (**PASS**) - Downtimed... [13:47:56] (VarnishPrometheusExporterDown) resolved: Varnish Exporter on instance cp4031:9331 is unreachable - https://alerts.wikimedia.org [13:51:42] ^ expected as we are decommissioning the instance [13:53:03] 10Traffic, 10SRE, 10decommission-hardware, 10ops-ulsfo, 10Patch-For-Review: decommission cp4031 - https://phabricator.wikimedia.org/T301269 (10MMandere) a:05MMandere→03RobH [16:14:00] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, 10ops-drmrs: Q3:(Need By: ASAP) rack/setup/install cr[12]-drmrs - https://phabricator.wikimedia.org/T300277 (10RobH) They closed the old ticket due to delays in the router's shipment arrival, so opened a new one today as the routers are now at drmr... [17:09:34] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, 10ops-drmrs: Q3:(Need By: ASAP) rack/setup/install cr[12]-drmrs - https://phabricator.wikimedia.org/T300277 (10RobH) Arzhel reminded me in our sync up meeting about the too short cable for mr1, so we've appended this to the ticket: Support, > I... [17:50:19] what is this file about: modules/trafficserver/files/hostdb_ttls.stp ? If there is an IP in there that belongs to a host I am decom'ing, do I have to care? [17:50:36] example: max age for 10.64.32.178=730, ttl=3600 and I shutdown 1064.32.178 [17:50:53] decom cookbook will warn about it [17:51:51] commit a40290dc4dc7c8204bf8c1f81dd9c4db9a2c4916 explains it a bit [17:52:21] SystemTap script keeping track of the maximum age and TTL of ATS hostdb entries on a per-IP basis... [17:53:38] yea.. should I make a patch to replace/remove that IP or ignore it [17:55:03] oh, my IP is only an "example output". well then.. ignoring :)