[06:47:56] <jinxer-wm>	 (EdgeTrafficDrop) firing: 68% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=codfw&var-cache_type=text - https://alerts.wikimedia.org
[06:57:56] <jinxer-wm>	 (EdgeTrafficDrop) resolved: 69% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=codfw&var-cache_type=text - https://alerts.wikimedia.org
[07:00:56] <jinxer-wm>	 (EdgeTrafficDrop) firing: 67% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=codfw&var-cache_type=text - https://alerts.wikimedia.org
[07:05:56] <jinxer-wm>	 (EdgeTrafficDrop) resolved: 68% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=codfw&var-cache_type=text - https://alerts.wikimedia.org
[08:19:41] <volans|off>	 bblack: [for the higher level note don't worry with the above patch we don't mangle dns names anymore from the puppetdb import script run during reimage. That was legacy code needed during the transition phase. Netbox is the source of truth ;) ]
[08:26:28] <volans|off>	 as for the IPs above those unused should be deleted from Netbox, those used ahould have either a DNS name set on Netbox or a comment for Keep manual DNS.
[08:52:56] <jinxer-wm>	 (EdgeTrafficDrop) firing: 69% request drop in text@ulsfo during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=ulsfo&var-cache_type=text - https://alerts.wikimedia.org
[08:57:56] <jinxer-wm>	 (EdgeTrafficDrop) resolved: 69% request drop in text@ulsfo during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=ulsfo&var-cache_type=text - https://alerts.wikimedia.org
[19:00:50] <wikibugs>	 10Traffic: Cookie value sent in HTTP requests changes too frequently - https://phabricator.wikimedia.org/T295619 (10Snaevar)
[19:04:27] <wikibugs>	 10Traffic: Image requests sending neither "Last-Modified" nor "ETag" HTTP headers. - https://phabricator.wikimedia.org/T295556 (10Snaevar)
[23:56:58] <legoktm>	 users are reporting "Error: 502, Next Hop Connection Failed" when trying to upload files using Special:Upload (directly POSTs the entire <100MB file in one go, not chunked)
[23:57:00] <legoktm>	 https://phabricator.wikimedia.org/T295343
[23:57:18] <legoktm>	 https://phabricator.wikimedia.org/T247454 is an older but similar report too
[23:58:43] <legoktm>	 It's unclear to me whether it's an issue at the caching layer or something underneath but I thought I'd start here