[06:17:56] <jinxer-wm>	 (HAProxyEdgeTrafficDrop) firing: (3) 63% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop  - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop
[06:18:16] <jinxer-wm>	 (VarnishTrafficDrop) firing: Varnish traffic in eqiad has dropped 44.759334221144066% - https://wikitech.wikimedia.org/wiki/Varnish - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DVarnishTrafficDrop
[06:22:56] <jinxer-wm>	 (HAProxyEdgeTrafficDrop) resolved: (3) 61% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop  - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop
[06:23:16] <jinxer-wm>	 (VarnishTrafficDrop) resolved: (2) Varnish traffic in eqiad has dropped 43.85650321595217% - https://wikitech.wikimedia.org/wiki/Varnish - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DVarnishTrafficDrop
[06:42:53] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10Observability-Metrics, 10SRE, and 2 others: LibreNMS seemingly not collecting data for many ports after migration to netmon1003 - https://phabricator.wikimedia.org/T314972 (10andrea.denisse)
[09:32:22] <wikibugs>	 10Traffic, 10SRE, 10Patch-For-Review: Don't set cookies for api.wikimedia.org at the caching layer - https://phabricator.wikimedia.org/T260943 (10Vgutierrez) https://gerrit.wikimedia.org/r/824793 submitted by @BCornwall removes `WMF-Last-Access` cookie from api.wikimedia.org, as he mentioned this also remove...
[11:31:56] <jinxer-wm>	 (HAProxyEdgeTrafficDrop) firing: 40% request drop in text@ulsfo during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=ulsfo&var-cache_type=text - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop
[11:36:56] <jinxer-wm>	 (HAProxyEdgeTrafficDrop) resolved: (5) 64% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop  - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop
[13:43:51] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE, 10netbox, 10Patch-For-Review: Netbox: use  FHRP Groups feature - https://phabricator.wikimedia.org/T311218 (10ayounsi) The two patches above should allow us to use the `FHRP group` feature in production, without leveraging additional fields like priority or...
[15:33:04] <wikibugs>	 10netops, 10Cloud-Services, 10Infrastructure-Foundations, 10SRE: Undocumented IP on WMCS network - https://phabricator.wikimedia.org/T315955 (10Andrew) a:05Andrew→03cmooney This additional range was set up by @cmooney -- Cathal, is this something you can document as needed?
[17:03:59] <wikibugs>	 10Traffic, 10SRE, 10Patch-For-Review: ATS should alert if the number of total or active connections reached maximum - https://phabricator.wikimedia.org/T292815 (10BCornwall) Change of plans: Kwaku has expressed an interest in backwards-compatibility so ATS 8 support will be added.
[19:45:59] <wikibugs>	 10netops, 10Cloud-Services, 10Infrastructure-Foundations, 10SRE: Undocumented IP on WMCS network - https://phabricator.wikimedia.org/T315955 (10cmooney) a:05cmooney→03Andrew @Andrew I indeed routed the subnet, which was already allocated to WMCS in codfw.  It seems I failed to update the description fo...
[20:17:09] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE, 10netbox, 10Patch-For-Review: Netbox: use  FHRP Groups feature - https://phabricator.wikimedia.org/T311218 (10cmooney) Nice work!  Eventually all things considered it's probably best to control it from Netbox.  But I agree the existing mechanism works well i...
[20:59:56] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE, 10netbox, 10Patch-For-Review: Netbox: use FHRP Groups feature - https://phabricator.wikimedia.org/T311218 (10Reedy)
[21:02:24] <wikibugs>	 10netops, 10Cloud-Services, 10Infrastructure-Foundations, 10SRE: Undocumented IP on WMCS network - https://phabricator.wikimedia.org/T315955 (10Andrew) The label should just be 'public floating IPs for cloud-vps codfw1dev' -- by their very nature the actual use of any particular IP will shift over time bas...
[22:22:37] <wikibugs>	 10netops, 10Cloud-Services, 10Infrastructure-Foundations, 10SRE: Undocumented IP on WMCS network - https://phabricator.wikimedia.org/T315955 (10cmooney) Thanks Andrew, I've updated the description for the codfw range now.  In terms of DNS I don't seem to get any PTR records back for the ranges in codfw:  `...