[00:59:44] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw row C/D upgrade racking task - https://phabricator.wikimedia.org/T360789#9844770 (10Papaul)
[02:47:26] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw row C/D upgrade racking task - https://phabricator.wikimedia.org/T360789#9844864 (10Papaul)
[03:44:37] <wikibugs>	 10Wikimedia-Apache-configuration, 06SRE, 06Traffic-Icebox, 10Wiki-Setup, 13Patch-Needs-Improvement: redirect sco.wiktionary.org/wiki/(.*?) -> sco.wikipedia.org/wiki/Define:$1 - https://phabricator.wikimedia.org/T249648#9844895 (10Pppery)
[03:44:43] <wikibugs>	 10Wikimedia-Apache-configuration, 06SRE, 06Traffic-Icebox, 13Patch-Needs-Improvement, 10Wiki-Setup (Delete / Redirect): redirect sco.wiktionary.org/wiki/(.*?) -> sco.wikipedia.org/wiki/Define:$1 - https://phabricator.wikimedia.org/T249648#9844896 (10Pppery)
[03:47:34] <wikibugs>	 10Wikimedia-Apache-configuration, 06SRE, 06Traffic-Icebox, 13Patch-Needs-Improvement, 10Wiki-Setup (Delete / Redirect): redirect sco.wiktionary.org/wiki/(.*?) -> sco.wikipedia.org/wiki/Define:$1 - https://phabricator.wikimedia.org/T249648#9844898 (10Pppery)
[08:54:48] <wikibugs>	 06Traffic, 10Data-Platform-SRE (2024.05.27 - 2024.06.16): Point datahub and datahub-next subdomains to traffic server - https://phabricator.wikimedia.org/T365668#9845420 (10Stevemunene) 05Open→03Resolved
[10:36:07] <wikibugs>	 06Traffic, 06SRE: Anycast ns1.wikimedia.org - https://phabricator.wikimedia.org/T366193#9845869 (10ayounsi) >  assign a /24 from https://netbox.wikimedia.org/ipam/aggregates/ to be used for this As we couldn't get a /24 from LACNIC for magru, we only have two free /24s We have to decide between a few options:...
[11:38:48] <wikibugs>	 06Traffic, 06SRE: Anycast ns1.wikimedia.org - https://phabricator.wikimedia.org/T366193#9846069 (10cmooney) >>! In T366193#9845869, @ayounsi wrote: >>  assign a /24 from https://netbox.wikimedia.org/ipam/aggregates/ to be used for this > As we couldn't get a /24 from LACNIC for magru, we only have two free /24...
[13:11:46] <wikibugs>	 06Traffic, 06SRE: Anycast ns1.wikimedia.org - https://phabricator.wikimedia.org/T366193#9846383 (10BBlack) Re: anycast-ns1 and future plans, etc (I won't quote all the relevant bits from both msgs above):  * Currently we still strongly prefer to avoid mixing any other service with the DoH anycast, due to the d...
[15:00:26] <wikibugs>	 06Traffic, 06SRE: Anycast ns1.wikimedia.org - https://phabricator.wikimedia.org/T366193#9846964 (10ssingh) >>! In T366193#9845869, @ayounsi wrote: >>  assign a /24 from https://netbox.wikimedia.org/ipam/aggregates/ to be used for this  Thanks for the detailed breakdown of the options!  > As we couldn't get a /...
[16:21:26] <wikibugs>	 06Traffic, 06SRE: Anycast ns1.wikimedia.org - https://phabricator.wikimedia.org/T366193#9847478 (10cmooney) > At least pdns-recursor seems to do this:  Anecdotally Bind seems to do the same, in a test this morning my local server went to ns2 626 times when I queried a bunch of wikis, and ns1 and ns0 5 times ea...
[16:23:43] <jinxer-wm>	 FIRING: SystemdUnitFailed: ncmonitor.service on ncmonitor1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[16:41:47] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: magru network setup - https://phabricator.wikimedia.org/T362421#9847581 (10cmooney) >>! In T362421#9808055, @cmooney wrote: > Cogent are picking the magru announcement as best globally from Novvacore it seems also.  We could add `28189:8094...
[17:10:30] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: magru network setup - https://phabricator.wikimedia.org/T362421#9847731 (10cmooney) Ok after change Cogent are going to Telia, which we have at all the other POPs, so I think a better result.  Novvacore are still announcing it at IX.br so d...
[18:09:55] <wikibugs>	 06Traffic, 06MW-Interfaces-Team, 06serviceops: map the /api/ prefix to /w/rest.php - https://phabricator.wikimedia.org/T364400#9848020 (10daniel)
[20:02:03] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw row C/D upgrade racking task - https://phabricator.wikimedia.org/T360789#9848597 (10Papaul)
[20:02:36] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw row C/D upgrade racking task - https://phabricator.wikimedia.org/T360789#9848600 (10Papaul)
[21:56:32] <wikibugs>	 06Traffic, 06Data-Engineering, 10Observability-Logging, 13Patch-For-Review: Switch HAProxy/Benthos to rfc5424 - https://phabricator.wikimedia.org/T365718#9848879 (10Fabfur) This has been reverted due to a bug in HAProxy regarding how log variables are escaped in the rfc5424 format using the `+E` flag.  See...