[05:14:58] <wikibugs>	 06Traffic, 10Hiddenparma, 06SRE, 13Patch-For-Review: Integrate code from the private repository into the CDN - https://phabricator.wikimedia.org/T404826#11305164 (10Joe) >>! In T404826#11304584, @bd808 wrote:  > This is a concern that should likely be discussed elsewhere, but to make it known I will state...
[06:35:24] <wikibugs>	 06Traffic, 06DC-Ops, 10ops-codfw, 06SRE, 13Patch-For-Review: Q4:rack/setup/install cp20[43-58] codfw - https://phabricator.wikimedia.org/T392851#11305224 (10elukey) The IDRAC 10 support for provision + upgrade-firmware is done, we should be good from the I/F and dcops point of view :)
[06:47:32] <wikibugs>	 06Traffic, 06MediaWiki-Platform-Team (Radar), 13Patch-For-Review: [Clean up] Redirect m-dot URLs to canonical domains - https://phabricator.wikimedia.org/T405931#11305227 (10Krinkle) Tested Varnish purge change with tcpdump in Beta.   >>! In T405931#11304897, @gerritbot wrote: > Change #1198412 had a related...
[07:03:26] <wikibugs>	 06Traffic, 06MediaWiki-Platform-Team (Radar), 13Patch-For-Review: [Clean up] Redirect m-dot URLs to canonical domains - https://phabricator.wikimedia.org/T405931#11305246 (10Krinkle) >>! In T405931#11305238, @gerritbot wrote: > Change #1198412 **merged** by jenkins-bot: > %%%[operations/mediawiki-config@mast...
[07:18:41] <_joe_>	 hey traffic folks, https://gerrit.wikimedia.org/r/c/operations/puppet/+/1198424 fixes beta breakages, and https://gerrit.wikimedia.org/r/c/operations/puppet/+/1198432 fixes varnishtests
[07:18:48] <_joe_>	 apologies again for the disruption
[07:19:05] <_joe_>	 I will merge the former with a +1 ASAP as it's a noop and it unbreaks beta
[07:19:14] <_joe_>	 fabfur: PTAL
[07:19:23] <_joe_>	 the other one is at your convenience really
[07:19:39] <fabfur>	 👀
[07:30:26] <fabfur>	 +1 for varnishtests too
[08:53:03] <stevemunene>	 Hi folks, we are looking to add a new service to LVS, thats the druid-public-coodinator which is part of the druid public cluster. T406222 We do have some patches up for review
[08:53:04] <stashbot>	 T406222: Add druid coordinator service to LVS for the druid_public cluster. - https://phabricator.wikimedia.org/T406222
[08:53:16] <wikibugs>	 06Traffic, 06MediaWiki-Platform-Team (Radar), 13Patch-For-Review: [Clean up] Redirect m-dot URLs to canonical domains - https://phabricator.wikimedia.org/T405931#11305492 (10Krinkle) Looking at the impact on Varnish, after removing half the purge load from MediaWiki.  > Change #1198412 **merged** by jenkins-...
[08:54:51] <stevemunene>	 1198500: DNS: Add druid-public-coordinator record | https://gerrit.wikimedia.org/r/c/operations/dns/+/1198500
[08:54:51] <stevemunene>	 1198498: LVS: etcd data for druid-public-coordinator | https://gerrit.wikimedia.org/r/c/operations/puppet/+/1198498
[08:54:51] <stevemunene>	 1198499: LVS: Add druid-public-coordinator to service list | https://gerrit.wikimedia.org/r/c/operations/puppet/+/1198499
[08:58:11] <fabfur>	 stevemunene: looking ASAP
[08:58:31] <fabfur>	 that should be added today ? 
[09:01:15] <_joe_>	 no.
[09:01:31] <_joe_>	 we do not deploy non-emergency changes to LVS on a friday.
[09:04:01] <stevemunene>	 no fabfur it is a non emergency and we can do it on Monday
[09:04:13] <fabfur>	 ack! 
[09:58:16] <wikibugs>	 10netops, 06Infrastructure-Foundations: Downgrade pfw1-codfw to Junos 23.4R2-S3 - https://phabricator.wikimedia.org/T393996#11305617 (10cmooney) Just on this one we hit another issue with gnmi stats for this one.  In a change made a few days ago we added the "openconfig:" prefix to the BGP stats collection, wh...
[10:46:13] <wikibugs>	 10netops, 06Infrastructure-Foundations: Downgrade pfw1-codfw to Junos 23.4R2-S3 - https://phabricator.wikimedia.org/T393996#11305689 (10cmooney) a:05cmooney→03None
[10:49:13] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Eqiad C/D refresh: 2 x test hosts for config validation - https://phabricator.wikimedia.org/T405560#11305694 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cmooney@cumin1003 for host sretest1006.eqiad.wmnet...
[11:37:22] <wikibugs>	 06Traffic, 06MW-Interfaces-Team, 06serviceops, 07Epic, and 3 others: Rename "limit group" to "limit policy" - https://phabricator.wikimedia.org/T408192 (10daniel) 03NEW
[11:38:09] <wikibugs>	 06Traffic, 06MW-Interfaces-Team, 06serviceops, 07Epic, and 3 others: Rename "limit group" to "limit policy" - https://phabricator.wikimedia.org/T408192#11305825 (10daniel)
[12:09:29] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Eqiad C/D refresh: 2 x test hosts for config validation - https://phabricator.wikimedia.org/T405560#11305892 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cmooney@cumin1003 for host sretest1006.eqiad.wmnet with...
[12:25:46] <wikibugs>	 06Traffic, 10bot-traffic-requests: FY 25/26 WE 5.4.6 Classify the top 30 spiders by traffic as known bots - https://phabricator.wikimedia.org/T408061#11305953 (10Fabfur) A not-so-refined search on Turnilo produced this paste: P84293 We can even refine it later
[12:38:56] <wikibugs>	 10Domains, 06Traffic, 10DNS, 06SRE: Request to create the donate.wikipedia25.org domain + 301 redirect to a donate.wiki page - https://phabricator.wikimedia.org/T408168#11306020 (10ssingh) a:03BCornwall
[13:44:16] <wikibugs>	 06Traffic: varnishtests are broken with podman - https://phabricator.wikimedia.org/T408202 (10Fabfur) 03NEW
[13:46:35] <wikibugs>	 06Traffic: varnishtests are broken with podman - https://phabricator.wikimedia.org/T408202#11306208 (10ssingh) a:03BCornwall
[13:49:32] <wikibugs>	 10Domains, 06Traffic, 10DNS, 06SRE: Request to create the donate.wikipedia25.org domain + 301 redirect to a donate.wiki page - https://phabricator.wikimedia.org/T408168#11306210 (10SCampos-WMF)
[13:50:02] <wikibugs>	 10Domains, 06Traffic, 10DNS, 06SRE: Request to create the donate.wikipedia25.org domain + 301 redirect to a donate.wiki page - https://phabricator.wikimedia.org/T408168#11306211 (10SCampos-WMF)
[13:50:33] <wikibugs>	 10Domains, 06Traffic, 10DNS, 06SRE: Request to create the donate.wikipedia25.org domain + 301 redirect to a donate.wiki page - https://phabricator.wikimedia.org/T408168#11306212 (10SCampos-WMF) Hi team, just updated the task with the final link to the donate.wiki portal.
[13:56:36] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Eqiad C/D refresh: 2 x test hosts for config validation - https://phabricator.wikimedia.org/T405560#11306225 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cmooney@cumin1003 for host sretest1006.eqiad.wmnet...
[14:19:43] <jinxer-wm>	 FIRING: [2x] HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages  - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages
[14:24:43] <jinxer-wm>	 FIRING: [12x] HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages  - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages
[14:30:40] <jinxer-wm>	 FIRING: [2x] VarnishHighThreadCount: Varnish's thread count on cp3067:0 is high - https://wikitech.wikimedia.org/wiki/Varnish  - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount
[14:34:43] <jinxer-wm>	 RESOLVED: [12x] HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages  - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages
[14:40:40] <jinxer-wm>	 FIRING: [2x] VarnishHighThreadCount: Varnish's thread count on cp3067:0 is high - https://wikitech.wikimedia.org/wiki/Varnish  - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount
[14:50:40] <jinxer-wm>	 RESOLVED: [2x] VarnishHighThreadCount: Varnish's thread count on cp3067:0 is high - https://wikitech.wikimedia.org/wiki/Varnish  - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount
[15:19:09] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Eqiad C/D refresh: 2 x test hosts for config validation - https://phabricator.wikimedia.org/T405560#11306704 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cmooney@cumin1003 for host sretest1006.eqiad.wmnet with...
[16:20:24] <wikibugs>	 10Domains, 06Traffic, 10DNS, 06SRE, 13Patch-For-Review: Request to create the donate.wikipedia25.org domain + 301 redirect to a donate.wiki page - https://phabricator.wikimedia.org/T408168#11307174 (10BCornwall) 05Open→03Resolved I've created both donate.wikipedia25.org and donate.wikipedia25.com...