[07:48:20] 06serviceops, 06SRE: Memcached, mcrouter in MediaWiki on Kubernetes - https://phabricator.wikimedia.org/T277711#10017109 (10jijiki) 05Open→03Resolved Closing since T346690 is done, daemonset is working [07:54:22] 06serviceops, 10Citoid, 10VisualEditor, 10VisualEditor-MediaWiki-References, and 2 others: Register Citoid as a "friendly bot" (or alternatively verified bot) with Cloudflare - https://phabricator.wikimedia.org/T370118#10017125 (10akosiaris) >>! In T370118#10016683, @Novem_Linguae wrote: > There's only 18... [08:08:24] 06serviceops, 07Wikimedia-production-error: Misbehaving mw-api-ext pods serving 5xx - https://phabricator.wikimedia.org/T370425#10017159 (10jijiki) p:05High→03Low Lowering priority as for the time being the impact is minor. Will raise again if we observe more occurrences [08:22:55] 06serviceops, 10MW-on-K8s, 10wikitech.wikimedia.org: Migrate Wikitech to Kubernetes - https://phabricator.wikimedia.org/T292707#10017167 (10jijiki) [08:34:26] 06serviceops, 10Charts, 10Shellbox, 06SRE: Figure out how a shellbox instance for the Chart extension would work - https://phabricator.wikimedia.org/T370739#10017209 (10akosiaris) The situation is indeed known, see also T309772, T357950. Some efforts did happen to modernize the codebase, however, as far as... [08:34:39] 06serviceops, 10MW-on-K8s, 10wikitech.wikimedia.org: Migrate Wikitech to Kubernetes - https://phabricator.wikimedia.org/T292707#10017214 (10jijiki) [08:34:49] 06serviceops, 10MW-on-K8s, 10wikitech.wikimedia.org: Migrate Wikitech to Kubernetes - https://phabricator.wikimedia.org/T292707#10017219 (10jijiki) [09:47:04] I have send a patch for docker-pkg to pin the linters black and flake8, that is to avoid having random failures whenever upstreams release new version or one local cache has a different version than what CI has its in cache [09:47:04] https://gerrit.wikimedia.org/r/c/operations/docker-images/docker-pkg/+/1043748 [09:47:14] should be straightforward :] [09:51:08] ;) thanks [10:30:59] 06serviceops, 10MW-on-K8s, 10Scap, 13Patch-For-Review: Evaluate the performance improvements brought in by prefetching MW images on WikiKube hosts - https://phabricator.wikimedia.org/T366778#10017533 (10akosiaris) No significant changes in the last month per {F56700235}, proceeding with the drop to 66% [11:13:56] 06serviceops, 13Patch-For-Review: deploy1003 implementation tracking - https://phabricator.wikimedia.org/T364417#10017593 (10akosiaris) I 've also performed a NOOP deployment from deploy1003 today, worked slowly (20minutes) due to having to build the images, but otherwise OK) [11:49:35] 06serviceops, 06SRE: confd setup left without configuration doesn't stop confd - https://phabricator.wikimedia.org/T356296#10017678 (10cmooney) I'm a little confused about this one. We have //defs_from_etcd_nft// set to false by default in heria for the firewall profile: ` hieradata/common/profile/firewall.ya... [14:26:47] 06serviceops, 10Wikimedia-Apache-configuration: Change redirect target of sep11.wikipedia.org - https://phabricator.wikimedia.org/T367014#10017985 (10Dzahn) Thanks to @akosiaris for deploying this. I expect this to be resolved except we are waiting for caches to expire. Maybe 24 hours and it should redirect t... [16:53:06] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE: Q1:rack/setup/install wikikube-worker1240 to wikikube-worker1304 - https://phabricator.wikimedia.org/T369743#10018850 (10VRiley-WMF) [17:40:26] 06serviceops, 10MoveComms-Support, 07Datacenter-Switchover: MoveComms support for Southward Datacenter Switchover (September 2024) - https://phabricator.wikimedia.org/T371130 (10Scott_French) 03NEW [18:17:15] 06serviceops, 10Charts, 10Shellbox, 06SRE: Figure out how a shellbox instance for the Chart extension would work - https://phabricator.wikimedia.org/T370739#10019055 (10Legoktm) Unless it's too slow performance wise, I do think using Shellbox is the easiest path to production in that it's already compliant... [18:55:28] 06serviceops, 10Wikimedia-Apache-configuration: Change redirect target of sep11.wikipedia.org - https://phabricator.wikimedia.org/T367014#10019178 (10Dzahn) 05Open→03Resolved a:03Dzahn ` curl -q https://sep11.wikipedia.org | grep -i moved ... 301 Moved Permanently

Moved Permanently... [20:29:31] 06serviceops, 06SRE: confd setup left without configuration doesn't stop confd - https://phabricator.wikimedia.org/T356296#10019459 (10Dzahn) @cmooney In `profile::firewall` there is a `if $defs_from_etcd and $defs_from_etcd_nft`. So if both are true that installs `confd::file { '/etc/nftables/sets/requestct... [20:33:49] 06serviceops, 06SRE: confd setup left without configuration doesn't stop confd - https://phabricator.wikimedia.org/T356296#10019478 (10Dzahn) Yea, so this: ` if $defs_from_etcd { confd::file { '/etc/ferm/conf.d/00_defs_requestctl': ensure => stdlib::ensure($provider == 'ferm'... [20:54:53] 06serviceops, 06SRE, 13Patch-For-Review: confd setup left without configuration doesn't stop confd - https://phabricator.wikimedia.org/T356296#10019540 (10Dzahn) This is an attempt to fix it per logic "**if the provider is nft and we do NOT pull requestctl data.. THEN ... remove confd**". https://gerrit.wik... [23:02:24] 06serviceops, 10Charts, 10Shellbox, 06SRE: Figure out how a shellbox instance for the Chart extension would work - https://phabricator.wikimedia.org/T370739#10019839 (10Catrope) @akosiaris I'm trying to figure out how we should proceed based on your comment. Should we develop a service based on (an up-to-d...