[00:10:52] 06serviceops, 10MediaWiki-Internationalization, 10MediaWiki-extensions-General, 10WMF-General-or-Unknown, and 2 others: Update footer links to direct to proper locations on Foundation Governance Wiki - https://phabricator.wikimedia.org/T331680#9585848 (10Varnent) On hold pending resolution of T358541 [03:15:12] 06serviceops, 10Cloud-VPS: OOM livelock stalls - https://phabricator.wikimedia.org/T358634#9586029 (10tstarling) >>! In T358634#9582468, @Joe wrote: > I don't think it holds any ground for systems involved in live responses or which have strict latency requirements in general. > > For instance, enabling swap... [03:51:59] 06serviceops, 10Cloud-VPS: OOM livelock stalls - https://phabricator.wikimedia.org/T358634#9586043 (10tstarling) >>! In T358634#9582894, @dcaro wrote: > To clarify, this task is to request enabling it on CloudVPS instances by default, or to enable it in wiki production machines? (or both?) I wanted to share m... [04:18:14] 06serviceops: etcdmirror does not recover from a cleared waitIndex - https://phabricator.wikimedia.org/T358636#9586057 (10Scott_French) After thinking about this a bit more today, I think I'm onboard with the idea of expanding replication to include /spicerack. If we're comfortable with that being the default f... [07:43:08] 06serviceops: etcdmirror does not recover from a cleared waitIndex - https://phabricator.wikimedia.org/T358636#9586170 (10Joe) In the meantime, I remembered why we were only replicating the `/conftool` keyspace: in the past, we had another prefix called `/eventlogging` that was used only in eqiad and was suppos... [07:55:12] 06serviceops: etcdmirror does not recover from a cleared waitIndex - https://phabricator.wikimedia.org/T358636#9586184 (10Joe) >>! In T358636#9586057, @Scott_French wrote: > If we're comfortable with that being the default for the entire keyspace (i.e., even for new workloads) then that's a fairly straightforwar... [10:59:47] 06serviceops, 06Content-Transform-Team, 10MW-on-K8s, 06SRE, and 2 others: Reimage parse* hosts as kubernetes nodes - https://phabricator.wikimedia.org/T358752 (10akosiaris) [11:33:54] 06serviceops, 06Content-Transform-Team-WIP, 10Page Content Service, 10RESTBase Sunsetting: Raise mw-api-int replicas for increased load from mobileapps - https://phabricator.wikimedia.org/T356497#9586677 (10Clement_Goubert) I'll prepare the patch. Going to 240 replicas would put us at around 700 CPUs avail... [11:49:24] 06serviceops, 06Data-Engineering, 10WMF-JobQueue, 07Unstewarded-production-error, and 2 others: Could not enqueue jobs: "Unable to deliver all events: 503: Service Unavailable" - https://phabricator.wikimedia.org/T249745#9586715 (10Clement_Goubert) >>! In T249745#9583374, @gmodena wrote: > Hey @Clement_Gou... [12:42:41] 06serviceops, 06Data-Engineering, 10WMF-JobQueue, 07Unstewarded-production-error, and 2 others: Could not enqueue jobs: "Unable to deliver all events: 503: Service Unavailable" - https://phabricator.wikimedia.org/T249745#9586820 (10Joe) When we're talking about errors, it's always a good idea to reason in... [14:16:24] 06serviceops, 06Data-Engineering, 06MediaWiki-Engineering, 10WMF-JobQueue, and 3 others: Could not enqueue jobs: "Unable to deliver all events: 503: Service Unavailable" - https://phabricator.wikimedia.org/T249745#9587080 (10MSantos) [14:36:28] 06serviceops, 10CX-cxserver, 10RESTBase Sunsetting, 10Language-2024-January-March, 13Patch-For-Review: Make cxserver call parsoid endpoints on MediaWiki, instead of going through RESTbase - https://phabricator.wikimedia.org/T344982#9587206 (10Nikerabbit) [14:37:07] 06serviceops, 10CX-cxserver, 10RESTBase Sunsetting, 10Language-2024-January-March, 13Patch-For-Review: Make cxserver call parsoid endpoints on MediaWiki, instead of going through RESTbase - https://phabricator.wikimedia.org/T344982#9587214 (10Nikerabbit) [14:37:23] 06serviceops, 10CX-cxserver, 10RESTBase Sunsetting, 10Language-2024-January-March: Make cxserver call parsoid endpoints on MediaWiki, instead of going through RESTbase - https://phabricator.wikimedia.org/T344982#9587216 (10Nikerabbit) 05In progress→03Resolved [14:37:26] 06serviceops, 10RESTBase Sunsetting, 07Epic, 13Patch-For-Review: Replace usage of RESTbase parsoid endpoints - https://phabricator.wikimedia.org/T328559#9587218 (10Nikerabbit) [14:37:30] 06serviceops, 06DC-Ops, 06Data-Persistence, 06Traffic, and 4 others: ☂️ Northward Datacentre Switchover (March 2024) - https://phabricator.wikimedia.org/T357547#9587221 (10Trizek-WMF) [14:37:46] 06serviceops, 10CX-cxserver, 10RESTBase Sunsetting, 10Language-2024-January-March: Make cxserver call parsoid endpoints on MediaWiki, instead of going through RESTbase - https://phabricator.wikimedia.org/T344982#9587222 (10Nikerabbit) [14:37:53] 06serviceops, 10CommRel-Specialists-Support: CommRel support for Northward Datacentre Switchover (March 2024) - https://phabricator.wikimedia.org/T358233#9587219 (10Trizek-WMF) 05Open→03In progress p:05Triage→03High [14:46:16] 06serviceops, 10CommRel-Specialists-Support, 07User-notice: CommRel support for Northward Datacentre Switchover (March 2024) - https://phabricator.wikimedia.org/T358233#9587248 (10Trizek-WMF) [14:46:33] 06serviceops, 10CommRel-Specialists-Support, 07User-notice: CommRel support for Northward Datacentre Switchover (March 2024) - https://phabricator.wikimedia.org/T358233#9587256 (10Trizek-WMF) @jijiki, I'll be your host on this journey. Have you changed anything major/noticeable compared to the previous rea... [14:48:36] 06serviceops, 10CommRel-Specialists-Support, 07User-notice: CommRel support for Northward Datacentre Switchover (March 2024) - https://phabricator.wikimedia.org/T358233#9587271 (10Trizek-WMF) a:03Trizek-WMF [16:24:15] 06serviceops, 06SRE: Memcached, mcrouter in MediaWiki on Kubernetes - https://phabricator.wikimedia.org/T277711#9587697 (10jijiki) [16:26:26] 06serviceops, 10MW-on-K8s, 13Patch-For-Review, 10Radar: mcrouter daemonset on mw-on-k8s - https://phabricator.wikimedia.org/T346690#9587695 (10jijiki) 05Stalled→03In progress [19:57:25] 06serviceops, 06Data-Engineering, 06Data-Platform-SRE, 06SRE, 10Event-Platform: DRY kafka broker declaration in helmfiles - https://phabricator.wikimedia.org/T253058#9588704 (10Ottomata) +1, or add this as a subtask of that? Either good with me! [20:31:19] 06serviceops, 10Wikimedia-Apache-configuration, 13Patch-For-Review: Investigate restricting match pattern on /wiki RewriteRule - https://phabricator.wikimedia.org/T357595#9588786 (10RLazarus) a:03RLazarus [23:07:23] 06serviceops: etcdmirror does not recover from a cleared waitIndex - https://phabricator.wikimedia.org/T358636#9589274 (10Scott_French) > We control what goes into etcd quite closely. Ah, that is a good point: changes in workload are strictly limited to what we already (or will in the future) allow by ACL. > O...