[01:33:31] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q3:rack/setup/install mw2420-mw2451 - https://phabricator.wikimedia.org/T326362 (10Papaul) [02:17:15] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q3:rack/setup/install mw2420-mw2451 - https://phabricator.wikimedia.org/T326362 (10Papaul) [03:08:12] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q3:rack/setup/install mw2420-mw2451 - https://phabricator.wikimedia.org/T326362 (10Papaul) [04:39:44] 10serviceops, 10ChangeProp, 10Content-Transform-Team-WIP, 10Page Content Service, and 3 others: Parsoid cache invalidation for mobile-sections seems not reliable - https://phabricator.wikimedia.org/T226931 (10Brycehughes) @akosiaris Actually, the Wikivoyage folks noted that this might flood people's watchl... [06:44:57] 10serviceops, 10ChangeProp, 10Content-Transform-Team-WIP, 10Page Content Service, and 3 others: Parsoid cache invalidation for mobile-sections seems not reliable - https://phabricator.wikimedia.org/T226931 (10Brycehughes) @akosiaris sorry about all the messages, but I notice that this still doesn't seem to... [12:05:57] if anyone fancies some easy friday reviewing, I want to start cleaning up the api-gateway chart so we can reuse it in the restbase deprecation stuff. This is a noop bar some spacing https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/887991 [12:34:30] 10serviceops, 10Data-Persistence, 10Toolhub, 10Datacenter-Switchover: Toolhub does not have a working Kubernetes deployment outside of eqiad - https://phabricator.wikimedia.org/T329319 (10akosiaris) >>! In T329319#8602389, @bd808 wrote: > @JMeybohm and @Clement_Goubert Do you expect the aux cluster to be r... [12:39:54] 10serviceops, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 9 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10EChetty) [12:42:21] 10serviceops, 10Data-Persistence, 10Toolhub, 10Datacenter-Switchover: Toolhub does not have a working Kubernetes deployment outside of eqiad - https://phabricator.wikimedia.org/T329319 (10Marostegui) I am also thinking, from a database point of view. If we want to use m5 databases in codfw _just_ for `tool... [12:46:25] 10serviceops, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 10 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10EChetty) [12:48:08] 10serviceops, 10Data-Persistence, 10Toolhub, 10Datacenter-Switchover: Toolhub does not have a working Kubernetes deployment outside of eqiad - https://phabricator.wikimedia.org/T329319 (10akosiaris) >>! In T329319#8604689, @Marostegui wrote: > I am also thinking, from a database point of view. If we want t... [12:56:52] 10serviceops, 10ChangeProp, 10Content-Transform-Team-WIP, 10Page Content Service, and 3 others: Parsoid cache invalidation for mobile-sections seems not reliable - https://phabricator.wikimedia.org/T226931 (10akosiaris) >>! In T226931#8603650, @Brycehughes wrote: > @akosiaris sorry about all the messages,... [13:48:04] 10serviceops, 10Product-Infrastructure-Team-Backlog-Deprecated, 10SRE, 10WMDE-TechWish-Maintenance, and 3 others: New Service Request geoshapes - https://phabricator.wikimedia.org/T274388 (10akosiaris) @Msantos, my current understanding is that we are pausing work on this. Should we set to `Stalled` ? [14:05:17] 10serviceops, 10Product-Infrastructure-Team-Backlog-Deprecated, 10SRE, 10WMDE-TechWish-Maintenance, and 3 others: New Service Request geoshapes - https://phabricator.wikimedia.org/T274388 (10awight) FWIW, there has been parallel work in {T216826} to containerize the whole kartotherian service, which curren... [15:03:16] 10serviceops, 10Thumbor: Incorrect thumbnail being returned by drmrs, eqiad and esams - https://phabricator.wikimedia.org/T328875 (10akosiaris) Thanks for this >>! In T328875#8598964, @Jonesey95 wrote: > https://en.wikipedia.org/wiki/Wikipedia:Village_pump_(technical)/Archive_202#Image_thumbnail_bug was less... [16:35:46] 10serviceops, 10Maps, 10Observability-Metrics, 10SRE, and 2 others: SLO dashboards with N latency targets - https://phabricator.wikimedia.org/T320749 (10herron) 05Open→03Resolved The updated dynamic SLO dashboard template and config structure is now live. I think we're good here! If any followup is n... [16:54:03] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q3:rack/setup/install mw2420-mw2451 - https://phabricator.wikimedia.org/T326362 (10Papaul) [17:38:21] 10serviceops, 10Service-deployment-requests: New Service Request 'security-api' - https://phabricator.wikimedia.org/T325147 (10sbassett) >>! In T325147#8590684, @STran wrote: > I can follow up on slack to set a meeting up with @Joe if we think that's the best next step. Feel free to add myself and @mstyles to... [18:08:52] 10serviceops, 10ChangeProp, 10Content-Transform-Team-WIP, 10Page Content Service, and 3 others: Parsoid cache invalidation for mobile-sections seems not reliable - https://phabricator.wikimedia.org/T226931 (10Brycehughes) @akosiaris Thanks much. Will keep an eye out for other examples. Will hold off on the... [18:11:48] 10serviceops, 10ChangeProp, 10Content-Transform-Team-WIP, 10Page Content Service, and 3 others: Parsoid cache invalidation for mobile-sections seems not reliable - https://phabricator.wikimedia.org/T226931 (10Brycehughes) @akosiaris so far as I can tell, even that link isn't working still. Compare the `ext... [21:41:21] 10serviceops, 10Data-Persistence, 10SRE, 10cloud-services-team, and 3 others: Wikitech issues for datacentre switchover (March 2023) - https://phabricator.wikimedia.org/T328768 (10nskaggs) Sounds like there's a plan in place here. Thank you! I did also want to add my support for {T237773} to avoid this typ...