[06:13:07] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover, and 2 others: March 2023 Datacenter Switchover - https://phabricator.wikimedia.org/T327920 (10Marostegui) [08:17:28] 10serviceops, 10SRE: Remove jessie and stretch-based images from our image registry - https://phabricator.wikimedia.org/T335333 (10MoritzMuehlenhoff) [09:04:08] 10serviceops, 10Infrastructure-Foundations, 10SRE: Annotate images in our registry with OS (and OS version) - https://phabricator.wikimedia.org/T335337 (10MoritzMuehlenhoff) [09:52:47] 10serviceops, 10Infrastructure-Foundations, 10SRE: Annotate images in our registry with OS (and OS version) - https://phabricator.wikimedia.org/T335337 (10JMeybohm) The initial idea (at least for production-images) was to not care (versioning wise) about the underlying OS version. This makes it more easy do... [10:07:35] 10serviceops, 10Infrastructure-Foundations, 10SRE, 10Datacenter-Switchover: sre.discovery.datacenter breaks on services not in "production" state - https://phabricator.wikimedia.org/T335341 (10Clement_Goubert) [10:32:19] 10serviceops, 10Infrastructure-Foundations, 10WikimediaDebug, 10Performance-Team (Radar): Upgrade php-excimer package from 1.0.4 to 1.1.1 - https://phabricator.wikimedia.org/T332964 (10MoritzMuehlenhoff) The updated Excimer has been rolled out across production, I'll resolve the task when I've updated the... [10:53:20] 10serviceops, 10Infrastructure-Foundations, 10SRE: Annotate images in our registry with OS (and OS version) - https://phabricator.wikimedia.org/T335337 (10MoritzMuehlenhoff) >>! In T335337#8803990, @JMeybohm wrote: > The initial idea (at least for production-images) was to not care (naming wise) about the un... [11:45:09] 10serviceops, 10Infrastructure-Foundations, 10SRE, 10User-MoritzMuehlenhoff: Annotate images in our registry with OS (and OS version) - https://phabricator.wikimedia.org/T335337 (10MoritzMuehlenhoff) [11:49:33] 10serviceops, 10RESTbase Sunsetting, 10Parsoid (Tracking): Enable WarmParsoidParserCache on all wikis - https://phabricator.wikimedia.org/T329366 (10daniel) [11:52:26] 10serviceops, 10RESTbase Sunsetting, 10Parsoid (Tracking): Enable WarmParsoidParserCache on all wikis - https://phabricator.wikimedia.org/T329366 (10daniel) It seems like the next step here is "Enable the jobs for wikis in batches, with SRE assistance. Possibly move more parsoid nodes to jobrunners if needed... [11:52:58] 10serviceops, 10RESTbase Sunsetting, 10Parsoid (Tracking): Enable WarmParsoidParserCache on all wikis - https://phabricator.wikimedia.org/T329366 (10daniel) > Noting that I think (not sure, Daniel can confirm?) this is not going to be enabled for commons and wikidata which is the biggest firehose of edits so... [12:53:17] o/ I have a long maint script running from mwmaint2002 and I'm not 100% sure that it'll be ended by tomorrow 14:00, if it's not should I kill it before the switch? [13:10:22] dcausse: the switchover cookbook will kill it anyway [13:13:03] akosiaris: ok good to know thanks! [13:32:38] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover, and 2 others: 25 April 2023 Service Switchback checklist - https://phabricator.wikimedia.org/T335015 (10Clement_Goubert) [13:32:56] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover, and 2 others: 25 April 2023 Service Switchback checklist - https://phabricator.wikimedia.org/T335015 (10Clement_Goubert) [13:33:51] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover, and 2 others: March 2023 Datacenter Switchover - https://phabricator.wikimedia.org/T327920 (10Clement_Goubert) [13:34:01] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover, and 2 others: 25 April 2023 Service Switchback checklist - https://phabricator.wikimedia.org/T335015 (10Clement_Goubert) 05Open→03In progress p:05Triage→03High [13:35:18] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover, and 2 others: 25 April 2023 Service Switchback checklist - https://phabricator.wikimedia.org/T335015 (10Clement_Goubert) [13:35:21] Last call for issues with https://phabricator.wikimedia.org/T335015 procedure [13:36:42] I'm wondering if I wouldn't be better served by just using service-route with the list of A/P services [13:36:45] akosiaris: ^ [13:39:34] Hmm since service-route does not dtrt on its own for A/P I'll keep with the "depool codfw completely, repool only a/a" strategy [13:41:26] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover, and 2 others: 25 April 2023 Service Switchback checklist - https://phabricator.wikimedia.org/T335015 (10Clement_Goubert) [13:44:48] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover, and 2 others: 25 April 2023 Service Switchback checklist - https://phabricator.wikimedia.org/T335015 (10Clement_Goubert) [13:46:44] 10serviceops, 10Machine-Learning-Team: docker-pkg fails to upload big Docker images to the registry - https://phabricator.wikimedia.org/T335177 (10akosiaris) High, thanks for this task. So, let me say that a 14G image is not just a corner case, it's a total first. The largest images we have up to date are the... [13:48:28] claime: https://phabricator.wikimedia.org/T335015 LGTM [13:48:44] ack thx [13:53:48] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover, and 2 others: 25 April 2023 Service Switchback checklist - https://phabricator.wikimedia.org/T335015 (10Clement_Goubert) [14:01:14] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover, and 2 others: 25 April 2023 Service Switchback checklist - https://phabricator.wikimedia.org/T335015 (10ops-monitoring-bot) cgoubert@cumin1001 - Cookbook cookbooks.sre.discovery.datacenter depool all services in codfw: Datacenter Services Sw... [14:04:37] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover, and 2 others: 25 April 2023 Service Switchback checklist - https://phabricator.wikimedia.org/T335015 (10ops-monitoring-bot) cgoubert@cumin1001 - Cookbook cookbooks.sre.discovery.datacenter depool all services in codfw: Datacenter Services Sw... [14:04:43] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover, and 2 others: 25 April 2023 Service Switchback checklist - https://phabricator.wikimedia.org/T335015 (10ops-monitoring-bot) cgoubert@cumin1001 - Cookbook cookbooks.sre.discovery.datacenter depool all services in codfw: Datacenter Services Sw... [14:19:00] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover, and 2 others: 25 April 2023 Service Switchback checklist - https://phabricator.wikimedia.org/T335015 (10ops-monitoring-bot) cgoubert@cumin1001 - Cookbook cookbooks.sre.discovery.datacenter depool all services in codfw: Datacenter Services Sw... [14:19:46] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover, and 2 others: 25 April 2023 Service Switchback checklist - https://phabricator.wikimedia.org/T335015 (10ops-monitoring-bot) cgoubert@cumin1001 - Cookbook cookbooks.sre.discovery.datacenter depool all services in codfw: Datacenter Services Sw... [14:25:10] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover, and 2 others: 25 April 2023 Service Switchback checklist - https://phabricator.wikimedia.org/T335015 (10Clement_Goubert) Encountering some intermittent lockups of the cookbook {P47279} Restarting the cookbook is idempotent, doing that pend... [14:25:20] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover, and 2 others: 25 April 2023 Service Switchback checklist - https://phabricator.wikimedia.org/T335015 (10ops-monitoring-bot) cgoubert@cumin1001 - Cookbook cookbooks.sre.discovery.datacenter depool all services in codfw: Datacenter Services Sw... [14:25:30] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover, and 2 others: 25 April 2023 Service Switchback checklist - https://phabricator.wikimedia.org/T335015 (10Clement_Goubert) [14:26:53] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover, and 2 others: 25 April 2023 Service Switchback checklist - https://phabricator.wikimedia.org/T335015 (10ops-monitoring-bot) cgoubert@cumin1001 - Cookbook cookbooks.sre.discovery.datacenter pool all active/active services in codfw: Datacenter... [14:27:07] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover, and 2 others: 25 April 2023 Service Switchback checklist - https://phabricator.wikimedia.org/T335015 (10Clement_Goubert) [14:39:54] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover, and 2 others: 25 April 2023 Service Switchback checklist - https://phabricator.wikimedia.org/T335015 (10Clement_Goubert) [14:42:36] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover, and 2 others: 25 April 2023 Service Switchback checklist - https://phabricator.wikimedia.org/T335015 (10ops-monitoring-bot) cgoubert@cumin1001 - Cookbook cookbooks.sre.discovery.datacenter pool all active/active services in codfw: Datacenter... [14:44:39] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover, and 2 others: 25 April 2023 Service Switchback checklist - https://phabricator.wikimedia.org/T335015 (10Clement_Goubert) [14:46:25] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover, and 2 others: 25 April 2023 Service Switchback checklist - https://phabricator.wikimedia.org/T335015 (10Clement_Goubert) [14:49:52] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover, and 2 others: 25 April 2023 Service Switchback checklist - https://phabricator.wikimedia.org/T335015 (10Clement_Goubert) [14:53:23] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover, and 2 others: 25 April 2023 Service Switchback checklist - https://phabricator.wikimedia.org/T335015 (10Clement_Goubert) [14:55:48] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover, and 2 others: 25 April 2023 Service Switchback checklist - https://phabricator.wikimedia.org/T335015 (10Clement_Goubert) [15:19:46] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover, and 2 others: 25 April 2023 Service Switchback checklist - https://phabricator.wikimedia.org/T335015 (10Clement_Goubert) [15:23:01] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover, and 2 others: March 2023 Datacenter Switchover - https://phabricator.wikimedia.org/T327920 (10Clement_Goubert) [15:24:09] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover, and 2 others: 25 April 2023 Service Switchback checklist - https://phabricator.wikimedia.org/T335015 (10Clement_Goubert) 05In progress→03Resolved [15:39:33] 10serviceops, 10Infrastructure-Foundations, 10SRE, 10Datacenter-Switchover: sre.discovery.datacenter should support only moving the active/passive services to the other datacenter - https://phabricator.wikimedia.org/T335364 (10Clement_Goubert) [15:41:13] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover: March 2023 Datacenter Switchover eqiad pooling schedule - https://phabricator.wikimedia.org/T328903 (10Clement_Goubert) 05In progress→03Resolved [15:41:31] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover, and 2 others: March 2023 Datacenter Switchover - https://phabricator.wikimedia.org/T327920 (10Clement_Goubert) [15:42:27] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover, 10Patch-For-Review: March 2023 Datacenter Switchover Excluded services - https://phabricator.wikimedia.org/T329193 (10Clement_Goubert) 05Open→03Resolved [15:42:39] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover, and 2 others: March 2023 Datacenter Switchover - https://phabricator.wikimedia.org/T327920 (10Clement_Goubert) [15:42:48] 10serviceops, 10Infrastructure-Foundations, 10SRE, 10Datacenter-Switchover: sre.discovery.datacenter should support switching the active/passive services to the other datacenter - https://phabricator.wikimedia.org/T335364 (10Clement_Goubert) [16:04:29] 10serviceops, 10Infrastructure-Foundations, 10SRE, 10Datacenter-Switchover, 10Patch-For-Review: sre.discovery.datacenter breaks on services not in "production" state - https://phabricator.wikimedia.org/T335341 (10Clement_Goubert) p:05Triage→03Medium a:03Clement_Goubert [16:53:17] jayme: [16:53:17] https://gerrit.wikimedia.org/r/c/operations/docker-images/production-images/+/911905 [20:38:53] 10serviceops, 10SRE-OnFire, 10Traffic, 10conftool, 10Sustainability (Incident Followup): Pybal maintenances break safe-service-restart.py (and thus prevent scap deploys of mediawiki) - https://phabricator.wikimedia.org/T334703 (10BCornwall) @bblack and @cdanis: Could the ticket title/description be updat... [21:33:07] 10serviceops, 10MediaWiki-extensions-PropertySuggester, 10Wikidata, 10wdwb-tech, and 2 others: New Service Request SchemaTree - https://phabricator.wikimedia.org/T301471 (10Michaelcochez) The testing code is now implemented, and we found two small issues with it. These have now been resolved and the code i... [22:00:11] 10serviceops, 10MediaWiki-extensions-PropertySuggester, 10Wikidata, 10wdwb-tech, and 2 others: New Service Request SchemaTree - https://phabricator.wikimedia.org/T301471 (10Dzahn) @Michaelcochez Probably it makes sense to merge but also comment on the ticket you linked to and get in touch with thcipriani t...