[01:02:58] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover, and 2 others: March 2023 Datacenter Switchover - https://phabricator.wikimedia.org/T327920 (10Dzahn) [01:18:19] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover, and 2 others: March 2023 Datacenter Switchover - https://phabricator.wikimedia.org/T327920 (10Dzahn) [07:58:57] 10serviceops, 10Data-Engineering, 10Data-Persistence, 10Infrastructure-Foundations, and 8 others: eqiad row B switches upgrade - https://phabricator.wikimedia.org/T330165 (10Marostegui) [09:20:52] 10serviceops, 10Citoid: citoid having stability issues - https://phabricator.wikimedia.org/T330768 (10akosiaris) p:05Triage→03Medium I 've updated https://grafana.wikimedia.org/d/NJkCVermz/citoid?orgId=1 a bit (more could be done, but I 'd rather do it in a consistent automated way across all of the Servic... [09:24:20] Who knows anything about extensions/PageTriage/cron/updatePageTriageQueue.php ? [09:24:34] It's failing for test2wiki (not a big deal but still) [09:29:33] 10serviceops, 10Infrastructure-Foundations, 10SRE, 10Datacenter-Switchover: apt.wikimedia.org post-switchover - https://phabricator.wikimedia.org/T330985 (10Clement_Goubert) [09:30:30] 10serviceops, 10Infrastructure-Foundations, 10SRE, 10Datacenter-Switchover: apt.wikimedia.org post-switchover - https://phabricator.wikimedia.org/T330985 (10Clement_Goubert) p:05Triage→03Medium a:05Clement_Goubert→03jBond_WMF [09:31:47] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover: Post March 2023 Datacenter Switchover Tasks - https://phabricator.wikimedia.org/T328907 (10Clement_Goubert) [09:31:57] 10serviceops, 10Infrastructure-Foundations, 10SRE, 10Datacenter-Switchover: apt.wikimedia.org post-switchover - https://phabricator.wikimedia.org/T330985 (10Clement_Goubert) 05Open→03Resolved Fixed in https://gerrit.wikimedia.org/r/c/operations/puppet/+/893669 [10:38:14] 10serviceops, 10Infrastructure-Foundations, 10SRE, 10Datacenter-Switchover: apt.wikimedia.org post-switchover - https://phabricator.wikimedia.org/T330985 (10jbond) a:05jBond_WMF→03jbond [10:58:05] 10serviceops, 10Citoid: citoid having stability issues - https://phabricator.wikimedia.org/T330768 (10Mvolz) >>! In T330768#8659732, @akosiaris wrote: > I 've updated https://grafana.wikimedia.org/d/NJkCVermz/citoid?orgId=1 a bit (more could be done, but I 'd rather do it in a consistent automated way across a... [11:02:02] 10serviceops, 10DBA, 10Data-Engineering, 10Infrastructure-Foundations, and 8 others: eqiad row B switches upgrade - https://phabricator.wikimedia.org/T330165 (10Marostegui) @ayounsi @akosiaris @Joe to confirm, we are going to depool eqiad before this maintenance like we've done in codfw right? [11:02:18] 10serviceops, 10DBA, 10Data-Engineering, 10Infrastructure-Foundations, and 8 others: eqiad row B switches upgrade - https://phabricator.wikimedia.org/T330165 (10Marostegui) [11:24:43] 10serviceops, 10SRE, 10Datacenter-Switchover: Add scap lock/unlock steps to sre.switchdc.mediawiki cookbook - https://phabricator.wikimedia.org/T330996 (10Clement_Goubert) [11:26:04] 10serviceops, 10Infrastructure-Foundations, 10SRE, 10Datacenter-Switchover: Add scap lock/unlock steps to sre.switchdc.mediawiki cookbook - https://phabricator.wikimedia.org/T330996 (10Clement_Goubert) p:05Triage→03Medium [11:26:08] 10serviceops, 10Infrastructure-Foundations, 10SRE, 10Datacenter-Switchover: Add scap lock/unlock steps to sre.switchdc.mediawiki cookbook - https://phabricator.wikimedia.org/T330996 (10Clement_Goubert) [11:28:48] 10serviceops, 10Infrastructure-Foundations, 10SRE, 10Datacenter-Switchover: Support locking cookbooks run except for switchover related cookbooks - https://phabricator.wikimedia.org/T330997 (10Clement_Goubert) [11:29:14] 10serviceops, 10Infrastructure-Foundations, 10SRE, 10Datacenter-Switchover: Support locking cookbooks run except for switchover related cookbooks - https://phabricator.wikimedia.org/T330997 (10Clement_Goubert) p:05Triage→03Medium [11:41:09] 10serviceops, 10Infrastructure-Foundations, 10SRE, 10Datacenter-Switchover: Support locking cookbooks run except for switchover related cookbooks - https://phabricator.wikimedia.org/T330997 (10Clement_Goubert) [11:41:12] 10serviceops, 10Data-Persistence, 10SRE, 10Datacenter-Switchover: Post March 2023 Datacenter Switchover Tasks - https://phabricator.wikimedia.org/T328907 (10Clement_Goubert) [12:12:40] 10serviceops, 10DBA, 10Data-Engineering, 10Infrastructure-Foundations, and 8 others: eqiad row B switches upgrade - https://phabricator.wikimedia.org/T330165 (10akosiaris) >>! In T330165#8660042, @Marostegui wrote: > @ayounsi @akosiaris @Joe to confirm, we are going to depool eqiad before this maintenance... [12:16:57] 10serviceops, 10Data-Persistence (work done), 10SRE, 10Datacenter-Switchover, 10Sustainability (Incident Followup): Globalize mwconfig ReadOnly - https://phabricator.wikimedia.org/T330304 (10Clement_Goubert) a:05Clement_Goubert→03None [12:49:48] 10serviceops, 10MW-on-K8s, 10Scap, 10Upstream: Kubernetes configuration file is group-readable - https://phabricator.wikimedia.org/T329899 (10Clement_Goubert) p:05Triage→03Low [13:02:52] 10serviceops, 10Infrastructure-Foundations, 10SRE, 10SRE-tools, 10Datacenter-Switchover: Support locking cookbooks run except for switchover related cookbooks - https://phabricator.wikimedia.org/T330997 (10Volans) [13:15:09] 10serviceops, 10DBA, 10Data-Engineering, 10Infrastructure-Foundations, and 8 others: eqiad row B switches upgrade - https://phabricator.wikimedia.org/T330165 (10BTullis) [13:18:36] 10serviceops, 10DBA, 10Data-Engineering, 10Infrastructure-Foundations, and 8 others: eqiad row B switches upgrade - https://phabricator.wikimedia.org/T330165 (10Marostegui) >>! In T330165#8660202, @akosiaris wrote: >>>! In T330165#8660042, @Marostegui wrote: >> @ayounsi @akosiaris @Joe to confirm, we are g... [14:02:05] FYI puppet on aphlict2001 is broken: Execution of '/usr/bin/scap deploy-local --repo phabricator/deployment -D log_json:False' returned 1 [14:21:28] 10serviceops, 10Data-Engineering-Planning, 10Event-Platform Value Stream (Sprint 09), 10Service-deployment-requests: New Service Request mediawiki-page-content-change-enrichment - https://phabricator.wikimedia.org/T330507 (10JArguello-WMF) [14:24:40] also on the releases hosts File[/srv/patches] owner changed 'jenkins' to 'root' (corrective) and group changed 705 to 'root' (corrective) [14:24:43] at evey run [14:28:59] scap deployment is another instance of git safe.directory issues [14:29:18] looks like it, yes [14:32:54] There's already a task for it https://phabricator.wikimedia.org/T330393 [14:34:03] Ah no different issue [14:36:25] 10serviceops, 10Citoid: citoid having stability issues - https://phabricator.wikimedia.org/T330768 (10akosiaris) > my gut says we should count 4xx Should NOT count. My mistake, sorry about that. [15:23:33] 10serviceops, 10Citoid, 10Platform Team Workboards (Platform Engineering Reliability): citoid having stability issues - https://phabricator.wikimedia.org/T330768 (10hnowlan) [15:24:57] volans: CR sent for the two mentioned issues [15:25:37] <3 thx [15:57:00] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad: Reset management module of mc1039 - https://phabricator.wikimedia.org/T330072 (10Cmjohnson) 05Open→03Resolved a:05Jclark-ctr→03Cmjohnson The server password was not set correctly, fixed and you should be good to go. [15:57:03] 10serviceops: Upgrade mc* and mc-gp* hosts to Debian Bullseye - https://phabricator.wikimedia.org/T293216 (10Cmjohnson)