[03:13:41] o/ Today I will start migrating testcommons and commons plus cleaning up logging of some wikis that have flagged revs logs there [07:32:08] 10DBA, 10Infrastructure-Foundations, 10SRE, 10netops, 10cloud-services-team (Kanban): Switch buffer re-partition - Eqiad Row A - https://phabricator.wikimedia.org/T286032 (10elukey) >>! In T286032#7197078, @MoritzMuehlenhoff wrote: > Looking at Ganeti VMs, they broadly fall under three/four categories: >... [07:39:45] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, and 2 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10elukey) [08:03:53] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, and 2 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10cmooney) [08:04:04] FYI, dborch* and dbmonitor* will be briefly unavailable now, going to upgrade Apache on these [08:04:07] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, 10netops: Switch buffer re-partition - Eqiad Row C - https://phabricator.wikimedia.org/T286065 (10cmooney) [08:04:21] 10DBA, 10Infrastructure-Foundations, 10SRE, 10netops: Switch buffer re-partition - Eqiad Row B - https://phabricator.wikimedia.org/T286061 (10cmooney) [08:04:35] 10DBA, 10Infrastructure-Foundations, 10SRE, 10netops, 10cloud-services-team (Kanban): Switch buffer re-partition - Eqiad Row A - https://phabricator.wikimedia.org/T286032 (10cmooney) [08:26:43] 10Blocked-on-schema-change, 10DBA, 10Dumps-Generation: Schema change for dropping rev_page_id index - https://phabricator.wikimedia.org/T285149 (10ArielGlenn) >>! In T285149#7199965, @Marostegui wrote: > ok, let's see what @ArielGlenn says. Thanks! Not ignoring you all, but I'll have to dig into this a bit... [08:32:49] 10Blocked-on-schema-change, 10DBA, 10Dumps-Generation: Schema change for dropping rev_page_id index - https://phabricator.wikimedia.org/T285149 (10Ladsgroup) No worries. There is no rush. Take your time. Thanks for investigating. [08:58:26] 10DBA, 10Patch-For-Review: Rename dbstore1004 to db1183 and place it on m5 - https://phabricator.wikimedia.org/T284622 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by kormat on cumin1001.eqiad.wmnet for hosts: ` db1183.eqiad.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/202107120858... [08:59:57] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, 10netops: Switch buffer re-partition - Eqiad Row C - https://phabricator.wikimedia.org/T286065 (10aborrero) >>! In T286065#7194569, @Bstorm wrote: > @aborrero does cloudgw require manual failover? it doesn't require manual failover, but we could... [09:02:15] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, and 2 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10Kormat) [09:07:07] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, 10netops: Switch buffer re-partition - Eqiad Row C - https://phabricator.wikimedia.org/T286065 (10cmooney) [09:07:53] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, 10netops: Switch buffer re-partition - Eqiad Row C - https://phabricator.wikimedia.org/T286065 (10cmooney) [09:12:00] 10DBA, 10SRE, 10Datacenter-Switchover: switchdc should automatically downtime "Read only" checks on DB masters being switched - https://phabricator.wikimedia.org/T285803 (10Kormat) >>! In T285803#7203653, @RLazarus wrote: > The downtime cookbook uses [[https://gerrit.wikimedia.org/r/plugins/gitiles/operation... [09:12:06] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, 10netops: Switch buffer re-partition - Eqiad Row C - https://phabricator.wikimedia.org/T286065 (10cmooney) [09:13:25] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, 10netops: Switch buffer re-partition - Eqiad Row C - https://phabricator.wikimedia.org/T286065 (10cmooney) [09:15:19] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, 10netops: Switch buffer re-partition - Eqiad Row C - https://phabricator.wikimedia.org/T286065 (10cmooney) [09:15:56] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, 10netops: Switch buffer re-partition - Eqiad Row C - https://phabricator.wikimedia.org/T286065 (10cmooney) [09:19:50] 10DBA: Rename dbstore1004 to db1183 and place it on m5 - https://phabricator.wikimedia.org/T284622 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['db1183.eqiad.wmnet'] ` and were **ALL** successful. [09:29:30] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, 10netops: Switch buffer re-partition - Eqiad Row C - https://phabricator.wikimedia.org/T286065 (10MoritzMuehlenhoff) [09:38:52] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, and 2 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10cmooney) [09:52:25] 10DBA, 10Infrastructure-Foundations, 10SRE, 10netops: Switch buffer re-partition - Eqiad Row B - https://phabricator.wikimedia.org/T286061 (10cmooney) [09:59:18] 10DBA, 10Infrastructure-Foundations, 10SRE, 10netops, 10cloud-services-team (Kanban): Switch buffer re-partition - Eqiad Row A - https://phabricator.wikimedia.org/T286032 (10cmooney) [10:26:36] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, 10netops: Switch buffer re-partition - Eqiad Row C - https://phabricator.wikimedia.org/T286065 (10cmooney) [10:29:13] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, 10netops: Switch buffer re-partition - Eqiad Row C - https://phabricator.wikimedia.org/T286065 (10cmooney) @BStorm / @aborrero as mentioned on IRC I messed up with the list of servers here, inadvertently including those in the row connected to //cl... [10:33:10] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, and 2 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10cmooney) [10:33:43] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, and 2 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10cmooney) @Bstorm / @aborrero as mentioned on IRC I messed up with the list of servers here, inadvertently including those in the row connected to cl... [10:43:46] 10DBA, 10SRE, 10Datacenter-Switchover: switchdc should automatically downtime "Read only" checks on DB masters being switched - https://phabricator.wikimedia.org/T285803 (10Volans) If I may add my 2 cents, `icinga-downtime` is a very simple and old bash script that only ensures that a given host is defined i... [14:56:39] 10DBA: fa_deleted_timestamp and fa_timestamp are binary(14) in code but varbinary(14) in production - https://phabricator.wikimedia.org/T277116 (10Kormat) [15:01:53] 10DBA: fa_deleted_timestamp and fa_timestamp are binary(14) in code but varbinary(14) in production - https://phabricator.wikimedia.org/T277116 (10Kormat) [15:08:56] 10DBA: fa_deleted_timestamp and fa_timestamp are binary(14) in code but varbinary(14) in production - https://phabricator.wikimedia.org/T277116 (10Kormat) [15:15:40] 10DBA: fa_deleted_timestamp and fa_timestamp are binary(14) in code but varbinary(14) in production - https://phabricator.wikimedia.org/T277116 (10Kormat) [15:25:28] 10DBA: fa_deleted_timestamp and fa_timestamp are binary(14) in code but varbinary(14) in production - https://phabricator.wikimedia.org/T277116 (10Kormat) [15:26:58] 10DBA: fa_deleted_timestamp and fa_timestamp are binary(14) in code but varbinary(14) in production - https://phabricator.wikimedia.org/T277116 (10Kormat) [15:27:13] 10DBA, 10Infrastructure-Foundations, 10SRE, 10netops, 10cloud-services-team (Kanban): Switch buffer re-partition - Eqiad Row A - https://phabricator.wikimedia.org/T286032 (10hnowlan) [15:30:33] 10DBA: fa_deleted_timestamp and fa_timestamp are binary(14) in code but varbinary(14) in production - https://phabricator.wikimedia.org/T277116 (10Kormat) [15:30:57] 10DBA: fa_deleted_timestamp and fa_timestamp are binary(14) in code but varbinary(14) in production - https://phabricator.wikimedia.org/T277116 (10Kormat) [15:45:32] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, and 2 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10MoritzMuehlenhoff) [15:45:56] 10DBA, 10Infrastructure-Foundations, 10SRE, 10netops: Switch buffer re-partition - Eqiad Row B - https://phabricator.wikimedia.org/T286061 (10MoritzMuehlenhoff) [17:02:20] 10DBA: fa_deleted_timestamp and fa_timestamp are binary(14) in code but varbinary(14) in production - https://phabricator.wikimedia.org/T277116 (10Kormat) [17:54:46] 10DBA, 10DiscussionTools, 10Performance-Team, 10Editing-team (Tracking): Post-deployment: evaluate impact on site performance - https://phabricator.wikimedia.org/T280606 (10Krinkle) >>! In T282761#7202587, @Krinkle wrote: > * disk space is still quite rapidly increasing despite shortened retention and dail... [17:55:16] 10DBA, 10DiscussionTools, 10Editing-team, 10Performance-Team (Radar): Post-deployment: (partly) ramp parser cache retention back up - https://phabricator.wikimedia.org/T280604 (10Krinkle) >>! In T282761#7202587, @Krinkle wrote: > * disk space is still quite rapidly increasing despite shortened retention an... [17:57:28] 10DBA, 10DiscussionTools, 10Editing-team, 10Performance-Team (Radar): Post-deployment: (partly) ramp parser cache retention back up - https://phabricator.wikimedia.org/T280604 (10Krinkle) [18:29:58] 10DBA, 10SRE, 10Datacenter-Switchover: switchdc should automatically downtime "Read only" checks on DB masters being switched - https://phabricator.wikimedia.org/T285803 (10RLazarus) a:03RLazarus Oh, that does sound better! I didn't realize `icinga-status` was out there, thanks for the pointer. Your plan s... [18:55:42] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, 10netops: Switch buffer re-partition - Eqiad Row C - https://phabricator.wikimedia.org/T286065 (10Dwisehaupt) [18:56:56] 10DBA, 10SRE, 10Datacenter-Switchover: switchdc should automatically downtime "Read only" checks on DB masters being switched - https://phabricator.wikimedia.org/T285803 (10Volans) >>! In T285803#7206669, @RLazarus wrote: > I'll start by refactoring the icinga module to write to the command file directly (f... [18:57:23] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, 10netops: Switch buffer re-partition - Eqiad Row C - https://phabricator.wikimedia.org/T286065 (10Dwisehaupt) Still need to confirm the window with Advancement, but it is looking ok right now. There will be some work on the FR-Tech side to ensure d... [21:28:12] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, 10netops: Switch buffer re-partition - Eqiad Row C - https://phabricator.wikimedia.org/T286065 (10Bstorm) >>! In T286065#7205088, @cmooney wrote: > @BStorm / @aborrero as mentioned on IRC I messed up with the list of servers here, inadvertently inc... [21:31:15] 10DBA, 10Infrastructure-Foundations, 10SRE, 10netops: Switch buffer re-partition - Eqiad Row B - https://phabricator.wikimedia.org/T286061 (10Bstorm) @cmooney Do the cloudsw switches get impacted by row B updates?