[06:12:25] 10Blocked-on-schema-change, 10DBA: Schema change for watchlist.wl_notificationtimestamp going binary(14) from varbinary(14) - https://phabricator.wikimedia.org/T268392 (10Marostegui) [06:27:12] 10Blocked-on-schema-change, 10DBA, 10MW-1.37-notes (1.37.0-wmf.12; 2021-06-28): Schema change for renaming several indexes in logging table - https://phabricator.wikimedia.org/T270620 (10Marostegui) 05Stalled→03Open [06:34:42] 10Blocked-on-schema-change, 10DBA: Schema change for dropping default of img_timestamp and making it binary(14) - https://phabricator.wikimedia.org/T273360 (10Marostegui) [06:35:34] 10Blocked-on-schema-change, 10DBA: Schema change for dropping default of img_timestamp and making it binary(14) - https://phabricator.wikimedia.org/T273360 (10Marostegui) 05Stalled→03Open [06:35:36] 10DBA: DB maintenance work to do while eqiad is passive (June 2021) - https://phabricator.wikimedia.org/T285139 (10Marostegui) [06:39:30] 10Blocked-on-schema-change, 10DBA: Schema change for watchlist.wl_notificationtimestamp going binary(14) from varbinary(14) - https://phabricator.wikimedia.org/T268392 (10Marostegui) [06:40:30] 10Blocked-on-schema-change, 10DBA: Schema change for dropping default of img_timestamp and making it binary(14) - https://phabricator.wikimedia.org/T273360 (10Marostegui) [06:41:40] 10Blocked-on-schema-change, 10DBA: Schema change for dropping default of img_timestamp and making it binary(14) - https://phabricator.wikimedia.org/T273360 (10Marostegui) [06:43:57] 10Blocked-on-schema-change, 10DBA: Schema change for dropping default of img_timestamp and making it binary(14) - https://phabricator.wikimedia.org/T273360 (10Marostegui) [06:48:00] 10Blocked-on-schema-change, 10DBA: Schema change for dropping default of img_timestamp and making it binary(14) - https://phabricator.wikimedia.org/T273360 (10Marostegui) I have started the alter on `commonswiki.image`, this might take a while... [07:29:22] 10DBA, 10Infrastructure-Foundations, 10SRE, 10netops, 10cloud-services-team (Kanban): Switch buffer re-partition - Eqiad Row A - https://phabricator.wikimedia.org/T286032 (10MoritzMuehlenhoff) Looking at Ganeti VMs, they broadly fall under three/four categories: SPOF, will need a maintenance window decl... [07:34:58] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, and 2 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10Marostegui) [07:35:45] 10DBA, 10Infrastructure-Foundations, 10SRE, 10netops: Switch buffer re-partition - Eqiad Row B - https://phabricator.wikimedia.org/T286061 (10Marostegui) [07:36:13] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, 10netops: Switch buffer re-partition - Eqiad Row C - https://phabricator.wikimedia.org/T286065 (10Marostegui) [07:38:00] 10DBA, 10SRE, 10Datacenter-Switchover: switchdc should automatically downtime "Read only" checks on DB masters being switched - https://phabricator.wikimedia.org/T285803 (10Marostegui) p:05Triage→03Medium [09:19:45] 10Blocked-on-schema-change, 10DBA: Schema change for watchlist.wl_notificationtimestamp going binary(14) from varbinary(14) - https://phabricator.wikimedia.org/T268392 (10Marostegui) [09:19:54] 10DBA: DB maintenance work to do while eqiad is passive (June 2021) - https://phabricator.wikimedia.org/T285139 (10Marostegui) [09:20:06] 10Blocked-on-schema-change, 10DBA: Schema change for watchlist.wl_notificationtimestamp going binary(14) from varbinary(14) - https://phabricator.wikimedia.org/T268392 (10Marostegui) 05Open→03Resolved All done [09:24:26] 10DBA, 10Infrastructure-Foundations, 10SRE, 10netops: Switch buffer re-partition - Eqiad Row B - https://phabricator.wikimedia.org/T286061 (10MoritzMuehlenhoff) Looking at Ganeti VMs, they fall under three/four categories: SPOF, will need a maintenance window declared: * otrs1001 * an-tool1008 * an-tool10... [09:30:30] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, 10netops: Switch buffer re-partition - Eqiad Row C - https://phabricator.wikimedia.org/T286065 (10MoritzMuehlenhoff) Looking at Ganeti VMs, they fall under three/four categories: SPOF, will need a maintenance window declared: * an-tool1005 * an-to... [09:35:22] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, and 2 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10MoritzMuehlenhoff) [09:36:29] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, and 2 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10MoritzMuehlenhoff) [09:40:02] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, 10netops: Switch buffer re-partition - Eqiad Row C - https://phabricator.wikimedia.org/T286065 (10MoritzMuehlenhoff) [09:41:41] 10DBA, 10Infrastructure-Foundations, 10SRE, 10netops: Switch buffer re-partition - Eqiad Row B - https://phabricator.wikimedia.org/T286061 (10MoritzMuehlenhoff) [09:45:43] 10DBA, 10Infrastructure-Foundations, 10SRE, 10netops, 10cloud-services-team (Kanban): Switch buffer re-partition - Eqiad Row A - https://phabricator.wikimedia.org/T286032 (10MoritzMuehlenhoff) [10:28:32] 10DBA: Please optimize logging table in ruwiki - https://phabricator.wikimedia.org/T286102 (10Marostegui) I have tried this to see the benefits: Compressed table: ` root@db1096:~# ls -lh /srv/sqldata.s6/ruwiki/logging.ibd -rw-rw---- 1 mysql mysql 20G Jul 5 10:07 /srv/sqldata.s6/ruwiki/logging.ibd ` After: ` roo... [10:28:49] 10DBA: Please optimize logging table in ruwiki - https://phabricator.wikimedia.org/T286102 (10Marostegui) [10:37:43] 10Blocked-on-schema-change, 10DBA: Schema change for making cuc_id in cu_changes unsigned - https://phabricator.wikimedia.org/T283093 (10Marostegui) [10:43:28] marostegui: half is not that bad, I wish I could clean it more :( [10:43:37] It is pretty good! [11:19:19] 10DBA: Please optimize logging table in ruwiki - https://phabricator.wikimedia.org/T286102 (10Marostegui) [11:19:45] 10DBA: Please optimize logging table in ruwiki - https://phabricator.wikimedia.org/T286102 (10Marostegui) 05Open→03Stalled a:03Marostegui eqiad is done, codfw will be done once we switch back. [11:30:31] 10Blocked-on-schema-change, 10DBA: Schema change for renaming several indexes in change_tag table - https://phabricator.wikimedia.org/T284619 (10Marostegui) [11:34:42] 10DBA: Rename dbstore1004 to db1183 and place it on m5 - https://phabricator.wikimedia.org/T284622 (10Marostegui) I have removed dbstore1004:331* from zarcillo and tendril. [12:10:22] 10Blocked-on-schema-change, 10DBA: Schema change for renaming several indexes in change_tag table - https://phabricator.wikimedia.org/T284619 (10Marostegui) [12:25:57] 10Blocked-on-schema-change, 10DBA: Schema change for renaming several indexes in change_tag table - https://phabricator.wikimedia.org/T284619 (10Marostegui) [12:27:10] 10Blocked-on-schema-change, 10DBA: Schema change for making cuc_id in cu_changes unsigned - https://phabricator.wikimedia.org/T283093 (10Marostegui) [12:28:06] 10Blocked-on-schema-change, 10DBA: Schema change for making cuc_id in cu_changes unsigned - https://phabricator.wikimedia.org/T283093 (10Marostegui) [13:25:03] 10DBA: Move db1124 and db1125 to misc services temporarily - https://phabricator.wikimedia.org/T286042 (10Marostegui) [13:25:38] 10DBA: Move db1124 and db1125 to misc services temporarily - https://phabricator.wikimedia.org/T286042 (10Marostegui) db1125 is now serving in m1. Pending: place db1125 into haproxy (let's give it 24h to make sure it runs smoothly) [15:09:04] 10DBA, 10Infrastructure-Foundations, 10SRE, 10netops, 10cloud-services-team (Kanban): Switch buffer re-partition - Eqiad Row A - https://phabricator.wikimedia.org/T286032 (10wkandek)