[07:46:25] 10DBA: iw_url in interwiki is varbinary(127) in production but blob in code - https://phabricator.wikimedia.org/T277118 (10Kormat) >>! In T277118#7208385, @Kormat wrote: > Deployed to an s6 eqiad replica (db1096:3316). Will leave to soak overnight. Replication still working fine, going for general deployment now. [07:50:21] 10DBA: iw_url in interwiki is varbinary(127) in production but blob in code - https://phabricator.wikimedia.org/T277118 (10Kormat) [09:04:41] 10Data-Persistence-Backup, 10DC-Ops, 10SRE, 10ops-eqiad: hw troubleshooting: system right cp board missing in new host backup1006 - https://phabricator.wikimedia.org/T286625 (10LSobanski) Adding @jcrespo for visibility. [09:13:03] 10DBA: iw_url in interwiki is varbinary(127) in production but blob in code - https://phabricator.wikimedia.org/T277118 (10Kormat) [09:16:56] 10DBA: iw_url in interwiki is varbinary(127) in production but blob in code - https://phabricator.wikimedia.org/T277118 (10Kormat) [09:18:01] 10DBA, 10SRE, 10ops-eqiad: Upgrade db1104 firmware - https://phabricator.wikimedia.org/T286226 (10LSobanski) @RobH Thanks for digging into it. Let's wait until we have people onsite and can crash cart into the host. Who would be the best person to assign this to so it gets scheduled as soon as possible? [09:30:23] 10DBA, 10Infrastructure-Foundations, 10SRE, 10Traffic, and 2 others: Switch buffer re-partition - Eqiad Row A - https://phabricator.wikimedia.org/T286032 (10cmooney) [09:30:46] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, 10netops: Switch buffer re-partition - Eqiad Row C - https://phabricator.wikimedia.org/T286065 (10cmooney) [09:31:34] 10DBA, 10Infrastructure-Foundations, 10SRE, 10netops: Switch buffer re-partition - Eqiad Row B - https://phabricator.wikimedia.org/T286061 (10cmooney) [09:32:10] 10DBA: iw_url in interwiki is varbinary(127) in production but blob in code - https://phabricator.wikimedia.org/T277118 (10Kormat) [10:20:19] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, and 2 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10cmooney) [11:03:49] 10DBA, 10Infrastructure-Foundations, 10SRE, 10netops: Switch buffer re-partition - Eqiad Row B - https://phabricator.wikimedia.org/T286061 (10cmooney) [11:43:00] 10DBA, 10Infrastructure-Foundations, 10SRE, 10netops: Switch buffer re-partition - Eqiad Row B - https://phabricator.wikimedia.org/T286061 (10cmooney) [11:43:57] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, 10netops: Switch buffer re-partition - Eqiad Row C - https://phabricator.wikimedia.org/T286065 (10cmooney) [11:58:10] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, and 2 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10cmooney) [11:59:20] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, 10netops: Switch buffer re-partition - Eqiad Row C - https://phabricator.wikimedia.org/T286065 (10cmooney) [11:59:48] 10DBA, 10Infrastructure-Foundations, 10SRE, 10netops: Switch buffer re-partition - Eqiad Row B - https://phabricator.wikimedia.org/T286061 (10cmooney) [12:04:17] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, and 2 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10cmooney) [12:09:44] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, 10netops: Switch buffer re-partition - Eqiad Row C - https://phabricator.wikimedia.org/T286065 (10cmooney) [12:46:04] 10DBA: iw_url in interwiki is varbinary(127) in production but blob in code - https://phabricator.wikimedia.org/T277118 (10Kormat) [12:53:54] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, and 2 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10cmooney) [12:54:42] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, 10netops: Switch buffer re-partition - Eqiad Row C - https://phabricator.wikimedia.org/T286065 (10cmooney) [13:01:21] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, 10netops: Switch buffer re-partition - Eqiad Row C - https://phabricator.wikimedia.org/T286065 (10cmooney) [13:05:57] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, and 2 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10cmooney) [13:10:19] 10DBA: iw_url in interwiki is varbinary(127) in production but blob in code - https://phabricator.wikimedia.org/T277118 (10Kormat) [13:10:42] 10DBA: DB maintenance work to do while eqiad is passive (June 2021) - https://phabricator.wikimedia.org/T285139 (10Kormat) [13:10:47] 10DBA, 10Datasets-General-or-Unknown, 10Patch-For-Review, 10Sustainability (Incident Followup), and 2 others: Detect object, schema and data drifts between mediawiki HEAD, production masters and replicas - https://phabricator.wikimedia.org/T104459 (10Kormat) [13:10:57] 10DBA: iw_url in interwiki is varbinary(127) in production but blob in code - https://phabricator.wikimedia.org/T277118 (10Kormat) 05Open→03Stalled Stalling this until we switch back to eqiad. [13:18:03] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, and 2 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10cmooney) [13:34:56] 10DBA: ipb_timestamp is varbinary(14) in old wikis while being binary(14) in the code since 2007 - https://phabricator.wikimedia.org/T278619 (10Kormat) [13:35:15] Amir1: can you confirm the ALTER statement i added to ^ is correct, please? [14:28:29] kormat: hi, I check it ASAP [14:32:18] kormat: yup, looks right [14:32:32] Amir1: 🎉 thanks :) [14:51:22] 10DBA: ipb_timestamp is varbinary(14) in old wikis while being binary(14) in the code since 2007 - https://phabricator.wikimedia.org/T278619 (10Kormat) [14:52:16] 10DBA: ipb_timestamp is varbinary(14) in old wikis while being binary(14) in the code since 2007 - https://phabricator.wikimedia.org/T278619 (10Kormat) Deploying to s6 eqiad replica (db1098) to soak overnight. [14:52:46] 10DBA: ipb_timestamp is varbinary(14) in old wikis while being binary(14) in the code since 2007 - https://phabricator.wikimedia.org/T278619 (10Kormat) a:03Kormat [15:06:21] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, and 2 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10cmooney) [15:07:06] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, 10netops: Switch buffer re-partition - Eqiad Row C - https://phabricator.wikimedia.org/T286065 (10cmooney) [15:08:00] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, 10netops: Switch buffer re-partition - Eqiad Row C - https://phabricator.wikimedia.org/T286065 (10cmooney) [15:13:02] 10DBA, 10SRE, 10ops-eqiad: Upgrade db1104 firmware - https://phabricator.wikimedia.org/T286226 (10wiki_willy) a:05LSobanski→03Cmjohnson Moving over to @Cmjohnson, who will be back before John next week. Thanks, Willy [15:38:29] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, and 2 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10JMeybohm) [15:40:18] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, 10netops: Switch buffer re-partition - Eqiad Row C - https://phabricator.wikimedia.org/T286065 (10JMeybohm) [15:41:25] 10DBA, 10Infrastructure-Foundations, 10SRE, 10netops: Switch buffer re-partition - Eqiad Row B - https://phabricator.wikimedia.org/T286061 (10JMeybohm) [15:42:08] 10DBA, 10Infrastructure-Foundations, 10SRE, 10Traffic, and 2 others: Switch buffer re-partition - Eqiad Row A - https://phabricator.wikimedia.org/T286032 (10JMeybohm) [15:48:37] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, 10netops: Switch buffer re-partition - Eqiad Row C - https://phabricator.wikimedia.org/T286065 (10JMeybohm) [15:49:00] 10DBA, 10Infrastructure-Foundations, 10SRE, 10Traffic, and 2 others: Switch buffer re-partition - Eqiad Row A - https://phabricator.wikimedia.org/T286032 (10JMeybohm) [16:53:05] 10DBA, 10SRE, 10ops-eqiad: Upgrade db1104 firmware - https://phabricator.wikimedia.org/T286226 (10RobH) FYI: My understanding is with a crash cart, one can activate the lifecycle controller manually on rebooting the host in POST and then fail backwards to the last version of any firmware pushed. This option... [19:24:18] 10Data-Persistence-Backup, 10SRE, 10Goal, 10Patch-For-Review: Puppetize media backups infrastructure - https://phabricator.wikimedia.org/T276442 (10jcrespo) All hosts have been setup including TLS. For now we are using Puppet's CA and certs, which for an internal service with internal IPs and that should n... [19:25:20] 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Prepare and check storage layer for banwikisource - https://phabricator.wikimedia.org/T284390 (10nskaggs) 05Open→03Resolved a:03nskaggs Confirmed I can query from toolforge [19:25:24] 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Prepare and check storage layer for dagwiki - https://phabricator.wikimedia.org/T284456 (10nskaggs) 05Open→03Resolved a:03nskaggs Confirmed I can query from toolforge [19:25:40] 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Prepare and check storage layer for shiwiki - https://phabricator.wikimedia.org/T284928 (10nskaggs) a:03nskaggs [19:26:52] 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Prepare and check storage layer for jvwikisource - https://phabricator.wikimedia.org/T286245 (10nskaggs)