[06:40:01] 10Data-Persistence-Backup, 10serviceops, 10GitLab (Initialization), 10Patch-For-Review, 10User-brennen: Backups for GitLab - https://phabricator.wikimedia.org/T274463 (10Sergey.Trofimovsky.SF) Following on a previous discussion, noting some concerns that current implementation (copying over the latest ba... [07:12:09] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, and 3 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10MoritzMuehlenhoff) [09:15:19] 10DBA: db1170 mysql process crashed - https://phabricator.wikimedia.org/T286888 (10Kormat) The issue is bad ram: ` [Sun Jul 18 10:23:58 2021] {1}[Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 4 [Sun Jul 18 10:23:58 2021] {1}[Hardware Error]: It has been corrected by h/w and requires n... [09:18:58] 10DBA, 10DC-Ops, 10ops-eqiad: db1170 mysql process crashed - https://phabricator.wikimedia.org/T286888 (10Kormat) a:05Kormat→03Cmjohnson Hi @Cmjohnson, can you get us a new dimm please? Cheers :) [09:43:25] FYI: i broke replication on db1096:s6. working on fixing it now. [09:51:57] fixed 😅 [09:52:52] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, and 3 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10aborrero) [09:54:28] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, and 3 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10aborrero) [13:13:39] 10Blocked-on-schema-change, 10DBA, 10AbuseFilter: Rename AbuseFilter indexes for consistency - https://phabricator.wikimedia.org/T281058 (10Kormat) [13:43:49] 10Blocked-on-schema-change, 10DBA, 10AbuseFilter: Rename AbuseFilter indexes for consistency - https://phabricator.wikimedia.org/T281058 (10Kormat) [13:49:48] 10Blocked-on-schema-change, 10DBA, 10AbuseFilter: Rename AbuseFilter indexes for consistency - https://phabricator.wikimedia.org/T281058 (10Kormat) [13:59:50] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, and 3 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10ArielGlenn) [14:03:05] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, and 3 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10MoritzMuehlenhoff) [14:03:48] 10Blocked-on-schema-change, 10DBA, 10AbuseFilter: Rename AbuseFilter indexes for consistency - https://phabricator.wikimedia.org/T281058 (10Kormat) [14:23:03] Amir1: i merged+deployed https://gerrit.wikimedia.org/r/c/operations/puppet/+/705045. looks like it works fine. 👍 [14:32:54] kormat: awesome, thanks. I'll make a patch to drop the crons soon [14:40:50] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, and 3 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10ops-monitoring-bot) Icinga downtime set by vgutierrez@cumin1001 for 1:00:00 4 host(s) and their services with reason: eqiad row D maintenance ` cp[1... [14:43:11] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, and 3 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10Vgutierrez) [14:48:34] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, and 3 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10ops-monitoring-bot) Icinga downtime set by vgutierrez@cumin1001 for 1:00:00 1 host(s) and their services with reason: eqiad row D maintenance ` dns1... [14:49:34] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, and 3 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10Vgutierrez) [14:51:12] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, and 3 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10ops-monitoring-bot) Icinga downtime set by vgutierrez@cumin1001 for 1:00:00 1 host(s) and their services with reason: eqiad row D maintenance ` lvs1... [14:55:30] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, and 3 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10cmooney) [15:06:00] 10Blocked-on-schema-change, 10DBA, 10AbuseFilter: Rename AbuseFilter indexes for consistency - https://phabricator.wikimedia.org/T281058 (10Kormat) [15:06:08] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, and 3 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10cmooney) [15:11:37] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, and 3 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10cmooney) [15:52:15] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, and 3 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10cmooney) All works complete, no signs of any issues really, I had no ping loss on 16 pings towards 2 hosts connected off each member switch. Very h... [16:45:41] kormat, sobanski: http://code.openark.org/blog/mysql/reducing-my-oss-involvement-and-how-it-affects-orchestrator-gh-ost [16:46:46] Thanks, bookmarked for a read later [16:52:08] 10DBA, 10Toolhub, 10User-bd808: Discuss database needs with the DBA team - https://phabricator.wikimedia.org/T271480 (10bd808) >>! In T271480#7220832, @LSobanski wrote: > * QPS: Honestly no idea. This is a green field project that is separate and distinct from the wikis which makes guessing about adoption a... [16:52:48] 10DBA, 10Toolhub, 10User-bd808: Discuss database needs with the DBA team - https://phabricator.wikimedia.org/T271480 (10bd808) [17:08:44] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, and 3 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10Andrew) [17:14:04] Amir1, "Last dump for s4 at eqiad (db1145.eqiad.wmnet:3314) taken on 2021-07-20 00:00:02 is 346 GB, but previous one was 382 GB, a change of 9.5%" [17:53:31] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, and 3 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10Bstorm) [18:06:38] jynus: oh nice, I think it'll be much better in the next week [18:54:27] 10DBA, 10DC-Ops, 10SRE, 10ops-eqiad: db1170 mysql process crashed - https://phabricator.wikimedia.org/T286888 (10wiki_willy) a:05Cmjohnson→03Jclark-ctr Hi @Kormat - Chris is out this week, so moving over to @Jclark-ctr for him to check out this machine. (under warranty thru Nov 2023) Thanks, Willy