[08:11:08] You might wanna check db1127, icinga says it rebooted and mysql ain't started right [08:21:43] RhinosF1: thanks for the heads up [08:21:51] It has broken RAM, I'm going to depool it until we can get it fixed? [08:22:04] multi-bit memory errors on DIMM_A3 [08:22:21] moritzm: thank you, that'd be great [08:22:22] started to log yesterday, if we restart it, I'm sure it'll trip over the weekend [08:22:34] i can also take it over [08:23:05] sure, go ahead :-) I'm going to open a DC ops task in the mean time since I already have the SEL logs, ok? [08:27:21] perfect, thanks! [08:28:33] https://phabricator.wikimedia.org/T286763 [08:29:12] and added a note https://phabricator.wikimedia.org/T253810, which would be helpful to catch these before they lead to a reboot [08:48:33] moritzm: 👍 [08:57:57] Ty kormat moritzm [09:27:17] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, 10netops: Switch buffer re-partition - Eqiad Row C - https://phabricator.wikimedia.org/T286065 (10cmooney) [10:09:18] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, and 2 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10cmooney) [10:10:17] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, 10netops: Switch buffer re-partition - Eqiad Row C - https://phabricator.wikimedia.org/T286065 (10cmooney) [10:37:52] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, and 2 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10cmooney) [11:06:06] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, and 2 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10cmooney) [11:06:35] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, 10netops: Switch buffer re-partition - Eqiad Row C - https://phabricator.wikimedia.org/T286065 (10cmooney) [11:13:51] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, and 2 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10cmooney) [11:14:11] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, 10netops: Switch buffer re-partition - Eqiad Row C - https://phabricator.wikimedia.org/T286065 (10cmooney) [11:16:42] 10DBA, 10Infrastructure-Foundations, 10SRE, 10netops: Switch buffer re-partition - Eqiad Row B - https://phabricator.wikimedia.org/T286061 (10cmooney) [11:17:10] 10DBA, 10Infrastructure-Foundations, 10SRE, 10Traffic, and 2 others: Switch buffer re-partition - Eqiad Row A - https://phabricator.wikimedia.org/T286032 (10cmooney) [11:41:52] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, and 2 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10cmooney) [11:45:53] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, 10netops: Switch buffer re-partition - Eqiad Row C - https://phabricator.wikimedia.org/T286065 (10cmooney) [12:14:18] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, and 2 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10cmooney) [12:14:36] 10DBA, 10Infrastructure-Foundations, 10SRE, 10netops: Switch buffer re-partition - Eqiad Row B - https://phabricator.wikimedia.org/T286061 (10cmooney) [12:35:39] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, and 2 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10cmooney) [12:36:05] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, 10netops: Switch buffer re-partition - Eqiad Row C - https://phabricator.wikimedia.org/T286065 (10cmooney) [12:37:51] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, 10netops: Switch buffer re-partition - Eqiad Row C - https://phabricator.wikimedia.org/T286065 (10cmooney) [12:39:12] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, 10netops: Switch buffer re-partition - Eqiad Row C - https://phabricator.wikimedia.org/T286065 (10cmooney) [12:43:20] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, and 2 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10cmooney) [12:44:23] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, and 2 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10cmooney) [12:47:49] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, 10netops: Switch buffer re-partition - Eqiad Row C - https://phabricator.wikimedia.org/T286065 (10cmooney) [13:43:07] 10Blocked-on-schema-change, 10DBA, 10AbuseFilter: Rename AbuseFilter indexes for consistency - https://phabricator.wikimedia.org/T281058 (10Kormat) a:03Kormat Schema change made on s6 eqiad replica (db1096:3316). Will leave to soak over the weekend. [16:10:55] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, and 2 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10cmooney) [16:15:29] 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, 10netops: Switch buffer re-partition - Eqiad Row C - https://phabricator.wikimedia.org/T286065 (10cmooney) [18:09:12] 10DBA, 10SRE, 10ops-eqiad: Upgrade db1104 firmware - https://phabricator.wikimedia.org/T286226 (10wiki_willy) a:05Cmjohnson→03Jclark-ctr It looks like Chris is going to be out for a while. Moving this task over to @Jclark-ctr, who should be back Tuesday or Wednesday. Thanks, Willy