[10:24:54] <Emperor>	 s.ukhe: those are two nodes with dead disks that we have no spares for and are being gradually drained prior to decom
[17:53:49] <elukey>	 as FYI the dump-cloud-ip-ranges timer/unit on puppetservers now works fine, it seems that Microsoft fixed their CDN
[22:35:27] <fabfur>	 !incidents
[22:35:27] <sirenbot>	 5586 (UNACKED)  db2212 (paged)/MariaDB Replica SQL: s1 (paged)
[22:35:28] <sirenbot>	 5587 (UNACKED)  db2212 (paged)/MariaDB Replica Lag: s1 (paged)
[22:42:48] <fabfur>	 Amir1: do you think it's worth to depool db2212?
[23:03:03] <fabfur>	 depooled db2212 and downtimed for 72h
[23:03:32] <fabfur>	 followd procedure on https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting#Depooling_a_replica
[23:05:27] <fabfur>	 Amir1: for when you'll read: I've updated the Index corruption doc as described in the "Temporary incident response steps" doc
[23:08:49] <fabfur>	 !incidents
[23:08:49] <sirenbot>	 5586 (UNACKED)  db2212 (paged)/MariaDB Replica SQL: s1 (paged)
[23:08:49] <sirenbot>	 5587 (UNACKED)  db2212 (paged)/MariaDB Replica Lag: s1 (paged)
[23:10:15] <sukhe>	 thanks fabfur
[23:10:27] <sukhe>	 (ack the page too)
[23:11:22] <fabfur>	 can I do that from here correct? 
[23:11:35] <fabfur>	 `!ack <ID>`? 
[23:13:14] <fabfur>	 !ack 5586
[23:13:14] <sirenbot>	 5586 (ACKED)  db2212 (paged)/MariaDB Replica SQL: s1 (paged)
[23:13:20] <fabfur>	 !ack 5587
[23:13:32] <fabfur>	 !incidents
[23:13:32] <sirenbot>	 5586 (ACKED)  db2212 (paged)/MariaDB Replica SQL: s1 (paged)
[23:13:32] <sirenbot>	 5587 (UNACKED)  db2212 (paged)/MariaDB Replica Lag: s1 (paged)
[23:13:37] <fabfur>	 !ack 5587
[23:13:38] <sirenbot>	 5587 (ACKED)  db2212 (paged)/MariaDB Replica Lag: s1 (paged)
[23:13:43] <fabfur>	 !incidents
[23:13:43] <sirenbot>	 5586 (ACKED)  db2212 (paged)/MariaDB Replica SQL: s1 (paged)
[23:13:44] <sirenbot>	 5587 (ACKED)  db2212 (paged)/MariaDB Replica Lag: s1 (paged)
[23:13:49] <fabfur>	 well, going to sleep
[23:13:49] <sirenbot>	 Could not ack the alert. Please check the parameters.
[23:14:09] <fabfur>	 !incidents
[23:14:09] <sirenbot>	 5586 (ACKED)  db2212 (paged)/MariaDB Replica SQL: s1 (paged)
[23:14:09] <sirenbot>	 5587 (ACKED)  db2212 (paged)/MariaDB Replica Lag: s1 (paged)
[23:38:58] <Amir1>	 Thanks. Let me take a look
[23:44:59] <Amir1>	 I started the rebuild for now