[10:15:59] !incidents [10:16:00] 4464 (UNACKED) Host db2118 (paged) - PING - Packet loss = 100% [10:16:00] 4465 (UNACKED) db2159 (paged)/MariaDB Replica IO: s7 (paged) [10:16:00] 4466 (UNACKED) db2150 (paged)/MariaDB Replica IO: s7 (paged) [10:16:00] 4467 (UNACKED) db2182 (paged)/MariaDB Replica IO: s7 (paged) [10:16:01] 4468 (UNACKED) db2168 (paged)/MariaDB Replica IO: s7 (paged) [10:16:01] 4469 (UNACKED) db2120 (paged)/MariaDB Replica IO: s7 (paged) [10:16:01] 4470 (UNACKED) db2122 (paged)/MariaDB Replica IO: s7 (paged) [10:16:01] 4471 (UNACKED) db2108 (paged)/MariaDB Replica IO: s7 (paged) [10:16:01] 4472 (UNACKED) db2121 (paged)/MariaDB Replica IO: s7 (paged) [10:16:02] 4473 (UNACKED) db2159 (paged)/MariaDB Replica Lag: s7 (paged) [10:16:02] 4474 (UNACKED) db2182 (paged)/MariaDB Replica Lag: s7 (paged) [10:16:03] 4475 (UNACKED) db2150 (paged)/MariaDB Replica Lag: s7 (paged) [10:17:59] I just acked them [10:18:41] db2118 is s7 master [10:18:55] and has crashed, so will be read-only, which I think is "ring a DBA" time [10:19:06] https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Master_comes_back_in_read_only [10:19:10] Emperor: already done by sobanski [10:19:11] Emperor: yes, see -operations [10:19:16] Manuel is on his way [10:19:36] ack, thanks [10:21:37] !incidents [10:21:37] 4464 (ACKED) Host db2118 (paged) - PING - Packet loss = 100% [10:21:37] 4465 (ACKED) db2159 (paged)/MariaDB Replica IO: s7 (paged) [10:21:38] 4467 (ACKED) db2182 (paged)/MariaDB Replica IO: s7 (paged) [10:21:38] 4468 (ACKED) db2168 (paged)/MariaDB Replica IO: s7 (paged) [10:21:38] 4469 (ACKED) db2120 (paged)/MariaDB Replica IO: s7 (paged) [10:21:38] 4471 (ACKED) db2108 (paged)/MariaDB Replica IO: s7 (paged) [10:21:38] 4472 (ACKED) db2121 (paged)/MariaDB Replica IO: s7 (paged) [10:21:39] 4473 (ACKED) db2159 (paged)/MariaDB Replica Lag: s7 (paged) [10:21:39] 4474 (ACKED) db2182 (paged)/MariaDB Replica Lag: s7 (paged) [10:21:39] 4475 (ACKED) db2150 (paged)/MariaDB Replica Lag: s7 (paged) [10:21:40] 4476 (ACKED) db2168 (paged)/MariaDB Replica Lag: s7 (paged) [10:21:40] 4477 (ACKED) db2122 (paged)/MariaDB Replica Lag: s7 (paged) [10:21:41] 4478 (ACKED) db2120 (paged)/MariaDB Replica Lag: s7 (paged) [10:21:41] 4479 (ACKED) db2108 (paged)/MariaDB Replica Lag: s7 (paged)