[07:24:49] may soon be new ADSL router time :-/ [08:36:47] Emperor: re: fstab entries, yeah we've had that issue in the past too heh, I thought we fixed it as some point tho [08:56:47] I'm planning to continue with this `maintain_views` on the clouddb hosts today. T313281 - So far clouddb10[13-16] are effectively depooled. [08:56:48] T313281: pagetriage-copyvio log entries don't appear in the replica databases - https://phabricator.wikimedia.org/T313281 [09:07:11] The command I'm looking to run in each of clouddb10[13-16] is `sudo maintain-views --table pagetriage-copyvio --all-databases --replace-all` [09:08:44] Is anyone able to confirm that this looks right before I do please? [09:52:21] btullis: the table is very likely wrong [09:52:40] Ah table must be logging [09:52:46] Not pagetriage [09:56:15] Ah, thanks so much. I couldn't really work it out from looking a the ticket or the maintain-views script - and these updates have only recently become the responsibility of our team. [09:59:21] So, just to confirm: `sudo maintain-views --table logging --all-databases --replace-all` [09:59:23] Right? [10:08:15] Yup [10:37:43] Great, that appears to have worked, so I'll repool the 'web' wikireplica servers, depool the 'analytics' wikireplica servers, then repeat on clouddb10[17-20] [15:13:02] PROBLEM - MariaDB sustained replica lag on s2 on db2170 is CRITICAL: 2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2170&var-port=13312 [15:24:40] PROBLEM - MariaDB sustained replica lag on s2 on db2170 is CRITICAL: 2.2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2170&var-port=13312 [15:48:20] PROBLEM - MariaDB sustained replica lag on s2 on db2170 is CRITICAL: 2.6 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2170&var-port=13312 [15:56:42] PROBLEM - MariaDB sustained replica lag on s2 on db2170 is CRITICAL: 2.8 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2170&var-port=13312 [16:01:12] do you know what is going on with s2? https://grafana.wikimedia.org/goto/_WV2hEi4z?orgId=1 [16:12:45] PROBLEM - MariaDB sustained replica lag on s2 on db2170 is CRITICAL: 2.2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2170&var-port=13312 [16:15:49] PROBLEM - MariaDB sustained replica lag on s2 on db2170 is CRITICAL: 2.8 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2170&var-port=13312 [16:21:09] RECOVERY - MariaDB sustained replica lag on s2 on db2170 is OK: (C)2 ge (W)1 ge 0.4 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2170&var-port=13312 [16:38:29] PROBLEM - MariaDB sustained replica lag on s2 on db2170 is CRITICAL: 2.8 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2170&var-port=13312 [16:40:49] RECOVERY - MariaDB sustained replica lag on s2 on db2170 is OK: (C)2 ge (W)1 ge 0.8 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2170&var-port=13312 [22:12:02] sigh [22:34:35] downtimed it for a bit