[01:09:48] PROBLEM - MariaDB sustained replica lag on m1 on db1117 is CRITICAL: 11.6 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1117&var-port=13321 [01:09:52] PROBLEM - MariaDB sustained replica lag on m1 on db2132 is CRITICAL: 11.8 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2132&var-port=9104 [01:09:54] PROBLEM - MariaDB sustained replica lag on m1 on db2160 is CRITICAL: 14.4 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2160&var-port=13321 [01:11:26] RECOVERY - MariaDB sustained replica lag on m1 on db1117 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1117&var-port=13321 [01:11:28] RECOVERY - MariaDB sustained replica lag on m1 on db2132 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2132&var-port=9104 [01:11:30] RECOVERY - MariaDB sustained replica lag on m1 on db2160 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2160&var-port=13321 [07:39:06] A.mir1: thanks for your work on thumbs yesterday; your CR looks plausible to me, but I don't know the PHP internals anything like well enough to offer a +1, sorry [07:39:58] Emperor: the truth is... nobody does! :-D [07:54:21] 😿 [09:28:59] Emperor: I actually need to double check something with you. This changes the list of thumbs being returned, so if 200px exists in eqiad only, it still tries to delete them in both eqiad and codfw, would that cause issues? [09:29:34] and yeah, getting this reviewed and merged is going to be ... interesting [09:30:32] Amir1: the codfw deletion would say 404, but I think that's OK [09:31:12] * Emperor needs to have a discussion with the rclone author on how clients should handle 404 from DELETE, but that's a side-note [09:31:29] let me test the patch in mwdebug [09:31:41] 👍 [10:32:32] Emperor: tested it, didn't work, debugged it, made some changes and tested again, it works [10:34:15] \o/ [10:49:16] marostegui: how to handle decom of old hosts for T326669, separate task, something else? [10:49:16] T326669: Productionize db1206-db1225 - https://phabricator.wikimedia.org/T326669 [10:49:46] jynus: yeah, basically: https://phabricator.wikimedia.org/maniphest/task/edit/form/52/ [10:49:56] that's it [10:50:00] ok [10:50:27] will file one for db1102 when I make sure the new one is working well [10:50:27] jynus: and if you can, make it a subtask of https://phabricator.wikimedia.org/T326683 [10:50:35] 👍 [10:50:57] Ah, if it is for db1102, no need to make it a subtask of that one [10:51:00] As they aren't related [10:51:03] oh [10:51:04] So just the decom task [10:51:37] * Emperor now knows more about PHP array syntax than they did earlier... [11:03:34] jynus: i just disabled prometheus-mysqld-exporter.service on db1225 [11:03:39] and ran a reset-failed [11:03:53] it was still being setup [11:04:13] thanks