[05:16:28] The script deleted 250M rows of flaggedtemplates on Polish Wikipedia only, it deleted 1 million rows in a wiki with 50k pages :/ [07:47:07] marostegui: does the spoofuser depool? [07:47:26] yes [07:47:28] ah, I forget, please go ahead, tempaltelinks is done there [07:47:34] yeah, I checked the map :) [07:47:36] and s4 is clean [07:48:36] yup [07:56:17] Amir1: https://phabricator.wikimedia.org/T312865#8164345 1021 just caught up [07:56:22] So I am going to sanitize it [07:56:28] Thanks [07:57:00] sigh, the maintain views needs depooling for s5 [08:01:32] and double sigh, I need to run the templatelinks schema changes on the new wikis as well [08:02:11] Amir1: done with the sanitization and all that jazz [08:02:31] Thanks! [08:09:23] Just promoted pc1013 as pc2 master [08:09:47] pc3 [08:10:43] wohoo [08:11:02] misc masters left [08:11:51] and external store [08:14:06] ah yeah [08:36:55] sigh, another day, another swift disk failure [09:56:16] (PrometheusMysqldExporterFailed) resolved: Prometheus-mysqld-exporter failed (db2179:9104) - https://grafana.wikimedia.org/d/000000278/mysql-aggregated - https://alerts.wikimedia.org/?q=alertname%3DPrometheusMysqldExporterFailed [10:39:56] jynus: you said some time ago you had some hosts you might be giving "us" back? [10:40:05] Or did I dreamed that? [10:40:06] yeah [10:40:12] the mediabackupstemp [10:40:16] we bought the replacement [10:40:19] ah cool [10:40:22] you have an ETA for that? [10:40:49] sadly, I depend on vendods + dcops, it is not blocked on me ATM [10:40:57] Ah ok [10:40:58] I can give you an estimation [10:40:58] No problem [10:41:04] Sure, that helops [10:41:05] helps [10:41:06] which is mid-next Q? [10:41:16] cool, that helps thanks! [10:42:13] no, thank you for sharing those- this made us realized I was too optimistic about resources needed [10:42:27] mediabackups won't "fit" into m1 at all [12:01:16] (PrometheusMysqldExporterFailed) firing: Prometheus-mysqld-exporter failed (db2180:9104) - https://grafana.wikimedia.org/d/000000278/mysql-aggregated - https://alerts.wikimedia.org/?q=alertname%3DPrometheusMysqldExporterFailed [12:01:24] ^ me [12:56:16] (PrometheusMysqldExporterFailed) resolved: Prometheus-mysqld-exporter failed (db2180:9104) - https://grafana.wikimedia.org/d/000000278/mysql-aggregated - https://alerts.wikimedia.org/?q=alertname%3DPrometheusMysqldExporterFailed