[06:30:29] I am going to start a schema change on s4 codfw master on categorylinks, it is likely to take maybe 2 days or so, and then it will need to replicate to all the slaves [08:29:12] moritzm: should we do the firewall merge? [08:54:39] sure, we can do that now [08:55:25] maybe let's use backup1004 to run iptables -L and I will merge afterwards [08:56:22] let me know when ready [08:58:09] done, I've dumped the current iptables rules on backup1004, going to merge the patch now,ok? [08:58:24] you do it then? [08:58:37] fine either way, you can also do it [08:58:57] ok, then merging, will ping you when finished [08:59:00] ack [09:01:18] moritzm: you can run puppet now [09:02:22] on it [09:04:14] ran puppet on 1004 and the diff between the old and new config is zero, so all good [09:04:20] nice [13:51:17] interesting, categorylinks "only" took 8, I thought it would take a day or two [13:51:36] the table was very fragmented (200G -> 150G) [14:41:48] (PuppetFailure) firing: Puppet has failed on dbprov2005:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure