[02:09:59] (PuppetFailure) firing: Puppet has failed on db1189:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [02:14:59] (PuppetFailure) firing: (2) Puppet has failed on db1186:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [02:34:59] (PuppetFailure) firing: (3) Puppet has failed on db1186:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [02:34:59] (PuppetFailure) firing: Puppet has failed on ms-be1060:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [02:50:59] (PuppetFailure) firing: Puppet has failed on thanos-be1002:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [02:50:59] (PuppetFailure) firing: Puppet has failed on db1235:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [03:05:59] (PuppetFailure) firing: (3) Puppet has failed on db1234:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [03:10:59] (PuppetFailure) firing: (5) Puppet has failed on db1234:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [03:15:59] (PuppetFailure) firing: (7) Puppet has failed on db1234:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [03:20:59] (PuppetFailure) firing: (8) Puppet has failed on db1234:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [04:25:59] (PuppetFailure) firing: (2) Puppet has failed on thanos-be1002:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [05:29:59] (PuppetFailure) firing: (2) Puppet has failed on ms-be1060:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [06:10:59] (PuppetFailure) firing: (3) Puppet has failed on thanos-be1002:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [06:35:14] (PuppetFailure) firing: (3) Puppet has failed on db1186:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [06:49:59] (PuppetFailure) firing: (3) Puppet has failed on ms-be1060:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [07:21:14] (PuppetFailure) firing: (8) Puppet has failed on db1234:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [08:59:59] (PuppetFailure) firing: (3) Puppet has failed on ms-be1060:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [09:00:03] (PuppetFailure) firing: (3) Puppet has failed on db1186:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [09:00:59] (PuppetFailure) resolved: (8) Puppet has failed on db1234:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [09:00:59] (PuppetFailure) firing: (3) Puppet has failed on thanos-be1002:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [09:02:06] Last dump for m1 at codfw (db2160) taken on 2023-10-31 02:59:29 is 54 GiB, but the previous one was 60 GiB, a change of -10.6 % [09:04:17] librenms table [09:04:59] (PuppetFailure) resolved: (3) Puppet has failed on ms-be1060:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [09:05:07] (PuppetFailure) resolved: (3) Puppet has failed on db1186:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [09:05:15] yeah [09:05:59] (PuppetFailure) resolved: (3) Puppet has failed on thanos-be1002:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [09:51:14] running the last schema update on s4 (T343198) [09:51:14] T343198: Add pl_target_id column to pagelinks in production - https://phabricator.wikimedia.org/T343198 [09:51:33] jbond: I want to provide one data point that hopefully will be useful, one of the errors I saw was a verification error towards m1-master. This is weird because this is a cname, and with the old puppet PKI we didn't use tls because it didn't match the puppet host cert (this is why I connect directly without the proxy for some services). So not sure why that errored out, as it shouldn't be using TLS in the first place [09:55:14] jynus: for the main pki system we have verify=false. [09:56:00] we also have a python script that connects to refresh the ocsp data. this is what was failing and was fixed by updating the bundle. i suspect we have disabled cn validation but still need to dig into that [10:00:15] I see, so it actually was using tls [10:01:28] This means I could do the same to enable it on dbbackups and use the proxy [10:02:12] thank you, this didn't help you, but it helped ME! [11:18:43] no problem :) [15:50:34] arnaudb: I'm seeing a diff from you in puppet-merge. Are we gtg? [15:50:50] some config removal for db1131 [15:50:53] brouberol: you anticipated my highlight [15:50:55] please proceed! [15:51:07] 👍