[01:49:01] <jinxer-wm>	 FIRING: SystemdUnitFailed: wmf_auto_restart_krb5-admin-server.service on krb2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[05:49:01] <jinxer-wm>	 FIRING: SystemdUnitFailed: wmf_auto_restart_krb5-admin-server.service on krb2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[09:12:48] <jinxer-wm>	 FIRING: PuppetFailure: Puppet has failed on netflow1003:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure
[09:14:48] <jinxer-wm>	 FIRING: PuppetFailure: Puppet has failed on cumin2003:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure
[09:17:48] <jinxer-wm>	 FIRING: [4x] PuppetFailure: Puppet has failed on netflow1003:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure
[09:22:48] <jinxer-wm>	 FIRING: [7x] PuppetFailure: Puppet has failed on netflow1002:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure
[09:27:48] <jinxer-wm>	 FIRING: [9x] PuppetFailure: Puppet has failed on netflow1002:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure
[09:32:48] <jinxer-wm>	 FIRING: [11x] PuppetFailure: Puppet has failed on netflow1002:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure
[09:34:48] <jinxer-wm>	 FIRING: [2x] PuppetFailure: Puppet has failed on cumin1004:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure
[09:37:48] <jinxer-wm>	 FIRING: [13x] PuppetFailure: Puppet has failed on netflow1002:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure
[09:40:34] <XioNoX>	 elukey: PKI down again?
[09:40:39] <topranks>	 XioNoX: yeah seems to be 
[09:40:56] <topranks>	 netflow* hosts failing to download the network cert bundle from them when puppet runs 
[09:41:38] <elukey>	 yeah I am changing the pki discovery to an lvs endpoint
[09:41:40] <elukey>	 Could not evaluate: Could not retrieve file metadata for http://pki.discovery.wmnet/bundles/network_devices.pem:
[09:42:03] <elukey>	 and of course it is not working
[09:42:17] <elukey>	 we can hardcode pki1002 or similar in puppet, and then resolve for the moment
[09:42:29] <topranks>	 elukey: ah ok thanks 
[09:42:55] <topranks>	 the failures aren't a big deal if it will only be temporary I think 
[09:49:01] <jinxer-wm>	 FIRING: SystemdUnitFailed: wmf_auto_restart_krb5-admin-server.service on krb2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[10:03:47] <elukey>	 topranks, XioNoX - https://gerrit.wikimedia.org/r/c/operations/puppet/+/1346494 :)
[10:04:05] <elukey>	 since the lvs service is going to wait I am temporary patching the check
[10:04:17] <elukey>	 and using https, since now we don't have the mTLS requirement anymore
[10:04:52] <topranks>	 elukey: thanks!  +1 on the patch 
[10:05:30] <XioNoX>	 elukey: how temporary is it? Should it be a variable, and should you also update profile::netops::netdev_intermediate_ca ?
[10:07:11] <elukey>	 XioNoX: $days, until we have nftables support for ipip. I can also fix that, one thing that doesn't add up though is https://pki1003.eqiad.wmnet/bundles/network_devices.pem returning 404
[10:07:39] <elukey>	 ah wow on http works
[10:07:42] <elukey>	 oh my
[10:07:48] <elukey>	 it must be a special httpd config
[10:07:54] * elukey cries in a corner
[10:07:59] <elukey>	 ok filing another patch
[10:11:15] <elukey>	 XioNoX, topranks - second part https://gerrit.wikimedia.org/r/c/operations/puppet/+/1346497
[10:11:56] <elukey>	 it is the best that we can do, after pki.discovery is available again I'll also update the httpd config to allow that fetch over https
[10:12:42] <XioNoX>	 elukey: thx, lgtm, I just worry that we forget to set it back to the proper value once its all done
[10:13:16] <elukey>	 XioNoX: dooon't worry I have a long list of cleanup things :D
[10:15:28] <XioNoX>	 thx!
[10:24:48] <jinxer-wm>	 FIRING: [2x] PuppetFailure: Puppet has failed on cumin1004:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure
[10:27:48] <jinxer-wm>	 FIRING: [13x] PuppetFailure: Puppet has failed on netflow1002:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure
[10:32:48] <jinxer-wm>	 FIRING: [13x] PuppetFailure: Puppet has failed on netflow1002:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure
[10:37:48] <jinxer-wm>	 FIRING: [13x] PuppetFailure: Puppet has failed on netflow1002:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure
[10:42:48] <jinxer-wm>	 FIRING: [12x] PuppetFailure: Puppet has failed on netflow1002:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure
[10:47:48] <jinxer-wm>	 FIRING: [10x] PuppetFailure: Puppet has failed on netflow1002:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure
[10:57:48] <jinxer-wm>	 RESOLVED: [3x] PuppetFailure: Puppet has failed on netflow2004:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure
[11:14:48] <jinxer-wm>	 RESOLVED: PuppetFailure: Puppet has failed on cumin1004:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure
[13:28:43] <wikibugs>	 10netops, 06cloud-services-team, 10Horizon, 06Infrastructure-Foundations, and 3 others: Move cloudweb hosts to cloud racks? - https://phabricator.wikimedia.org/T411783#12379579 (10taavi) 05Open→03Declined Per the network sync meeting, we will be doing {T392478} instead.
[13:49:01] <jinxer-wm>	 FIRING: SystemdUnitFailed: wmf_auto_restart_krb5-admin-server.service on krb2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[14:17:24] <wikibugs>	 10CFSSL-PKI, 06Infrastructure-Foundations, 13Patch-For-Review: Add a load balanced service in front of PKI - https://phabricator.wikimedia.org/T436809#12379983 (10elukey) Current status:  * mTLS has been disabled for port 443. Clients (puppet execs) need to be updated as well, but this will generate a lot of...