[03:11:04] <jinxer-wm>	 FIRING: NetboxAccounting: Netbox - Accounting job failed - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxAccounting
[04:11:04] <jinxer-wm>	 RESOLVED: NetboxAccounting: Netbox - Accounting job failed - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxAccounting
[15:05:55] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: Cloudcephosd: migrate to single network uplink - https://phabricator.wikimedia.org/T399180#11057922 (10Andrew) 05Open→03Stalled
[16:16:20] <elukey>	 jhathaway: o/ while testing the cacheproxy-efi.cfg recipe for the new cp2xxx nodes I started to wonder about the EFI partition, since IIUC they want it to be replicated on a MD array (no idea if it is possible or not). I checked what we currently do and the partition that is not mounted/installed by d-i as /boot/efi is completely empty, so in theory if the underlying ssd fails we end up with an unbootable server. Is it the right 
[16:16:20] <elukey>	 understanding?
[16:16:55] <jinxer-wm>	 FIRING: MaxConntrack: Max conntrack at 82.35% on krb1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack
[16:18:05] <elukey>	 better sorry - I assumed that having `d-i	grub-installer/bootdev` in the recipe would help to mirror the /boot/efi partition, but of course it is modified every now and then during the lifetime of a debian os
[16:18:35] <jhathaway>	 yup, you are correct, I have a proof of concept patch out for discussion, https://phabricator.wikimedia.org/T376949#10252274
[16:20:51] <elukey>	 ahhhh all right nice! I think Filippo may be interested as well, thanks for the pointer!
[16:21:55] <jinxer-wm>	 RESOLVED: MaxConntrack: Max conntrack at 82.35% on krb1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack
[16:25:45] <elukey>	 filed https://gerrit.wikimedia.org/r/c/operations/puppet/+/1175549 as well
[19:33:55] <jinxer-wm>	 FIRING: MaxConntrack: Max conntrack at 83.4% on krb1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack
[19:38:55] <jinxer-wm>	 RESOLVED: MaxConntrack: Max conntrack at 82.48% on krb1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack
[20:34:37] <wikibugs>	 07Puppet, 10Beta-Cluster-Infrastructure, 06Infrastructure-Foundations, 13Patch-For-Review: Puppet configures kernel.core_pattern |/usr/lib/systemd/systemd-coredump, but systemd-coredump is not installed - https://phabricator.wikimedia.org/T400247#11059070 (10bd808) a:03Lucas_Werkmeister_WMDE
[22:39:25] <jinxer-wm>	 FIRING: MirrorHighLag: Mirrors - /srv/mirrors/debian synchronization lag - https://wikitech.wikimedia.org/wiki/Mirrors - https://grafana.wikimedia.org/d/dbd8a904-eab2-48d1-a3b9-fa1851ef3ed2/mirrors?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DMirrorHighLag
[23:06:17] <wikibugs>	 10Mail, 06Infrastructure-Foundations, 10MediaWiki-Email, 10MediaWiki-extensions-EmailAuth, and 3 others: Could not send confirmation email: Unknown error in PHP's mail() function. - https://phabricator.wikimedia.org/T383047#11059422 (10Scott_French) I've backported @jhathaway's upstream fixes onto PHP 8.1....