[00:26:22] 10Mail, 06collaboration-services, 06Infrastructure-Foundations, 10VPS-project-Phabricator: The test Phabricator instance doesn't seem to be successfully sending emails to @wikimedia.org addresses - https://phabricator.wikimedia.org/T422559#11797249 (10Peachey88) [05:52:40] moritzm: fyi I've just deployed https://gerrit.wikimedia.org/r/1265453 [06:09:46] thanks, I'll enable the role next and then I'll bootstrap the esqin02 cluster [06:35:05] good morning, I'd like to know which Debian version the production cumin server is using (Bookworm, Trixie)? I will rebuild a Cumin server on the CI WMCS project and I think it is nice if the Debian version is aligned with production ;) [06:35:46] maybe Trixie would make sense, not sure whether it is supported already [06:45:10] we currently use Bookworm, but the update to Trixis is close, we have a server refresh pending and we'll use that to move to trixie next [06:45:43] if you strictly only need/use Cumin, then my recommendation would be to directly start with trixie [06:46:27] if you also want the full range of cookbooks/spicerack, then maybe initiall go with Bookworm since some of the longer tail of Python deps still needs to be investigated for compat [06:46:57] also, cumin2002 is on Cumin 6 as of yesterday, so you can also directly start with the best Cumin release ever [07:37:12] volans: https://phabricator.wikimedia.org/T422115 the one liner where do you run it? dns hosts? [07:37:31] I'm adding a bunch of IPs that will require some includes [07:37:36] XioNoX: I run it locally where I have both repos checked out in the same common directory [07:38:05] ok [07:53:50] 10CAS-SSO, 06Infrastructure-Foundations, 13Patch-For-Review: CAS login page overflows on iOS Safari (iPhone 16e) - https://phabricator.wikimedia.org/T422203#11797873 (10SLyngshede-WMF) 05Open→03In progress [07:53:56] 10CAS-SSO, 06Infrastructure-Foundations, 13Patch-For-Review: CAS login page overflows on iOS Safari (iPhone 16e) - https://phabricator.wikimedia.org/T422203#11797874 (10SLyngshede-WMF) a:03SLyngshede-WMF [08:10:25] FIRING: SystemdUnitFailed: prometheus-ganeti-exporter.service on ganeti5007:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:30:55] FIRING: [3x] SystemdUnitFailed: kube-scheduler.service on aux-k8s-ctrl2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:31:32] moritzm: thank you! I will rebuild my cumin instance with Trixie. Volans told me about cumin v6 I will see what happens :) [08:33:19] does anyone who love pain could review https://gerrit.wikimedia.org/r/c/operations/dns/+/1268899 (moritzm as it's related to routed Ganeti?) [08:34:43] hashar: yeah given it's only using cumin::master role IIRC that should be fine, let us know if you encounter any problem. The trixie pacakge of cumin is already in trixie-wikimedia, or you get 5.1.1 from upstream debian :) [08:35:27] and soon 6.0 in sid as well! [08:35:33] XioNoX: I'll have a look in ~ 5m [08:38:37] moritzm: also we should try https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/1117554 with the new move to routed ganeti [08:42:08] wouldn't that be a NOP for routed ganeti nodes? [08:44:53] moritzm: what I need to test is that the netbox script is properly ran after add-node [08:45:18] whatever the script does is not that important for the change itself [08:52:23] ah, I'll run it via test-cookbook when we've added ganeti5006 [08:59:15] great, thanks [09:39:41] volans: thanks, I will let you know how the Cumin rebuilt worked :) [10:55:55] RESOLVED: SystemdUnitFailed: prometheus-ganeti-exporter.service on ganeti5007:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [11:18:14] 10Mail, 06collaboration-services, 06Infrastructure-Foundations, 10VPS-project-Phabricator: @wikimedia.org email addresses don't seem to be receiving emails sent by the test Phabricator instance - https://phabricator.wikimedia.org/T422559#11798466 (10A_smart_kitten) [11:44:15] 10Mail, 06collaboration-services, 06Infrastructure-Foundations, 10VPS-project-Phabricator: @wikimedia.org email addresses don't seem to be receiving emails sent by the test Phabricator instance - https://phabricator.wikimedia.org/T422559#11798615 (10A_smart_kitten) Yeah, I guess it seems like this might po... [13:46:33] wherever someone has time, I would appreciate some feedback in https://phabricator.wikimedia.org/T419976 [16:14:53] 10netops, 06Infrastructure-Foundations, 06SRE: cr1-esams failed upgrade - https://phabricator.wikimedia.org/T422525#11800433 (10cmooney) Ok Juniper came back with the following: ` I found that your version 23.4R2-S7.4 is hitting the PR1933049. Unfortunately, this is a confidential PR, but in order to get thi... [19:11:36] 10Mail, 06Infrastructure-Foundations, 10Phabricator, 06SRE: Replace Exim on phabricator servers with Postfix - https://phabricator.wikimedia.org/T378029#11801471 (10A_smart_kitten) [19:12:57] 10Mail, 06Infrastructure-Foundations, 10Wikimedia-Mailing-lists, 07Upstream: lists.wikimedia.org - adhere to RFC8048 (one-click unsubscribe) dkim guidelines - https://phabricator.wikimedia.org/T355802#11801488 (10A_smart_kitten) [19:13:27] 10Mail, 06collaboration-services, 06Infrastructure-Foundations, 10Phabricator, 06SRE: Replace Exim on phabricator servers with Postfix - https://phabricator.wikimedia.org/T378029#11801490 (10Dzahn)