[10:11:16] <arturo>	 dcaro: thanks for being helpful and supporting the Toolforge user community 🥰
[10:15:32] <arturo>	 https://devops.com/consortium-driving-openstack-to-become-arm-of-the-linux-foundation/ <-- openstack joins the linux foundation
[10:26:16] <dcaro>	 ty!
[11:32:00] * dcaro lunch
[14:27:47] * arturo manages to get the tofu-infra refactor into a codfw1dev NOOP
[14:29:11] <andrewbogott>	 arturo: I just 'cleaned up' the tofu-infa repo on cloudcontrol2004-dev thinking it was just git misbehavior. Did I wipe out your work in progress?
[14:36:26] <arturo>	 that's OK, the changes are on gitlab
[14:37:07] <arturo>	 I was cherry picking from there
[14:37:08] <andrewbogott>	 that's good! Sorry for the toe-stepping
[16:29:53] <dcaro>	 apache is failing to load the config for tools-prometheus-* looking
[16:30:08] <dcaro>	 it surfaced when tools-prometheus-6 was rebooted
[16:31:32] <arturo>	 maybe some puppet change?
[16:31:48] <arturo>	 I was able to ssh to the VM after the reboot
[16:32:14] <arturo>	 and even checked the logs, but found nothing interesting beyond a bunch of apparmor logs
[16:32:17] <dcaro>	 probably, though there's no logs of it (the journal is not long enough)
[16:32:41] <dcaro>	 mod_ssl is not enabled
[16:34:29] <dcaro>	 hmm... manually enabled it and now it works, and puppet does not remove it :/
[16:40:07] <arturo>	  so it was manually removed?
[16:40:41] <arturo>	 was there a deb package upgrade or something?
[16:42:04] <dcaro>	 I was thinking the same
[16:43:25] <dcaro>	 fix: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1128909
[16:44:22] <arturo>	 LGTM
[16:44:39] <dcaro>	 it's been a while from the logs :/ (last apache2 upgrade was in december), but if we never restarted it it might not have noticed
[16:55:09] <dcaro>	 hmm... it needs https://gerrit.wikimedia.org/r/c/operations/puppet/+/1128912 also to fix some tests :/
[16:57:24] <arturo>	 +1'd
[16:59:38] <dcaro>	 thanks, merged
[16:59:48] <andrewbogott>	 I see a keystone authorization that fails every hour, on the hour. Does that sound familiar to anyone?  Unfortunately it doesn't log the creds that are failing
[17:01:34] <dcaro>	 I saw it the other day too, but I was looking for something else that crashed and forgot 🤦‍♂️
[17:01:52] <dcaro>	 my guess is opentofu-infra?
[17:01:58] <dcaro>	 (it had a cron or something?)
[17:02:23] <andrewbogott>	 hm, I'd expect us to see that failure elsewhere
[17:02:24] <arturo>	 tofu-infra creds are up to date as far as I can tell
[17:03:40] <dcaro>	 iirc the logs I saw was complaining about ec2 creds, is that relevant? (or it considers ec2 creds all creds)
[17:07:04] <andrewbogott>	 I'm going to hack the ldap code to log the bind name on failure
[17:14:12] <dcaro>	 that'd help
[17:17:54] * arturo offline
[18:04:09] <andrewbogott>	 it's 'traffic-cloud-dns-manager' -- likely something in acme-chief
[18:14:17] * dcaro off
[18:14:19] <dcaro>	 cya tomorrow!