[09:10:26] <blancadesal>	 morning!
[09:10:31] <blancadesal>	 quick review: https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/67
[09:18:58] <dcaro>	 done
[09:21:20] <blancadesal>	 thanks!
[09:38:16] <arturo>	 there is this alert
[09:38:19] <arturo>	 https://usercontent.irccloud-cdn.com/file/yQrAKWzu/image.png
[09:38:26] <arturo>	 which seems like a puppet problem somewhere?
[09:39:45] <arturo>	 slyngs: I think this is related to this patch https://gerrit.wikimedia.org/r/c/operations/puppet/+/1085515
[09:40:05] <arturo>	 can we safely ignore?
[09:41:01] <slyngs>	 Yes, we are removing the old conntrack alerting
[09:41:56] <arturo>	 ok
[09:42:19] <arturo>	 do we get a similar alert based on prometheus instead of icinga? do we need to configure something?
[09:42:36] <slyngs>	 No, same alert, just from Prometheus
[09:42:49] <arturo>	 great, thanks
[09:43:45] <slyngs>	 It's defined here instead: https://gerrit.wikimedia.org/r/plugins/gitiles/operations/alerts/+/refs/heads/master/team-sre/netfilter.yaml
[09:45:24] <arturo>	 slyngs: but if the alerts are team-sre, will we get them with a label `team=wmcs`?
[09:46:08] <slyngs>	 Ah... hmm
[09:49:30] <arturo>	 btw, I'm surprised the yaml merge `<<:` is working. Last time I checked, prometheus config did not support that, see https://github.com/prometheus/prometheus/issues/2347#issuecomment-1867689333
[09:50:13] <slyngs>	 It also doesn't seem to work when I test locally
[09:50:58] <dcaro>	 it's an issue with the yaml standard iirc, and some libs tend not to implement it, some do, but it's not a must
[09:52:02] <dcaro>	 I think it never left the draft stage https://yaml.org/type/merge.html
[09:52:29] <slyngs>	 Hmm I might need to revert the removal of the Icinga check, because I don't think there's a way to get AlertManager to route the alerts to WMCS as is. It's kinda all or nothing. That is an issue we had with other alerts, but I don't recall if there was a fix
[09:55:33] <slyngs>	 dcaro: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1087397 <- I'll talk to o11y and see if there's a way to route the alerts correctly, so that WMCS get the alerts from the cloud hosts and the reset go to SRE.
[09:55:57] <dcaro>	 slyngs: thanks!
[10:02:20] <slyngs>	 dcaro: Sorry about the noice
[10:02:49] <dcaro>	 np, thanks for moving us out of icinga :)
[10:20:43] <arturo>	 heads up, I will be testing stuff in codfw1dev, potentially making the APIs unstable
[10:20:53] <arturo>	 (similar to yesterday)
[10:28:58] <dcaro>	 ack
[10:32:51] <blancadesal>	 dcaro: is there a reason toolforge-deploy uses the python 3.9 CI image instead of 3.11? 
[10:34:32] <dcaro>	 blancadesal: I don't think so, it does run on bastions and k8s control nodes though, maybe it comes from there, looking
[10:35:30] <dcaro>	 they all have 3.11, maybe we forgot to change once the VMs were upgraded to bookworm
[10:39:59] <blancadesal>	 in the case of toolforge-deploy, we are not using pre-commit from tox but directly from the CI container,  and it has pre-commit==3.0.1 which also causes issues now
[10:40:09] <blancadesal>	 so I'd like to move it to 3.11
[10:40:33] <dcaro>	 +1 from me
[10:53:19] <blancadesal>	 mr: https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/578
[10:56:04] <dcaro>	 LGTM
[10:56:24] <dcaro>	 small usability improvement for lima-kilo https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/208
[10:58:33] <blancadesal>	 nice!
[11:07:22] * dcaro lunch
[11:07:32] <Raymond_Ndibe>	 how should I go about getting the logs of a bucket using either radosgw-admin or wmcs-openstack? are the logs enabled? is this possible? anyone?
[11:07:49] <Raymond_Ndibe>	 explored this the other day with dcaro but we didn't go far enough
[11:09:07] <arturo>	 Raymond_Ndibe: what are you looking for? 
[11:09:20] <Raymond_Ndibe>	 dcaro: I forgot to mention this in our meeting yesterday, but the harbor_tests bucket seems to be working. I can push and ever
[11:11:04] <dcaro>	 Raymond_Ndibe: awesome :)
[11:11:20] <Raymond_Ndibe>	 even replicate to harbor_tests, so the issue appears to be from buckets in the region https://object.eqiad1.wikimediacloud.org
[11:11:42] <dcaro>	 for the logs, there's several sources, are you interested on any specific layer? (openstack, ceph, rados, proxies...)
[11:12:22] <Raymond_Ndibe>	 arturo: I am trying to find out why buckets in the region https://object.codfw1dev.wikimediacloud.org cannot be used for harbor storage while those in the region https://object.codfw1dev.wikimediacloud.org can be
[11:12:24] <dcaro>	 you have https://logstash.wikimedia.org/app/home, this has openstack and ceph logs (using the openstack dashboard, deselect the openstack services and filter by cloudceph nodes)
[11:13:09] <Raymond_Ndibe>	 dcaro: tbh I want everything. Every possible place that logs might live
[11:13:33] <dcaro>	 logstash is the best place to start imo, as it aggregates all the nodes and services
[11:13:41] <arturo>	 yeah
[11:13:53] <arturo>	 also, debug level logs are not usually enabled in either ceph or openstack
[11:14:21] <arturo>	 Raymond_Ndibe: I don't understand what you mean by "cannot be used for harbor storage" is there a failure?
[11:14:22] <Raymond_Ndibe>	 I tried playing with logstash a bit but I had no idea what I was looking at or if I am looking at the logs I want. But maybe let me look again
[11:14:24] <dcaro>	 then you can try going to every cloudcontrol (or a specific one if you pinpoint it in logstash), and there you have journalctl I think that has some things
[11:15:55] <Raymond_Ndibe>	 arturo: the buckets just don't work. After connecting them to harbor for every operation I keep getting this weird 500 error unknown. also this happens without connecting to harbor
[11:16:38] <Raymond_Ndibe>	 for example trying to use s3cmd to push to any of such buckets fails with a weird error
[11:17:27] <Raymond_Ndibe>	 even something as simple as `s3cmd info <bucket>` retries a number of time with 500 error unkown before finally succeeding
[11:17:44] <Raymond_Ndibe>	 so not particularly related to harbor, it's from the buckets themselves
[11:17:49] <arturo>	 weird!
[11:18:19] <arturo>	 it may be related to this error Raymond_Ndibe T360626
[11:18:20] <stashbot>	 T360626: Frequent radosgw 500 errors with OpenTofu - https://phabricator.wikimedia.org/T360626
[11:18:37] <Raymond_Ndibe>	 had issues with that for so long I thought I was doing something wrong, until David created a bucket on testlabs in the region https://object.codfw1dev.wikimediacloud.org and that worked without a glitch 
[11:19:02] <dcaro>	 oh, I remember looking a bit, might be related to fernet issues?
[11:19:03] <Rook>	 Was it my imagination that we had a redis option running in openstack, or am I looking for it in the wrong place?
[11:19:45] <dcaro>	 Rook: might have been disabled from trove (I'm guessing you mean as DB in trove), not sure what was the last status there
[11:20:08] <Rook>	 Ok, I'll stop looking for it then. Thanks!
[11:20:14] <Raymond_Ndibe>	 arturo: yes, I think that's the exact error. Nice to see someone noticed it already in the past
[11:21:11] <arturo>	 Raymond_Ndibe: reasons for the error are unknown at the moment. Only know solution at the moment is to retry
[11:22:56] * dcaro gtg
[11:23:39] <Raymond_Ndibe>	 That is bad. Will keep investigating. Need it working before we can use obect storage for harbor
[11:24:54] <Raymond_Ndibe>	 maybe we can start by looking at the buckets that work and the ones that don't
[11:25:20] <arturo>	 I suspect this is not related to particular buckets, but for the API machinery itself
[11:25:31] <arturo>	 it maybe haproxy, radosgw, etc
[11:25:36] <Raymond_Ndibe>	 arturo: also any idea how to increase the size of a bucket?
[11:26:31] <Raymond_Ndibe>	 yes I suspect the same too. reason it's not exactly easy to pinpoint. But comparing the infra backing buckets might help
[11:28:00] <arturo>	 Raymond_Ndibe: what do you mean the size of the bucket? the quota?
[11:29:30] <Raymond_Ndibe>	 yes the quota. I think the default is around 8 - 9 GiB
[11:29:39] <arturo>	 yes, 8G is the default
[11:29:57] <arturo>	 this is documented here https://wikitech.wikimedia.org/wiki/Help:Object_storage_user_guide
[11:31:19] <Raymond_Ndibe>	 ok thanks
[11:35:40] <dcaro>	 Raymond_Ndibe: last time we debugged it we got to the point where we were seeing fernet token decpryption errors on all but one cloudcontrol (in logstash), did you continue exploring that?
[11:38:22] <dcaro>	 essentially https://logstash.wikimedia.org/goto/912e938cb8583c72736cf3b86ff00de3
[11:38:31] <Raymond_Ndibe>	 dcaro: are you having issue sshing into ssh toolsbeta-harbor-1.toolsbeta.eqiad1.wikimedia.cloud or is it just me?
[11:39:07] <dcaro>	 Raymond_Ndibe: can't login as my user, only root
[11:39:19] <dcaro>	 probably sssd down (sometimes it stops working after OOMing)
[11:40:08] <Raymond_Ndibe>	 dcaro: did we look at logstash? we mostly looked at the servers together playing with radosgw-admin and wmcs-openstack
[11:40:43] <dcaro>	 Raymond_Ndibe: yep we did
[11:41:36] <Raymond_Ndibe>	 I didn't look at that again. I just backed up the commands we were playing with on the servers. Maybe that's a good place to continue from
[11:41:47] <dcaro>	 hmm, for toolsbeta it's not sssd, it seems it's not getting replies from ldap
[11:42:13] <dcaro>	 Raymond_Ndibe: yep, can you update the ticket with that info?
[11:42:20] <dcaro>	 gtg, be back in a bit
[11:42:42] <Raymond_Ndibe>	 We should increase the size of toolsbeta-harbor server. the storage is like 20GB and harbor is already consuming close to that
[11:42:52] <Raymond_Ndibe>	 ok I'll do that
[13:07:31] <moritzm>	 there's bugfix updates for mod_oidc, is there anything to consider when updating it? the update would trigger an apache reload, but should be brief
[13:19:39] <arturo>	 moritzm: is that being used in the IDP setup?
[13:26:49] <moritzm>	 well, indirectly since the OIDC protocol ultimately ends up on the IDPs, but on cloudcontrol it is used by some uwsgi keystone proxy
[13:26:57] <moritzm>	 but not sure what this does exactly?
[13:27:36] <arturo>	 I'm not sure either, but you may safely update it. Keystone can generally survive an apache reload
[13:32:10] <moritzm>	 ok, thanks! I'll do that in a few minutes
[13:56:36] <moritzm>	 these are done now
[13:58:49] <arturo>	 thanks
[15:04:08] <blancadesal>	 dcaro: https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/64
[15:06:39] <dcaro>	 blancadesal: they all show empty to me :/?
[15:07:31] <dcaro>	 oh, because of https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/blob/main/.gitattributes?ref_type=heads#L2
[15:07:38] <dcaro>	 I'll have to checkout to review
[15:08:21] <blancadesal>	 locally they look fine
[15:08:25] <blancadesal>	 (I think?)
[15:08:34] * blancadesal afk for a while
[15:09:36] <dcaro>	 ohhh, 
[15:10:01] <dcaro>	 I think it might be related to the certs dates being hardcoded in the recording, and then the code might try to renew stuff and such
[15:10:24] <dcaro>	 we might want to use freezegun or similar
[16:16:51] <arturo>	 yeah, that makes sense, specially since the cert lifetime was reduced to 10 days, from a year
[16:25:30] <arturo>	 it seems I will make it to the toolforge monthly meeting at least partially
[17:54:25] <dhinus>	 /usr/local/bin/wmcs-dnsleaks is failing with "Could not find project: tf-infra-test"
[17:56:24] <dcaro>	 I think that project was removed? (I saw a task flying by or something)
[17:56:34] <dcaro>	 gtg
[17:56:36] * dcaro off
[17:57:03] <dhinus>	 yes the project was replaced by the newer "tofuinfratest"
[17:57:14] <dhinus>	 but maybe some traces were left
[17:57:31] <dhinus>	 I'm trying to figure out why the script is failing
[18:02:04] <dhinus>	 it looks like there is something still associated to that deleted project
[18:15:05] <dhinus>	 left some notes at T379076
[18:15:06] <stashbot>	 T379076: Remove tf-infra-test project - https://phabricator.wikimedia.org/T379076
[18:16:41] <dhinus>	 from what I can see this is not causing other issues apart from that dnsleaks script failing, which is not critical
[18:24:46] * dhinus off for today