[09:03:25] please review: https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/34 [09:19:31] taavi: LGTM. [09:20:48] taavi: please approve https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/22 [09:21:47] done [09:22:11] thanks [09:40:23] there was an alert for quarry being down yesterday for ~1h, does anyone know what happened? [09:40:54] dcaro: no idea, and there was another one during the weekend [09:42:25] let's keep it in mind then, it might happen again :/ [09:57:48] arturo: next up: https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/36 [09:58:11] in a few minutes [10:02:27] taavi: there's some ip changes, is that intentional? [10:02:31] https://www.irccloud.com/pastebin/QEcMUu8m/ [10:03:02] dcaro: yes, that's moving the proxy to target the new server that was created in the previous MR [10:03:14] ack [10:03:47] lgtm (feel free to wait for arturo for an experienced review) [10:12:38] I left a few comments post-merge :-) [10:28:19] arturo: thanks, will look and reply in some moments [12:09:54] arturo: https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/37 wdyt? [12:10:05] looking [12:11:32] taavi: LGTM. Thanks, I think this is better than before [13:10:06] andrewbogott: T394775 does this ring a bell? [13:10:06] T394775: wmcs-enc-cli: keystoneauth1.exceptions.http.Forbidden: You are not authorized to perform the requested action: identity:list_services. - https://phabricator.wikimedia.org/T394775 [13:33:48] I just double-checked and the policy for that should be [13:33:51] identity:list_services: '' [13:34:02] which means the endpoint is unrestricted. [13:34:04] or it /should/ [13:34:24] Is it possible that auth is failing entirely and that's just the first endpoint it tries? [14:03:48] andrewbogott: definitely possible? [14:03:53] s/?/!/ [15:15:03] what's up with the NeutronAgentDown alerts? [15:15:15] andrewbogott: is that your rabbitmq rebuild? [15:15:25] yes [15:15:46] And almost all of them are back up by now [15:42:39] didn't help :( [15:54:17] taavi: I added the source code repo you gave to the toolinfo of the tool, I think that might have been what was missing [16:39:18] those cinder backend alerts are me debugging [16:43:55] ack [16:44:59] I want to change the `docker::registry` hiera from tools to `docker-registry.svc.toolforge.org`, I have looked around a bit but I don't see anything suspicious that might break things, anyone remembers any special place in puppet that changing the value will break? [16:45:50] (it's used for the imagebuilder host) [16:53:48] hmm, I think that might not work, the ip it resolves to is different, and the auth fails, probably some setting on the registry side [17:02:50] yep, the `docker-registry.svc.toolforge.org` goes through the proxy (so the source ip is from the proxy), while `docker-registry.tools.wmflabs.org` does not (so the source ip is the one from the image-builder) :/ [17:06:54] got https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/39 , I'll leave it there for today to see if anyone has any issues/comments [17:12:22] hmm... tofu says it does not have any changes for that patch :/, it should have at least one... well [17:15:16] dcaro: I see "0 to add, 1 to change, 0 to delete" in the terraform reports [17:16:26] * dhinus offline [17:17:39] oh I see, it had not finished and only showed one, though CI/pipeline showed everything finished :/ [17:18:39] I have proof xd [17:18:42] https://usercontent.irccloud-cdn.com/file/9mGWuBjs/image.png [17:23:55] * dcaro off