[08:12:05] morning! [08:12:32] hello! [10:40:42] for awareness, data-persistence reported some big CPU spikes in clouddb1014, probably due to user activity. they have stopped now, I opened T429892 to track this in case it happens again. [10:40:43] T429892: CPU spikes in clouddb1014@s2 - https://phabricator.wikimedia.org/T429892 [12:06:50] the "scholia" tool that caused a page on sunday is another victim of scraping: https://github.com/WDscholia/scholia/issues/2794 [12:28:22] dhinus: found another regression in jobs-api, do you want to take it or prefer that I do? T429907 [12:28:22] T429907: [jobs-api] can't run a buildservice job with filelog - https://phabricator.wikimedia.org/T429907 [12:28:37] (it kinda messes with the tests for the jobs-cli/jobs-api changes a bit) [12:30:11] for scholia, might be interesting if we can offer an "easy" rate limiting for tools, and/or somehow make anubis work [13:09:24] dcaro: I can take it [13:09:46] re:rate limiting, I think we do have one in place? [13:12:35] Yes, global (I think), I mean configurable by users [13:13:02] Let me know if you need anything with the fix, it's prioritary (imo) [13:14:54] sure let me see if I can fix it now, if it gets messy I'll ping you :) [13:17:38] side thread: I'm after cloudvirt10[77-80] and they are lacking v4/v6 in the cloud-private subnet thus puppet fails, did I get it right I have to add an interface to netbox with the correct vlan for those hosts ? [13:20:39] mmhh or maybe not? I see from 1076 changelog that a bot did the vlan assignment, so perhaps a cookbook? https://netbox.wikimedia.org/dcim/devices/5754/changelog/ [13:22:10] godog: last time I set up cloudvirts, topranks had to do some behind-the-scenes magic to get them on cloud-private. [13:22:34] Well, it's possible that I'm supposed to know how to do it myself but it's still filed as 'cathal does magic' in my memory. [13:23:01] andrewbogott: that seems to check out, I'll inquire netops [13:24:42] I'm trying to make a cookbook to do it but I need to refactor [13:24:44] https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/1303397 [13:25:11] nice [13:25:22] you can run it right now shaddow ops style [13:25:36] test-cookbook -c 1303397 sre.network.cloud-host cloudvirtXXXX [13:25:54] lol ok happy to [13:28:08] godog: indeed I did it for one or two of the new hosts last week while testing [13:28:43] topranks: makes sense, if you have time now to hold my hand, I got this, if not that's fine too it can wait https://phabricator.wikimedia.org/P94369 [13:29:05] this being a change involving cloudcephosd1053 too [13:29:13] and 1054 [13:29:31] ok [13:29:43] so lines 54-66 are all I'd normally expect to see [13:30:10] the rest relate to two new hosts it seems? looks like DC-ops added them but never ran the script they are supposed to to configure the switch for them [13:30:22] cloudcephosd1053 / cloudcephosd1054 [13:30:38] but anyway it all seems safe to push, enabling those ports won't cause an issue [13:30:45] ok unfortunate timing, going ahead [13:30:55] yeah it could just be we're doing it at the same time [13:31:10] homer runs for the whole switch so will pick up any changes for it in Netbox since last run [13:32:37] makes sense, ok committed now, 1078 and 1080 got no changes which I think are the ones you used for testing? and I just did 1079 [13:33:01] yep that sounds about right [13:33:17] sorry didn't want to do them all as I was hoping you guys could beta test it for me with some of them [13:33:34] yeah totally [13:33:52] Arzhel is determined to make me a software engineer so I need to rework it more than I thought though :P [13:34:11] I'll get it over the line this week hopefully [13:34:22] it can also be re-run at any time for any host like you seen [13:34:29] if there are no changes needed it will do nothing [13:34:31] haha! I'm fine with beta-testing, ok cookbook ran on all now [13:35:20] great [13:35:21] not sure at what point in the provisionign we should be at now, I see no vlan interface on https://netbox.wikimedia.org/dcim/devices/6770/interfaces/ [13:35:38] don't worry about that [13:36:05] puppet creates the additional interfaces, there is a script we can run to import the network status from the host to Netbox [13:36:33] but that part is only for documentation purposes [13:36:52] the puppet part of this - however - I am not 100% sure on all aspects [13:37:16] there is an IP allocation part I think, possibly needs IPs added in Hiera ? [13:37:51] yeah puppet will require the dns name to be in place first to work, the ip allocation part [13:37:55] fwiw what the cookbook did is add the vlans to the switch port connecting to the host: https://netbox.wikimedia.org/dcim/interfaces/16075/ [13:38:26] ah I see ok so I think what's left is the host side bits [13:38:36] so for that one, cloud-private-c8, it's this IP block: [13:38:36] https://netbox.wikimedia.org/ipam/prefixes/653/ip-addresses/ [13:38:56] I could possibly add this allocation in to the cookbook, but I wasn't 100% sure how it worked [13:39:19] i.e. if it goes into Netbox first, or it goes into Hiera and we pull it from the live host into netbox [13:39:22] godog earned 🖧🥷++ [13:39:35] sounds like we allocate in Netbox first? and then puppet uses the domain name to configure it on the box? [13:39:56] topranks: that's my understanding too, puppet expects dns names to resolve to v4/v6 [13:40:05] dcaro: hahaha! [13:40:05] ok [13:40:41] should be possible to add the cookbook to allocate those then [13:40:56] said otherwise: there are no hardcoded private dns names or addresses in puppet for cloudvirts [13:41:30] topranks: I'm happy to wait and beta-test the cookbook with address allocation FWIW [13:41:32] ok cool, yeah I think this was different originally and got reworked [13:41:44] dns isn't the most perfect possible way to go but it will work [13:42:31] safe to say we can remove "to go but it will work [13:42:32] " [13:45:52] haha yep. but it's ok [13:46:17] dcaro: just looking it seems we don't allocate the cloud-storage IPs in Netbox? [13:47:03] that might be true, the internal range ones right? I think it's hardcoded in puppet [13:47:26] yeah the 192.168.x.x addressing, doesn't seem to be in Netbox [13:47:32] (iirc there was a reason, but if it's not the way to go anymore, happy to change) [13:47:41] yes cloudcephosd are puppet-driven instead [13:47:55] which makes me think to automate the current process I would add the assignment for the cloud-hosts vlans (and dns) but not cloud-storage [13:48:07] at least until/unless we change how we deal with the storage ones [13:48:36] +1, now with cephosd single-nic in place we can switch those to dns-driven too [13:48:45] and stop hardcoding stuff in puppet [13:48:53] cloudcephosd even [13:49:54] that would be good yeah [13:50:21] I'll leave it for now, easy to add later once we get it in place [13:50:31] SGTM [13:50:32] rather than making that change a dependency on getting the cookbook done [13:51:14] yeah auto ip allocation for cloudvirt via cookbook sounds good [13:52:58] +1 for that [17:42:05] * dcaro off [17:42:49] dhinus: I'll test the current patch tomorrow morning, but I left a message, I think we will also need to bump the validation/full job resolution before the storage is saved [17:42:55] cya! [17:43:10] thx! [17:46:15] hmm, sample-complex-app is failing to start :/, I'll open a task to investigate [17:55:33] oh, the deploy token expired, but the app is also failing to start, I'll fix tomorrow [19:00:44] there are multiple issues related to the same thing (inability to login to deployment prep)- T429978: Project members cannot ssh into newly created deployment-prep instances., and T429978. who can look at this? [19:00:44] T429978: Project members cannot ssh into newly created deployment-prep instances - https://phabricator.wikimedia.org/T429978 [19:01:12] T429542 is the other one [19:01:12] T429542: debian-12.0-bookworm and debian-13.0-trixie image still reference mirrors.wikimedia.org - https://phabricator.wikimedia.org/T429542 [19:55:22] bliviero: I'll have a look if/when I'm free of the paws maintenance [20:20:25] ok, that's paws sorted, now the mirrors thing... [22:15:53] bliviero: I'm pretty sure that both of the above issues are resolved now. Ahmon is re-testing and I'll be back to check his results after dinner. Thanks for flagging those! [22:21:57] thank you andrewbogott