[06:08:48] greetings [07:59:41] morning! [08:32:44] mornign [08:33:00] * dhinus starting the day with a typo :D [10:16:32] marostegui is putting the new clouddb1026 into service, but there's a firewall issue, cloudlb* cannot connect to clouddb1026 on the mysql port [10:16:47] anybody remembers where that is configured? [10:22:48] opened T430132 [10:22:49] T430132: cloudlb100X cannot talk to new host clouddb1026 - https://phabricator.wikimedia.org/T430132 [12:14:53] ok today I'm setting up cloudvirt1077, baby steps so far though VMs are being scheduled on it, nftables is active on the host unlike other cloudvirts though and I'm debugging traffic that's being blocked [12:16:19] still TBD why we got nft as opposed to existing cloudvirts [12:17:50] godog: the server was initially installed with role::insetup::wmcs_nftables [12:18:05] and the Nova code to configure access into VMs doesn't support nftables yet [12:18:42] moritzm: hah thank you, that explains, did I get it right that if I was to reimage the host then it wouldn't get nft ? [12:19:35] indeed, if you reimage now it will properly pick up ferm [12:20:06] but for 1078 to 1080 better change site.pp to insetup::wmcs_ferm and reimage them before you apply the role [12:20:22] ok thank you [12:29:27] FWIW I spot checked cloudvirt1046 and there's no firewall whatsoever, ferm or nft [12:39:13] oh, in fact I totally mis-remembered [12:39:31] these should in fact be initially installed with the noferm insetup role [12:39:53] since otherwise the iptables rules managed by nova-proxy clash with the iptables state managed by ferm [12:41:29] sorry for the confusion, but an in-place reimage of 1077 should still do the right thing [12:41:41] ok no worries, I'll reimage 1077 [13:17:58] * andrewbogott surprised that puppet doesn't remove the firewall when it's no longer defined [13:27:09] seeking two +1s for https://phabricator.wikimedia.org/T430157 since it is a big increase [13:28:18] +1'd [13:28:53] thank you [14:35:30] godog, T419892 is currently assigned to me, would you like me to just re-assign it to you or should we discuss what to do about the 25G switch thing? [14:35:30] T419892: Q3:rack/setup/install cloudcephosd105[3456] - https://phabricator.wikimedia.org/T419892 [15:51:55] andrewbogott: yes I'll take that on, thank you for the heads up [15:52:03] thx [15:52:56] {{done}} [16:12:29] heads up that fullstackd currently has three VMs scheduled on 1077 and no more VMs are going to be scheduled on it, I'll resume putting it in service tomorrow [16:53:11] godog: any reason for me to not delete those VMs? [16:58:54] also, unrelated: pretty sure this is what is breaking automated migration from old to new nfs servers. dcaro, godog, do you remember what was happening here? https://gerrit.wikimedia.org/r/c/cloud/wmcs-cookbooks/+/1189868/10/cookbooks/wmcs/nfs/add_server.py [17:25:55] I see several potential things going on here, from "tool owner seeing their tool (resource / user) consumption in a UI" to figuring out if the tool is misbehaving (via logs). There's a 3rd angle- if the tool is non-responsive, that discovery may have to happen from something outside of the tool. Is the mechanism that created the oncall page in https://phabricator.wikimedia.org/T429738 something that checks all tools? [17:48:27] * dcaro off [17:55:37] bliviero: yes, the prometheus data that the alert uses comes from our own istio ingress deployment, and that serves all the tools, not sure what you are referring to though xd [18:00:48] andrewbogott: I don't remember, it was not reviewed actually, added on the last patch right before merging, maybe a mistake [18:00:55] now /me off for sure :) [18:00:57] cya!