[07:35:31] good morning folks :) [07:36:20] I am going to upgrade the ml-staging k8s cluster with the new cookbook, hopefully you will not see too much noise (alerts, irc spam, etc..) but something may fire. In case (please-ping|blame) me :) [07:38:05] good luck! [07:42:05] XioNoX: <3 [09:01:56] is anyone here running Windows? [09:04:05] no but I have a windows close by if needed [09:09:55] volans: it's for https://phabricator.wikimedia.org/T329264#8603805 [09:10:04] dunno how correct the openssl instructions are [09:10:31] I'll reply to you in private [09:53:39] XioNoX: I have another WIndows 10 installation for testing if needed. [09:54:17] klausman: thanks, I think Riccardo is almost out of that rabbit hole [09:54:32] Roger! [10:00:05] yep, I ended up installing openssl in a different way [10:04:22] Isn't there a curl build for W10? [10:16:45] curl is built-in, s_client no [10:16:53] builtin as in is available [10:18:02] elukey: I'll be interested in your experience with the cookbook and things to do before/after [10:18:19] we may have to upgrade aux-k8s relatively quickly [10:18:30] claime: sure! The cookbook is broken atm but when it works I'll let you know :D [10:18:48] yes the idea is that aux/dse/ml-* should benefit from the cookbook [10:19:18] in theory also wikikube prod ones, but since we can do only one reimage at the time we'll need to figure out a different way to run it [10:19:41] (like we restrict the target worker nodes to reimage to say 2, and do the rest manually in paralle) [10:19:44] *parallel [10:20:01] the pre-requisite is only to have prepped a puppet change [10:22:02] A'ight :D [14:13:48] elukey: can I help with the cookbook :D [14:17:10] cdanis: o/ I have a very skilled python dev that is helping me right now, we should be close to a fix :D [14:23:48] The dev is your cat, isn't it? [14:24:19] nono the cat prefers Go :D [14:25:04] cdanis: I added the configs to do all k8s clusters so in theory we shouldn't add anything when it works [14:25:10] Good cat! [14:42:12] slyngs: o/ I left a comment in https://phabricator.wikimedia.org/T306661#8605007, the ganeti reimage cookbook seems broken [14:42:36] do you know what could be the issue? (Should I modify the dhcp config in puppet?) [14:42:45] Yeah, you need to remove the host from /etc/dhcp/linux-host-entries.ttyS0-115200 [14:43:08] Once fully tested we'll remove the include altogether [14:43:34] Like we've done here: https://gerrit.wikimedia.org/r/c/operations/puppet/+/727387 [14:44:51] slyngs: shall I proceed with https://gerrit.wikimedia.org/r/c/operations/puppet/+/888225 to test the cookbook then? [14:45:33] Yes, that should work [14:46:30] ack, and I need to run puppet on the install servers right? [14:46:42] (I mean only on those, or also elsewhere?) [14:50:16] Install server should be fine, unless I misunderstood how it works :-) [14:50:22] I'm not helping, I know [14:50:45] nono it is fine, just did it :) [14:53:46] ok it seems that now it works :) [14:54:15] ml-staging-etcd2001 got removed from puppet etc.. so it didn't show up in my cookbook's list, I'll reimage it manually [14:54:52] Wonderful, sorry, should have made the DHCP thing clear. In the long run the include will just go away, but volans says that I need to test EVERYTHING before I'm allowed to do so :-) [14:54:56] Or something like that [14:55:36] always listen to volans since he is a wise person :) No problem though! It is good so we can now test both cookbooks :) [14:58:50] slyngs: I said that before removing the hardcoded data we should make sure that the cookbooks works, and then once removing the DHCP hardcoded data it becomes the official (and only) way to perform Ganeti VM reimages. And should be documented and communicated. [14:59:27] Because the cookbook got merged, people will inevitably find it and use it... ideally the testing should happen within a reasonably short timeframe from when its merged. [14:59:55] They already found it, so you are very correct :-) [15:10:16] arturo: hello! I was looking at ipt-netflow and was wondering if there was an equivalent for nftables as they're not planning on porting it over https://github.com/aabc/ipt-netflow/issues/45 [15:14:34] XioNoX: has anyone implemented ebpf-netflow :D [15:15:00] cdanis: or go directly with smartnics? :) [15:15:45] 🤔 [15:16:59] slyngs: if you have a min, the reimage cookbook seems stuck in checking the uptime for ml-staging-etcd2002 (checked the ganeti console and it is blank) [15:17:00] cdanis: I was not expected Walmart https://medium.com/walmartglobaltech/introducing-walmarts-l3af-project-how-do-we-use-ebpf-to-provide-network-visibility-in-a-8b9ae4d26200 [15:17:08] expecting* [15:17:13] apergos: are you still the best person to contact regarding dumps? [15:17:16] elukey: Sure [15:17:25] XioNoX: hah! me neither [15:21:27] yeah unexpected wallmart go figure. looks pretty cool all in all [15:22:27] hare: yes, me and Hannah both [15:22:48] apergos: I emailed ops-dumps a couple weeks ago about hosting a mirror of RDF data; did you receive that? [15:22:49] if it's something emailable, there's the ops-dumps@wikimedia alias [15:23:01] hrm [15:23:22] got the subject line handy? I thought we'd gotten all replies in, let's see [15:24:15] "XML dumps mirror (actually RDF)" [15:25:50] nop0e nothing in jan or feb with a subject line even close to that [15:26:20] To: Ops-Dumps [15:26:25] lemme check spam but it would be odd if both Hannah and I got it in jumnk mail [15:26:27] Is that the right address? [15:27:58] in my spam folder [15:28:11] that's bad indeed [15:28:29] XioNoX: RE: netflow, I'll ask some friends and let you know [15:28:50] thx! [15:29:42] pulled it out, will respond Monday with Hannah and I looking at it together. (Also reported it as not spam, grrr) [15:30:01] thanks for pinging me, I never would have known [15:32:25] hare: ^^ [15:59:10] folks the ml-staging k8s cluster is currently half broken, the upgrade cookbook didn't work (still having some issues with the ganeti vm reimage and etcd is gone now) [15:59:28] I tried to downtime all, if you see any weird alert please ack or ignore them :) [16:00:30] :(