[00:11:07] <btullis_>	 Oh sorry
[00:12:37] <btullis_>	 That is me. I put snapshot1008 into insetup earlier as part of https://phabricator.wikimedia.org/T325228
[00:42:44] <mutante>	 btullis_: it makes sense to put it in setup role but that doesn't remove it from scap groups.
[00:42:53] <mutante>	 it's the dsh.yaml
[00:42:59] <mutante>	 hieradata/common/scap/dsh.yaml:      - snapshot1008.eqiad.wmnet
[00:43:29] <mutante>	 it means scap will still try to push files over there
[00:44:14] <mutante>	 snapshot are special in "not in conftool but ALSO in scap.. as individual special cases"
[00:44:42] <mutante>	 the "dsh group" mediawiki-installation is a combination of a couple conftool groups AND a list of "random" hostnames
[00:46:57] <btullis_>	 Does it require immediate action on my part? I am away from keyboard, but I could get there if it's important. 
[00:47:26] <mutante>	 not until next deployment for sure.. and no.. 
[00:47:34] <mutante>	 because I can merge this: https://gerrit.wikimedia.org/r/1032610
[00:47:59] <mutante>	 I'll remove it 
[00:48:21] <btullis_>	 ack, thanks. 
[00:49:09] <btullis_>	 Apologies, I do not know much about dsh yet. 
[00:49:23] <mutante>	 just keep it in mind when you go to the next snapshot host.. that you have this list of host names in that yaml
[00:49:49] <mutante>	 it's very confusing.. dsh is just the name of a tool we used in the ancient past
[00:50:03] <btullis_>	 affirmative. 
[00:50:07] <mutante>	 but snapshot is special :p
[00:50:22] <mutante>	 because it gets mw deployments
[05:25:20] <marostegui>	 I restarted wikibugs
[09:24:31] <Amir1>	 which host I should login to run authdns-update?
[09:24:38] <Amir1>	 dns100[12] doesn't work?
[09:26:03] <godog>	 Amir1: I usually ssh to their public names, e.g. ns1.w.o
[09:26:17] <Amir1>	 yup, thanks!
[09:26:22] <godog>	 sure np
[10:10:51] <topranks>	 that's odd... I often do it after sshing to dnsXXXX and it works ok 
[10:11:40] <XioNoX>	 Amir1: we're at dns1004
[10:12:04] <Amir1>	 yeah, I should have continued upwards, I gave up after 1002
[11:13:30] <topranks>	 ah ok - it was the actual login didn't work, not authdns-update once on there 
[11:19:43] <Amir1>	 yeah
[11:19:46] <Amir1>	 sorry, my bad
[11:54:22] <sukhe>	 Amir1: where did are you seeing dns100[12]? because we should update that; I tried to make sure that that was up to date but clearly not
[11:54:43] <sukhe>	 s/did are/are
[12:27:08] <Amir1>	 I think I was used to authdns1002
[12:50:44] <sukhe>	 that's fair. so now there is no authdnsN, just dnsN and any (one) host in A:dnsbox is fine to run authdns-update
[12:56:46] <cdanis>	 Amir1: there may be some way to teach your shell about ~/.ssh/known_hosts.d/wmf-prod and then you get tab completion
[13:04:29] <Amir1>	 I do have the wmf-sre package (whatever its name is) and use that, somehow didn't use tab completion, probably needed more coffee. My bad.
[13:05:34] <Amir1>	 sukhe: one thing in doc is that it would be great to have a section in DNS (https://wikitech.wikimedia.org/wiki/DNS) for standard dns update process. e.g. go to this host, run this.
[13:05:54] <Amir1>	 at least there, I will have the host name
[13:06:42] <sukhe>	 thanks, I will update https://wikitech.wikimedia.org/wiki/DNS#authdns-update and add it under there
[13:09:37] <Amir1>	 Thanks
[13:19:16] <sukhe>	 Amir1: https://wikitech.wikimedia.org/wiki/DNS#Deploying_DNS_changes
[13:19:27] <sukhe>	 I moved it as the first section as the rest of the page is quite detailed
[13:19:55] <Amir1>	 sukhe: Thank you so much <3
[13:19:59] <Amir1>	 you're the best
[13:20:21] <sukhe>	 ha hardly but ok :P
[13:32:43] <sukhe>	 :) 
[13:32:46] <sukhe>	 er wrong window
[16:18:24] <sukhe>	 very helpful wikitech page denisse!
[16:19:35] <mutante>	 still connects to ns0.wikimedia.org
[16:19:47] <denisse>	 sukhe: Thanks! :)
[17:46:52] <mutante>	 trying to rsync data between 2 hosts and used auto_firewall = true, with nftables instead of iptables. and it's like neither working nor not working.  "sent 32 bytes  received 232 bytes  2.02 bytes/sec"  what ?:p
[17:47:12] <inflatador>	 LOL
[17:47:20] <mutante>	 it seems weird to not just timeout completely but 'a few bytes'
[17:47:47] <inflatador>	 maybe there's some icmp reject or something to account for the bytes?
[17:48:02] <mutante>	 hm, good idea. yea
[17:48:44] <mutante>	 let me first try if everything works when I change the firewall provider back to iptables I guess
[17:48:51] <mutante>	 just to rule out things
[17:50:32] <mutante>	 oh.. permission problem to create a new subdirectory .. well...then
[18:21:41] <mutante>	 by default rsync server only listens on 0.0.0.0 , unless you actively pass "empty string" to the "address" parameter. which explains there was always a weird delay first when it tries IPv6 until it gives up. Since we include rsync::server in a bunch of places with the defaults.. but all the names have IPv6 records now, I assume this delay is all over the place but since it _eventually_ works we 
[18:21:47] <mutante>	 didnt notice it much.
[18:40:28] <cdanis>	 need rsync protocol filter for envoy
[18:56:52] <mutante>	 regardless of listening on IPv6, still can't push to it over IPv6 with auto_firewall in combination with nftables.  nftables::service uses wmflib::hosts2ips which uses dnsquery::lookup and the resulting nft rules also has both v4 and v6 ... but rsync only works with -4 
[18:57:24] <mutante>	 still trying to figure out the missing link
[19:01:57] <cdanis>	 is it the correct v6 address?
[19:03:06] <mutante>	 yea, looks correct. it resolves to the host I am coming from
[19:05:26] <mutante>	 ICMP works 
[19:43:17] <mutante>	 cdanis: it's because my source host as 2 public IPs bound to the same interface :p  it works better when I specify  --address with rsync to tell it which one to use as source IP 
[19:43:35] <cdanis>	 what are the two IPs out of curiosity?
[19:43:44] <mutante>	 lists.wikimedia.org and lists1001.wikimedia.org
[19:43:49] <cdanis>	 hah
[19:43:58] <mutante>	 indeed
[19:46:43] <mutante>	 somehow this matters only for v6 though