[07:02:29] <elukey>	 hello folks!
[07:03:16] <elukey>	 I'd need to restart the jvm of puppetdb, and IIUC the least impactful way is to disable puppet fleet-wide, wait a bit, and then restart both
[07:03:51] <elukey>	 would it be ok if I do it now, or is there anything ongoing/pending/with-more-priority ?
[07:11:41] <_joe_>	 elukey: not objecting, btu just curious
[07:11:52] <_joe_>	 what happens if you don't stop puppet everywhere?
[07:12:12] <moritzm>	 thunderstorms of failed puppet runs
[07:12:25] <moritzm>	 and puppetdb takes like two minutes to restart, so they add up
[07:12:35] <_joe_>	 ok so basically the same result, it's just that we reduce the noise, ack
[07:12:35] <elukey>	 ahahaha TIL about the two minute
[07:12:42] <elukey>	 *minutes
[07:12:56] <_joe_>	 maryum: just two minutes? pretty swift for a jvm application!
[07:13:03] <_joe_>	 err sorry maryum I meant moritzm 
[07:13:05] <_joe_>	 :)
[07:13:18] <moritzm>	 not sure if it's actually still two minutes :-)
[07:13:28] <moritzm>	 that data is from the time
[07:13:33] <moritzm>	 when puppetdb ran on 16G VMs
[07:14:06] <moritzm>	 since some time they are on 64G baremetal and probably need 4x the time, not sure :-)
[07:19:06] <elukey>	 the puppetdb's jvm runs with only 6GB, I didn't expect 2 minutes of bootstrap
[07:28:01] <elukey>	 puppet disabled fleetwide
[07:35:37] <elukey>	 proceeding with 2003
[07:39:14] <elukey>	 done with 2003 and 1003
[07:39:48] <elukey>	 was pretty quick, around 10/20 seconds
[07:42:00] <elukey>	 trying to run puppet on esams nodes first 
[07:55:29] <elukey>	 re-enabling puppet fleetwide
[08:08:22] <moritzm>	 ah, good to know that it's quicker on baremetal
[08:22:34] <elukey>	 we may think to test a restart (next time) without a fleetwide disable, even though it is good if anything happen to the JVM (like bad update etc..)
[09:17:58] <_joe_>	 elukey: it's not advisable to send requests to a JVM before you've properly warmed the glow plugs
[09:18:57] <_joe_>	 (a tractor diesel engine is my mental image of the JVM)
[09:26:20] <moritzm>	 etherpad.w.o will be briefly gone for a firewall fixup
[11:40:44] <claime>	 btullis: can i merge the DNS add for rgw.eqiad.dpe ?
[11:41:03] <claime>	 It's in my sre.dns.netbox run
[11:41:25] <claime>	 Rados Gateway (S3/Swift) interface of the DPE Ceph cluster in codfw apparently
[11:41:32] <btullis>	 Oh yes, please do go ahead. Sorry, I didn't realise I should have done it myself.
[11:41:40] <claime>	 no worries
[11:43:46] <topranks>	 claime: I was just about to ask Ben about those...
[11:44:10] <topranks>	 I think you may have some problems pushing the dns - the v6 reverse entries I think will cause an issue 
[11:44:12] <claime>	 E003|MISSING_OR_WRONG_PTR_FOR_NAME_AND_IP: Missing PTR '8.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.1.0.1.0.0.0.f.f.0.8.c.e.2.0.a.2.ip6.arpa.' for name 'rgw.eqiad.dpe.anycast.wmnet.' and IP '2a02:ec80:ff00:101:
[11:44:14] <claime>	 yes
[11:44:23] <topranks>	 yeah... 
[11:44:36] <topranks>	 are you running the dns cookbook itself?  or some other cookbook that calls it?
[11:45:04] <claime>	 other cookbook that calls it
[11:45:32] <topranks>	 can you just abort the dns part or will that bork the overall work ?
[11:45:53] <topranks>	 TL;DR to fix I need to create a manual patch for the dns repo and merge, then re-do the authdns-update (bit that is failing for you)
[11:45:54] <claime>	 the dns part is the work, it's the renumbering cookbook :)
[11:46:11] <claime>	 (well it's part of the work anyway)
[11:46:25] <claime>	 I can wait until you've done that and retry
[11:46:30] <topranks>	 ok... yeah so you need updated dns entries for the rest to complete, is it prompting you to retry?
[11:46:33] <claime>	 or skip
[11:46:37] <claime>	 yep
[11:46:57] <topranks>	 maybe pause for a few mins I'll get my path in, retry should work then 
[11:47:02] <topranks>	 *patch
[11:48:19] <claime>	 ack
[11:48:33] <claime>	 ping me when it's good :)
[12:05:03] <topranks>	 claime: ok I think we are back in a healthy state 
[12:05:13] <claime>	 cool ty
[12:05:14] <topranks>	 do you want to see if 'retry' handles things cleanly?
[12:05:16] <claime>	 yep
[12:05:34] <topranks>	 otherwise it's probably safe to skip as I've had to do the update in the meantime anyway 
[12:06:12] <claime>	 looks clean, thanks
[12:06:18] <topranks>	 nice
[12:06:34] <topranks>	 IPv6 PTRs are truly the gift that keeps giving :) 
[13:28:11] <andrewbogott>	 moritzm: ok if I merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/1070586?  I want to see if I can get it hooked up to horizon.
[13:37:43] <moritzm>	 andrewbogott: ack, go ahead. worst case we can always revert
[13:37:53] <andrewbogott>	 ok :)
[14:37:14] <andrewbogott>	 moritzm: it works!  But also I have created https://phabricator.wikimedia.org/T374123 for your team.
[14:40:26] <moritzm>	 ack, I'll have a look later